Skip to content

Posts from the ‘Methods’ Category

Better Group Research Sessions

Many user experience professionals cringe when clients want to do focus groups. In UX we extol the value of user research with individuals, either in usability testing or field studies, where we observe and interview individuals in the context of their tasks. Focus groups seem like a flaky, unreliable marketing activity. Unfortunately, because of our disdain for focus groups we sometimes extend that disdain to any research activity that involves a group of people.

Why Don’t We Like Focus Groups?

In UX we highly value observing the behavior of individuals in their natural context. Focus groups provide the exact opposite. They reveal what people say in a group out of context of their typical environment.

What People Say

Focus groups usually gather feelings, attitudes, opinions, and preferences; but they can’t tell us about behavior. We can’t rely much on what people say they do, because so much of what we do is done automatically without much conscious thought. So it’s difficult for people to accurately talk about what they do, out of context. That’s why we like to observe people in the context of performing their tasks. It’s more accurate, and it’s easier for people to show us what they do.

 In a Group

Being in a group has its limitations:

  • Group dynamics often influence individuals to go along with the group instead of speaking their own minds. They often say what they think will sound good to the group, and they may not feel comfortable speaking honestly about certain topics.
  • A few outspoken people often dominate the discussion, while others remain more silent.
  • You get much less information from each person than you would in individual sessions.
  • Group sessions are not useful for evaluation, because it’s hard for people to evaluate a design or prototype unless they can try it out themselves, which usually isn’t possible in a group.

 Out of Context

Focus groups bring people out of their usual context into an unnatural environment. We don’t get to see their usual behavior, the tasks they perform, their tools and technology, their environment, or the people they typically interact with.

Group Sessions Can be Useful

Because of these disadvantages with focus groups, we tend to overlook the value of any type of group research session. However, group research sessions can be valuable. Even when we are specifically requested to conduct focus groups, we don’t have to conduct traditional, marketing focus groups. We can modify them to make them more useful.

Why Conduct Group Research?

Group sessions provide some advantages over individual sessions:

  • You save time and money by meeting with many participants at once.
  • Group sessions are a useful way to quickly gather initial information about a topic, before going out to do individual user research sessions. For example, you could learn about a process at a high-level by having a group of employees each talk about their role in the process.
  • Interaction between participants can spark discussions that might never be revealed in individual interviews. For example, a group of employees from the same company may have a more productive discussion about their work than if you held individual interviews with each of them separately. Because they all know each other and have things in common, they may talk about things together that they would never bring up to you, as an outsider.
  • In group sessions, you can have participants work together in small groups and listen to their decision making and reasoning. For example, you can do group card sorting to hear how participants make decisions about how to group the cards.

Research Formats

In addition to group discussion, you can break up a group into smaller groups or give them individual activities.

The Entire Group

Discussing topics as a group takes advantage of the group dynamic to hear different perspectives, to get consensus, to debate, or to hear about different parts of a process.

Smaller Groups

Breaking into several smaller groups allows individuals to work together on activities that would be unwieldy to do with the entire group. As they work together, you can listen to and observe their decision making. You can then have each group present their results to the entire group to generate further discussion.

Individual Activities

Another option is to have the participants each perform an activity on their own and then present it to the group. This is effective with activities that require thought and are easier to do alone. For example, you could give the participants time to draw diagrams of their process, and then you could ask them to present their drawing to the entire group. You get the advantage of getting each person’s individual perspective but also with the advantage of group discussion.

Group Size

Any group activity becomes difficult to manage with too many participants. More than ten participants are usually too much. If there are more than ten people that need to be involved, it’s better to schedule multiple sessions.

Dyads and Triads

Market research uses terms like dyads and triads to describe group sessions with two and three participants. Groups of these sizes have different dynamics than a large group. They combine some of the dynamics of individual interviews with those of group sessions. In user experience, dyads (sessions with two participants) or triads (sessions with three participants) make sense when there’s a natural relationship between those people. For example, if you’re researching the car buying process, it might make sense to interview a husband and wife together. Or if three employees perform different parts of a related process, it may make sense to talk with them together to understand the process and how their roles relate.

Alternative Group Research Activities

In addition to group discussions, there are many creative and unique activities that you can use to gather information about the users and their tasks, and to get user input into the design. Instead of describing these techniques in detail, each of which could be its own article, I’ve provided links to read more about each technique.

Group Card Sorting

If organization of information and gathering common terminology is your goal, group card sorting is a quick way to gather that information. You can break up the group into smaller sub-groups of two or three participants each. They then work together to sort pieces of information on index cards or post-it notes into categories and sub-categories, and then name each category. For more information, see: http://www.usability.gov/how-to-and-tools/methods/card-sorting.html

Affinity Diagrams

Similar to card sorting, you can have sub-groups work together to group related items on post-it notes and then label those groups to create affinity diagrams. Afterwards, the entire group can discuss the themes that emerge. For more information, see: http://www.usabilitybok.org/affinity-diagram

Drawing or Modeling

Breaking the group into small groups or individuals and having them illustrate or build a model of their ideas, concepts, workflows, or processes is a great way to get insight into their thought processes and mental models. After each person presents their drawing or model to the group, discussion is facilitated by having these concrete items to talk about. For more information, see: http://uxmag.com/articles/creativity-based-research-the-process-of-co-designing-with-users and http://johnnyholland.org/2011/10/storyboarding-ux-part-3-storyboarding-as-a-workshop-activity/

Collaging

Individuals or groups can create collages from a set of provided images to express their feelings about a particular issue. The collages then can be described and discussed. For more information, see: http://www.smashingmagazine.com/2012/02/06/collaging-getting-answers-questions-you-dont-know-ask/

Priming Activities

You can give the participants assignments to do before the group session, such as writing about or taking photos of an experience, creating drawings, or creating collages. Because they do this outside the group session, it allows more time for presenting their work and discussing it as a group. It also brings in a little bit of each person’s real experience outside of the conference room. For more information, see: http://johnnyholland.org/2010/05/not-to-prime-is-a-crime/

Design Games

Several sources provide creative group activities to provide input to design problems. Gamestorming, a book by Dave Gray, Sunni Brown, and James Macanufo provides more than 80 group games to encourage creativity and generate new ideas. The site Design Games also provides a number of games with complete directions about how to play them.

Group Sessions as a First Step

Group sessions with users or stakeholders are an efficient way to get high-level information about subject matter, users, tasks, and processes. With participatory design and design games, you can get user input into the design. Ideally, group sessions shouldn’t replace individual observations and interviews with users, but they are a good first step to gather the initial information to understand the basics before going out to conduct field research.

 

 

Group photo courtesy of Kennisland on Flickr under Creative Commons license

What’s the Formula for Success in Recruiting User Research Participants?

If you believe that designing an effective user experience requires involving users throughout the design process, then you must admit that finding and recruiting participants for user research and usability testing is the most crucial step. After all, if you can’t get participants, you can’t do research or usability testing. A recent, difficult recruiting effort left me ruminating about what it is that makes recruiting either easy or difficult. Is there something like an equation that can account for the ease or difficulty of recruiting participants? Yes, there is: RE = ((K + S) x RM) x ((C+A) x PM) Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation) Okay, I just kind of made that up, but I think it’s a good way to visualize the various factors that influence how easy or difficult it will be to recruit participants. Let’s look at the components of the formula in more detail.

The Recruiter

The first part of the equation involves attributes of the recruiter: Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation) By recruiter, I mean whoever is doing the recruiting. It could be you, a recruiting company, your client, a salesperson, or someone else. Ideally, the researcher would also recruit the participants, because that person knows the most about who to recruit. But sometimes you have to rely on other people to do the recruiting, usually because they have a better connection to participants.

Knowledge

Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation) The first factor is the recruiter’s knowledge of the research and the type of people to recruit.

  • Does the recruiter know the characteristics of the people that are desired? Can the recruiter identify people who match those characteristics?
  • How well can the recruiter describe the research to the participants?

Without this knowledge, recruiters may select the wrong types of people or give them the wrong impression about the research. If you’re not doing the recruiting, it’s important to give the recruiter a clear definition of the types of people you’re looking for and a simple description of the research that they can use when contacting participants.

Skills

Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation)  Effective recruiting also depends on the recruiter’s interpersonal, communication, and organizational skills.

  • Is the recruiter personable and persuasive enough to convince people to participate?
  • Can the recruiter clearly and concisely communicate with the participants?
  • Is the researcher organized and coordinated enough to contact, schedule, and keep track of many participants?

Ensure that the person doing the recruiting has these skills. If not, have them delegate this task to someone who does. Administrative assistants often excel at recruiting because they are already well versed in communicating and coordinating schedules.

Recruiter Motivation

Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation) When finding participants becomes difficult, recruiter motivation is the most important criteria.

  • How much does the recruiter care about finding participants?
  • Will there be positive or negative consequences for the recruiter if participants are found or not found?

If you’re doing the research and the recruiting, you’re usually highly motivated to find participants, but when you have to rely on someone else, try to make sure that person has a stake in the recruiting. The negative consequences of not finding participants and derailing the project are highly motivating. Once a project starts and dates are set, there’s a ticking clock counting down the days to the research sessions. Delays in finding participants will delay the project.

With deadlines looming and money on the line, there’s a lot of pressure for whoever’s responsible to find participants. You can bet that most of the time, it gets done. Professional recruiting companies have a financial motivation to find participants, since they get paid a certain amount for each participant they recruit. However, if the recruiting turns out to be more difficult than they expected, they will begin to lose money as they spend more time finding each participant. When that happens, they’ll usually try to get you to loosen up the recruiting requirements to make it easier to find people, or they may say that they’ve exhausted their efforts and give up.

The Participants

The second part of the equation involves the recruiter’s relationship to the participants: Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation)

Connection

Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation) When people have a connection with the recruiter, they’re more likely to agree to participate. The strength of a connection can take various forms:

  • The participant may know the recruiter personally.
  • The participant may share a common affiliation with the recruiter. For example, they may be fellow employees or members of the same organization.
  • The participant may have an affiliation with an organization that the recruiter represents. For example, they may be a customer of the company the recruiter represents.

An email or phone call to participate in a study, often raises the questions about whether it’s legitimate, whether it’s worth considering, and whether it’s worth the time to participate. The closer the connection the person has with the recruiter, the more likely they will take the time to consider the request. Unless you already have this relationship, you might have someone else, who the potential participants know, do the recruiting. They’re more likely to listen to, consider, and respond to someone they know rather than a complete stranger.

Access

Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation) When you have access to lists of potential participants (such as employees, organization members, or customers) it’s much easier to recruit participants, especially when those lists contain characteristics that you can use to filter and narrow down those lists to the right types of people. It’s very difficult when you and your client have no access to the types of people you need. That’s when it makes sense to use a recruiting company.

Participant Motivation

Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation) Once they’ve been contacted and have considered the pitch, motivation determines whether a person will volunteer to participate. We can look at participant motivation as its own sub-equation: PM = A – D Participant Motivation = Advantages of participating – Disadvantages of participating To make it worth participating, the advantages have to be greater than the disadvantages. A > D = Participate A < D = Decline The advantages of participating can include things like:

  • Incentives – A $150 check, $50 Starbucks gift card, free software, etc.
  • Personal benefit from the improvements that come out of research – They’re going to improve this horrible software that I have to use every day at work!
  • Feelings of altruism
  • Feeling good about having your ideas heard and being able to improve something
  • Novelty of doing something different and interesting
  • Getting praise for participating and avoiding getting in trouble for not helping out

The disadvantages of participating can include things like:

  • The difficulty and effort it takes to participate
  • The time it takes to participate
  • Discomfort, anxiety, and fear of the unknown

Add it Up

Actually, something this complex can’t be distilled into an easy formula, but I think this equation provides a good visualization of the important factors. The quality and motivation of the recruiter, combined with the ease of finding participants, and the way people perceive the benefits they’ll receive versus the hassles they’ll have to go through, determines how easy or difficult it will be to recruit participants.

Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation)

 

Mathematics image courtesy of Tom Brown

Are Consent Forms Always Necessary?

Are consent forms always necessary? We’re told that consent forms are an indispensable part of ethical user research. Consent forms are the vehicle to give and get informed consent – they inform the participants of what the study will entail and they allow the participant to indicate consent – with a signature and date.A consent form

Yet consent forms can conflict with the informal, friendly rapport that we try to establish with participants. Anything you present for people to sign immediately looks like a legal document or liability waiver. It puts them on guard.

That’s ironic because consent forms are the opposite of legal waivers. Legal documents are created to protect the interests of the company that creates them, while consent forms are created to protect the rights of the people signing them. Yet most participants assume they are signing a typical legal waiver.

Consent forms seem acceptable in more formal user research situations, such as usability testing and focus groups, but they seem odd and even off-putting when used in more informal situations. I’ve found them to be especially awkward when doing field studies at people’s offices. You strive to set up an informal situation, such as asking someone to show you how they create reports or asking them to try out a new design for an expense report application. But when you show up with a consent form for them to sign, it shatters the informal, comfortable rapport you tried so hard to establish. I’ve had people react to consent forms in this kind of situation with, “Hey! I thought we were just talking here.” How many times in the course of your work-life have you had someone show up to a meeting with a legal document for you to sign?

So I say use your judgment. When a consent form feels like it would be overly formal, don’t use it (unless your legal department requires it). Instead, get informed consent informally by email. “Inform” with your email describing what will take place, and get “consent” from their reply email agreeing to participate. At the start of the session, you can inform them again with a summary of what you’ll be doing. They will then give consent by continuing to participate in the session.

A good guideline is how comfortable or uncomfortable you feel when giving participants the consent form. If you feel uncomfortable, you’re probably breaking a group norm. So you should find a more acceptable way of getting informed consent.

Two Rarely Used Research Methods

Observing in a public areaMy two most recent articles in UXmatters have been about two research techniques that are very common to anthropology and the social sciences but are rarely used in design research. Covert naturalistic observation and participant observation certainly require more work and time than we usually have in UX projects, but it’s worth taking a look at these two methods to see what we can adapt to design research.

Covert Naturalistic Observation
This type of study is known in psychology, anthropology, and other social sciences as covert naturalistic observation. It is the opposite of the techniques we typically use, which are forms of overt naturalistic observation. Being covert means observing behaviors in their natural contexts without any intervention or influence by the researcher and without participants knowing that they’re being observed.

Participatory Observation
Anthropologists and sociologists often practice participant observation, in which they join a group as a participating member to get a first-hand perspective of the group and their activities. Instead of observing as an outsider, they play two roles at once—objective observer and subjective participant…

UX Testing?!!

Old Man Usability

Old Man Usability

Okay, now wait just a goddamn minute! UX testing? U-X testing?!! Now that’s just going too far!

You think you’re all better than me and don’t need “usability” anymore? “User experience” is a more inclusive and descriptive term about the aspects we’re interested in these days. Yeah yeah, fine. It’s more than just usability. Okay, I get it.

But keep your damn UX hands off my usability testing!!! That’s my signature method. I invented that! Talk about kicking a man when he’s down.

What am I talking about, you say? I’ve begun to notice this disturbing trend of you UX creeps stealing my method and calling it “UX” testing. Just look at this recent article from those fancy-pants, “digital marketers” over at eConsultancy: A Case for UX Testing and Agile. And then I noticed this article from last year: UX Testing and Cultural Preferences. Even User Zoom has gotten into the act with this article: 17 Questions Answered About UX Testing and Agile. And it doesn’t stop there. I just Googled “ux testing” and got 28,300 results!

Usability testing has been providing more than just usability for a long time. So in some ways I see your point that perhaps the word “usability” only describes part of what this method provides insight into.

But usability testing is the one method that’s still primarily about usability. Put participants in a lab (or test them remotely), give them tasks to perform, observe their behavior, and ask them to tell you what they’re thinking – that’s usability testing. In addition to assessing usability, it can provide information about satisfaction, emotions, and opinions, but it doesn’t give you a true depiction of the user experience. Other UX research methods give you a better picture of the user experience by observing people in their natural contexts of use. You can test usability, but you can’t really test the user experience.

And what are these people who are doing “UX testing” really doing? You guessed it! Usability testing. It’s nothing different. Just a name change.

So, I agree that user experience makes sense, but that doesn’t mean you should do a global find and replace, turning every instance of “usability” into “user experience.”

So keep your damn hands off my usability testing! It will always be “usability testing” no matter what you want to call it.

By the way, Googling “usability testing” brings up 2,110,000 results. So there!

- Old Man Usability

How is usability testing like beer?

Glass of beer

Flickr: HeadCRasher

My manager asked me recently, “Do you think usability testing has become a commodity?” He was referring to the fact that in the last year or two we’ve seen clients go with cheaper usability testing companies. He was questioning whether clients have decided that there’s no difference between usability testing companies except price. Quality isn’t a differentiator to them anymore (if it ever was).

It does seem that there are companies out there doing usability testing at increasingly lower prices. How they cut corners to get their costs low enough to still make a profit amazes me, and it makes me wonder what kind of quality the clients receive.

Then an analogy dawned on me – usability testing, as a consulting service, is like beer. There are many people out there who are perfectly happy drinking cheap beer. It’s cheap, it’s bland, but it does the trick in the end – it gives you a buzz. But that doesn’t mean everyone is satisfied with cheap beer. There are still those out there who appreciate and will pay more for craft beer with quality, taste, and a better buzz.

If you want cheap usability testing, you can get it. It won’t taste good or be the best quality, but if all you’re looking for is a cheap buzz, it will do the trick in the end. On the other hand, if you have a sophisticated enough palette, you’ll be able to tell the difference between cheap usability testing and craft usability testing, and you’ll be more satisfied in the end. The bottom line is: you get what you pay for. There’s usability testing out there for all tastes and budgets.

Too Cool for Your Usability Test

CoolNo matter how well you recruit representative participants for a usability test and no matter how well you plan the testing, there are times when you’ll ask participants to perform a task that they might not normally perform themselves. It’s rare that every task you ask people to perform matches exactly what they would do. When this happens, most participants are agreeable enough to just “play along” for the purposes of the test.

Sure, it’s good to know what a participant would normally do instead of your planned task, but that’s more useful to learn during field studies. During a usability test, you usually just want to observe how well people can perform tasks.

During a recent usability test of an intranet design, I asked participants to browse the Blogs section to test out the usability of the filtering and searching functions common across the various sections of the intranet. I used the Blogs section as an example because that was the section we had built out in our prototype. Unfortunately, I came across two participants who were “too cool” to read blogs. In fact, they were too cool to even play along with my ridiculous and demeaning scenario.

It went something like this:

Me: Show me where you’d go if you wanted to see all the blogs in the company.

Joe Cool: Oh, I wouldn’t do that.

Me: Why?

Joe Cool: I don’t read the blogs.

Me: Why is that?

Joe Cool: Who cares about blogs? I don’t have time to read blogs.

Me: Okay, but if you did want to see all the blogs, where would you find them?

Joe Cool: I really wouldn’t do that. People here don’t really pay attention to the blogs. Who has time for that? We have enough to do with…

[Two minutes later]

Me: Okay, well that’s good to know, but just for the sake of this session, let’s say that you did want to read the blogs, where would you go to do that?

Joe Cool: [Sigh] Well, I guess I’d go here, and – here it is. But you see the problem with blogs is that…

[One minute rant later]

Me: Okay, what would you do if you wanted this to show you the most popular blog posts in the company?

Joe Cool: I don’t really care about what other people think is popular, especially from people who read blogs.

Me: Okay, but if someone else wanted to see the most popular blog posts in the company, what should they do here?

Joe Cool: Maybe they should ask someone else who reads blogs a lot? Or they should get a life and do something more productive.

Me: Okay, let’s move on to the next part…

Luckily, the next task was cool enough for him. Sometimes that’s all you can hope for.

What’s Wrong With Usability Anyway?

old man

Old Man Usability

 

Okay, fine. I get it. You don’t think that I, usability, am cool anymore, and you don’t want to be seen with me now. I’m the dorky, embarassing parent, and you want to hang out with your cool, “user experience” friends. That’s okay. It’s only a natural part of growing up, I guess.

 

Although I brought you into this field, gave you your first job, supported you, and brought you respect and recognition, I guess you’re ready to go out on your own now, and you need to establish your own identity. That’s understandable, but I must admit I was a little hurt when even my most loyal child, the Usability Professionals Association changed its name to the User Experience Professionals Association. Okay, actually that one hurt a lot.

Don’t get me wrong, I do admit that “user experience” makes sense. There’s more to what people experience than just usability. I realize that. But don’t ever forget that usability is still very important. In fact, I’m probably the most important of the elements that make up user experience. If something isn’t usable, then it can’t really be useful, desirable, or valuable can it?

In fact, most of what you and your friends do under the name “user experience” today is what we did back in my day, under the name “usability.” So I don’t really see the big difference.

I think I deserve a little respect, though. I spent many years making a name for myself and getting people to think about the needs of the user. The current popularity of user experience wouldn’t be possible without the trail I blazed first. At least people know my name, usability, and what it means. Try finding a consistent definition of user experience, ha!

So after all I’ve done for you, this is the thanks I get? People declare that usability is old, tired, boring, uncool, not innovative, and even claim that I’m dead? Just wait until you have offspring of your own. See how you feel when they move on from user experience to the next buzz word.

Already I can see it beginning. Everyone’s jumping on the bandwagon and calling themselves a user experience professional these days. The term user experience is getting too broadly defined and overexposed. I can feel the pendulum starting to swing back. At this point, I’m so uncool that I’m actually becoming cool again. Soon I’ll be able to say, “I’m back!” Just you wait and see!

Tattle Tale Participants

Have you ever come across a tattle-tale participant? See if this sounds familiar.

You conduct a user research session (interview, contextual inquiry, focus group, etc.), asking simple questions to get the participants to speak about the subject at a basic level of understanding. Although you may know some of the answers to the questions you’re asking, you ask the questions anyway to hear the answers from the participants’ perspective. Misunderstanding your intent, the participant is alarmed that you don’t know what you’re doing and tattles on you to the client, “It was clear from his questions that he didn’t know anything about our process/system/technology.”

For example, a few years ago I was doing research on a company’s use of SAP. We had a group of interns show us the HR tasks that they do in SAP. Our questions led them to conclude that we didn’t know anything about SAP (which was mostly correct). They tattled to their manager who contacted our main client, alarmed that we weren’t SAP experts. Fortunately, he reassured her that we were user experience experts, and we weren’t supposed to be SAP experts.

What turns a participant into a tattle tale? Usually, it comes from a misunderstanding of our role. As user experience consultants working on many different projects, we constantly have to learn about new organizations, new systems, new processes, new technologies, and new types of people. We interview business stakeholders and conduct user research to learn about these things, but our goal isn’t to become experts. In fact, it’s often better to not be an expert. We have the advantage of seeing a group, system, process, or set of tasks from an outside perspective. Because we don’t already have the same insider knowledge as the business stakeholders and participants,  we can get them to explain things to us as outsiders. To do this, we often need to ask basic questions and sometimes even act “dumb” to get participants to fully explain things that they would otherwise forget to explain or gloss over at a high level.

The worst thing about tattling is that it can make us afraid to ask questions for fear that they might expose our “ignorance.” So make it clear who the experts are – the users and the business stakeholders, and where your expertise lies – user experience. Combining the expertise of business stakeholders, users, and user experience professionals is the key to a successful project.

New Article: Capturing User Research

My latest article for UXmatters is about Capturing User Research. It discusses the pros and cons of various methods of capturing user research from handwritten notes, typing up notes on a laptop or tablet, having someone else take notes, recording the audio, recording with video, taking photos, logging, and simply relying on your memory. 

Read the entire article at UXmatters: Capturing User Research.

Follow

Get every new post delivered to your Inbox.