Skip to content

This One Goes to 11

I just published an article on UXmatters, 10 User Research Myths and Misconceptions. It addresses common misunderstandings about user research that I’ve encountered over the years.

Here’s a bonus outtake from the article, Myth 11…

Myth 11: Field Research Is Better Than Usability Testing

On the other end of the spectrum from those who don’t understand the difference between user research and usability testing, are the user research elitists who think up-front, generative user research methods are far superior to usability testing. In this view, field studies take researchers out of the lab to observe people in their natural environments performing their usual activities, while usability testing takes place in the sterile, artificial environment of a usability lab and asks people to perform a limited set of artificial tasks. Instead of learning about people and what they really do, usability testing provides the limited value of learning whether people can perform your artificial tasks.

The Truth: Both Field Research and Usability Testing Have Their Places

Field studies and usability testing are two different methods used for different, but equally important, purposes. Field studies provide information to inform design, while usability testing evaluates a design. You have to make interpretations and conclusions from the user research and apply that to a design. Even after very thorough user research, you’re never completely sure that what you’ve designed will work well for the users. Usability testing is the evaluation that either confirms your decisions or points you to refinements. Both user research and usability testing are important and necessary. There’s no reason we can’t appreciate the value of both methods.

Analysis Is Cool

Affinity diagram

Analyzing the data is the most interesting part of user research. That’s where you see the trends, spot insights, and make conclusions. It’s where all the work comes together and you get the answers to your questions.

Why, then, did I publish an article in UXmatters – Analysis Isn’t Cool? All too often I’ve realized that clients, management, and project stakeholders underestimate the analysis phase and just want to get to the answers. People like to say that they did user research, but they don’t like to spend the time to analyze the data. They like the deliverables, whether they read them or not, but they don’t want to spend a lot of time on the analysis to produce those deliverables.

In this article, I discuss what analysis involves, methods for individual and group analysis, and ways to speed up the analysis process.

 

Photo by Josh Evnin on Flickr

User Experience in Bulgaria

Me presenting at UXify

I’m happy to report that user experience is alive and well in Bulgaria. I just got back a week ago from attending and presenting at UXify 2015 in Sofia, Bulgaria. A very enthusiastic audience of over 300 people attended a day of presentations, followed by a day of in-depth workshops. UXify was the end of a month of user experience events – UX Month Sofia.

On Friday, June 19th, I presented User Research in the Wild, a presentation about visiting people in their natural environments to observe their tasks. In addition to being a how-to guide, I covered common problems you face in conducting field studies and how to solve them, and how to overcome obstacles to field research.

On Saturday, June 20th, I gave a workshop on Paper Prototyping to about 20 attendees. After an introduction to creating paper prototypes and how to test them, the participants divided into groups to create their own paper prototypes. We only had a limited time to create them (about 40 minutes), so I was very surprised by how detailed and creative their prototypes turned out in such a short period of time. Each group of two people then joined with another group to take turns conducting usability testing on their paper prototypes. Each group did four rounds of tests, switching roles, so that each person was able to experience the role of the facilitator, “the computer,” and the participant twice. The attendees really seemed to have a good time, with a lot of laughing and joking around.

Me presenting at the workshop

In between my own presentations, I attended several interesting presentations about dashboard design, gamification, socially responsible design, and ecommerce user experience.

It was a fun trip, and it was nice to meet a lot of similar people, in another culture, who are similarly committed and enthusiastic about user experience.

Better Group Research Sessions

Group working together

Many user experience professionals cringe when clients want to do focus groups. In UX we extol the value of user research with individuals, either in usability testing or field studies, where we observe and interview individuals in the context of their tasks. Focus groups seem like a flaky, unreliable marketing activity. Unfortunately, because of our disdain for focus groups we sometimes extend that disdain to any research activity that involves a group of people.

Why Don’t We Like Focus Groups?

In UX we highly value observing the behavior of individuals in their natural context. Focus groups provide the exact opposite. They reveal what people say in a group out of context of their typical environment.

What People Say

Focus groups usually gather feelings, attitudes, opinions, and preferences; but they can’t tell us about behavior. We can’t rely much on what people say they do, because so much of what we do is done automatically without much conscious thought. So it’s difficult for people to accurately talk about what they do, out of context. That’s why we like to observe people in the context of performing their tasks. It’s more accurate, and it’s easier for people to show us what they do.

 In a Group

Being in a group has its limitations:

  • Group dynamics often influence individuals to go along with the group instead of speaking their own minds. They often say what they think will sound good to the group, and they may not feel comfortable speaking honestly about certain topics.
  • A few outspoken people often dominate the discussion, while others remain more silent.
  • You get much less information from each person than you would in individual sessions.
  • Group sessions are not useful for evaluation, because it’s hard for people to evaluate a design or prototype unless they can try it out themselves, which usually isn’t possible in a group.

 Out of Context

Focus groups bring people out of their usual context into an unnatural environment. We don’t get to see their usual behavior, the tasks they perform, their tools and technology, their environment, or the people they typically interact with.

Group Sessions Can be Useful

Because of these disadvantages with focus groups, we tend to overlook the value of any type of group research session. However, group research sessions can be valuable. Even when we are specifically requested to conduct focus groups, we don’t have to conduct traditional, marketing focus groups. We can modify them to make them more useful.

Why Conduct Group Research?

Group sessions provide some advantages over individual sessions:

  • You save time and money by meeting with many participants at once.
  • Group sessions are a useful way to quickly gather initial information about a topic, before going out to do individual user research sessions. For example, you could learn about a process at a high-level by having a group of employees each talk about their role in the process.
  • Interaction between participants can spark discussions that might never be revealed in individual interviews. For example, a group of employees from the same company may have a more productive discussion about their work than if you held individual interviews with each of them separately. Because they all know each other and have things in common, they may talk about things together that they would never bring up to you, as an outsider.
  • In group sessions, you can have participants work together in small groups and listen to their decision making and reasoning. For example, you can do group card sorting to hear how participants make decisions about how to group the cards.

Research Formats

In addition to group discussion, you can break up a group into smaller groups or give them individual activities.

The Entire Group

Discussing topics as a group takes advantage of the group dynamic to hear different perspectives, to get consensus, to debate, or to hear about different parts of a process.

Smaller Groups

Breaking into several smaller groups allows individuals to work together on activities that would be unwieldy to do with the entire group. As they work together, you can listen to and observe their decision making. You can then have each group present their results to the entire group to generate further discussion.

Individual Activities

Another option is to have the participants each perform an activity on their own and then present it to the group. This is effective with activities that require thought and are easier to do alone. For example, you could give the participants time to draw diagrams of their process, and then you could ask them to present their drawing to the entire group. You get the advantage of getting each person’s individual perspective but also with the advantage of group discussion.

Group Size

Any group activity becomes difficult to manage with too many participants. More than ten participants are usually too much. If there are more than ten people that need to be involved, it’s better to schedule multiple sessions.

Dyads and Triads

Market research uses terms like dyads and triads to describe group sessions with two and three participants. Groups of these sizes have different dynamics than a large group. They combine some of the dynamics of individual interviews with those of group sessions. In user experience, dyads (sessions with two participants) or triads (sessions with three participants) make sense when there’s a natural relationship between those people. For example, if you’re researching the car buying process, it might make sense to interview a husband and wife together. Or if three employees perform different parts of a related process, it may make sense to talk with them together to understand the process and how their roles relate.

Alternative Group Research Activities

In addition to group discussions, there are many creative and unique activities that you can use to gather information about the users and their tasks, and to get user input into the design. Instead of describing these techniques in detail, each of which could be its own article, I’ve provided links to read more about each technique.

Group Card Sorting

If organization of information and gathering common terminology is your goal, group card sorting is a quick way to gather that information. You can break up the group into smaller sub-groups of two or three participants each. They then work together to sort pieces of information on index cards or post-it notes into categories and sub-categories, and then name each category. For more information, see: http://www.usability.gov/how-to-and-tools/methods/card-sorting.html

Affinity Diagrams

Similar to card sorting, you can have sub-groups work together to group related items on post-it notes and then label those groups to create affinity diagrams. Afterwards, the entire group can discuss the themes that emerge. For more information, see: http://www.usabilitybok.org/affinity-diagram

Drawing or Modeling

Breaking the group into small groups or individuals and having them illustrate or build a model of their ideas, concepts, workflows, or processes is a great way to get insight into their thought processes and mental models. After each person presents their drawing or model to the group, discussion is facilitated by having these concrete items to talk about. For more information, see: http://uxmag.com/articles/creativity-based-research-the-process-of-co-designing-with-users and http://johnnyholland.org/2011/10/storyboarding-ux-part-3-storyboarding-as-a-workshop-activity/

Collaging

Individuals or groups can create collages from a set of provided images to express their feelings about a particular issue. The collages then can be described and discussed. For more information, see: http://www.smashingmagazine.com/2012/02/06/collaging-getting-answers-questions-you-dont-know-ask/

Priming Activities

You can give the participants assignments to do before the group session, such as writing about or taking photos of an experience, creating drawings, or creating collages. Because they do this outside the group session, it allows more time for presenting their work and discussing it as a group. It also brings in a little bit of each person’s real experience outside of the conference room. For more information, see: http://johnnyholland.org/2010/05/not-to-prime-is-a-crime/

Design Games

Several sources provide creative group activities to provide input to design problems. Gamestorming, a book by Dave Gray, Sunni Brown, and James Macanufo provides more than 80 group games to encourage creativity and generate new ideas. The site Design Games also provides a number of games with complete directions about how to play them.

Group Sessions as a First Step

Group sessions with users or stakeholders are an efficient way to get high-level information about subject matter, users, tasks, and processes. With participatory design and design games, you can get user input into the design. Ideally, group sessions shouldn’t replace individual observations and interviews with users, but they are a good first step to gather the initial information to understand the basics before going out to conduct field research.

 

 

Group photo courtesy of Kennisland on Flickr under Creative Commons license

Two Rarely Used Research Methods

Observing in a public areaMy two most recent articles in UXmatters have been about two research techniques that are very common to anthropology and the social sciences but are rarely used in design research. Covert naturalistic observation and participant observation certainly require more work and time than we usually have in UX projects, but it’s worth taking a look at these two methods to see what we can adapt to design research.

Covert Naturalistic Observation
This type of study is known in psychology, anthropology, and other social sciences as covert naturalistic observation. It is the opposite of the techniques we typically use, which are forms of overt naturalistic observation. Being covert means observing behaviors in their natural contexts without any intervention or influence by the researcher and without participants knowing that they’re being observed.

Participatory Observation
Anthropologists and sociologists often practice participant observation, in which they join a group as a participating member to get a first-hand perspective of the group and their activities. Instead of observing as an outsider, they play two roles at once—objective observer and subjective participant…

User Experience to be Thankful For

The First Thanksgiving, painting by Jean Louis...

The First Thanksgiving, painting by Jean Louis Gerome Ferris (Photo credit: Wikipedia)

This Thanksgiving, before you carve the turkey, you may take a moment to think about the things you’re thankful for. But what about the things you’re thankful for in your work? After all, work takes up a large part of your life. If you’re a user experience geek like me, you may want to think about all the things that you’re thankful for in user experience. So let’s count our UX blessings:

The user experience of most products, websites, and applications is better than ever.

People are having a better experience than ever with most products, websites, and applications. Things are far from perfect, but we’ve made a big difference in improving the overall user experience over the last 20 years.

UX professionals are needed more than ever.

Although the overall user experience has definitely improved, it hasn’t improved enough that we’re not needed. There will always be a need for someone to focus on user research and designing the user experience. As we move from software and websites to mobile devices, wearable computing, and ubiquitous computing, there are many interesting challenges ahead.

We don’t have to do as much educating and selling UX as we used to.

More people than ever have a general idea of what user experience is and believe in its importance. They may not be able to define user experience, but they are primed to understand what it means with a little explanation. More people than ever know what usability (or at least “ease of us”) is. Even if they don’t know it by the term “user experience,” they can already sense it and value it. It’s easier to explain to people and we are less often challenged to prove its value.

There are more user experience jobs than ever before.

I don’t know this for sure, but it does seem like there are more user experience jobs than ever before, and it seems like that will only increase. There’s room in this field for generalists and specialists.

User experience is a very inclusive field.

User experience is still a very multidisciplined field. We welcome many different specialties and perspectives.

We don’t fight as much as we used to.

Compared to a few years ago, there doesn’t seem to be as much fighting about job titles, definitions, and which field or organization really represents the user experience.

User experience professionals are more connected than ever before.

We have more opportunities to connect with other UX professionals, either in-person or virtually, than ever before. The list of conferences and local events seems to continue to grow with more networking opportunities than ever before. With Twitter and LinkedIn, we can communicate and connect with people that we would never have met or heard from just five years ago.

There are more UX resources to learn from than ever before.

If you want to keep up to date and continue learning, there are more opportunities than ever before. There is a lot of great UX-related content out there, from web magazines, blogs, publications from UX organizations and conferences, and books. UX books have become shorter, more specialized, and aimed at experienced professionals.

We’ve never had better tools.

We’ve never had better tools for user research, prototyping, and design. Tools for remote usability testing, unmoderated testing, card sorting, tree testing, and eyetracking have expanded the type of research we can do. Prototyping and design tools make it easier than ever to create prototypes that we can evaluate with clients and test with users.

We do good work. We are part of a noble calling.

We’re lucky to be in a line of work where we solve interesting problems and help make people’s lives a little better. It may sound corny, but our work is a noble calling. We create better experiences for people. That may be as minor in the grand scheme of things as creating an easy, pleasant online shopping experience; or it may be as serious as preventing major safety errors that could endanger lives. Most of the time it’s somewhere in between, but it’s nice to know that to some degree we’re making the world a better place.

Effectively Communicating User Research Findings

I presented at UXPA 2013 today on Effectively Communicating User Research Findings.

This is the reason I’ve been way too busy to blog lately. I’ve been working non-stop on this presentation and also my UXmatters article published this week on Creating Better UX Research Videos: http://www.uxmatters.com/mt/archives/2013/07/creating-better-ux-research-videos.php

So check either of these out.

UX Testing?!!

Old Man Usability

Old Man Usability

Okay, now wait just a goddamn minute! UX testing? U-X testing?!! Now that’s just going too far!

You think you’re all better than me and don’t need “usability” anymore? “User experience” is a more inclusive and descriptive term about the aspects we’re interested in these days. Yeah yeah, fine. It’s more than just usability. Okay, I get it.

But keep your damn UX hands off my usability testing!!! That’s my signature method. I invented that! Talk about kicking a man when he’s down.

What am I talking about, you say? I’ve begun to notice this disturbing trend of you UX creeps stealing my method and calling it “UX” testing. Just look at this recent article from those fancy-pants, “digital marketers” over at eConsultancy: A Case for UX Testing and Agile. And then I noticed this article from last year: UX Testing and Cultural Preferences. Even User Zoom has gotten into the act with this article: 17 Questions Answered About UX Testing and Agile. And it doesn’t stop there. I just Googled “ux testing” and got 28,300 results!

Usability testing has been providing more than just usability for a long time. So in some ways I see your point that perhaps the word “usability” only describes part of what this method provides insight into.

But usability testing is the one method that’s still primarily about usability. Put participants in a lab (or test them remotely), give them tasks to perform, observe their behavior, and ask them to tell you what they’re thinking – that’s usability testing. In addition to assessing usability, it can provide information about satisfaction, emotions, and opinions, but it doesn’t give you a true depiction of the user experience. Other UX research methods give you a better picture of the user experience by observing people in their natural contexts of use. You can test usability, but you can’t really test the user experience.

And what are these people who are doing “UX testing” really doing? You guessed it! Usability testing. It’s nothing different. Just a name change.

So, I agree that user experience makes sense, but that doesn’t mean you should do a global find and replace, turning every instance of “usability” into “user experience.”

So keep your damn hands off my usability testing! It will always be “usability testing” no matter what you want to call it.

By the way, Googling “usability testing” brings up 2,110,000 results. So there!

– Old Man Usability

What’s Wrong With Usability Anyway?

old man

Old Man Usability

 

Okay, fine. I get it. You don’t think that I, usability, am cool anymore, and you don’t want to be seen with me now. I’m the dorky, embarassing parent, and you want to hang out with your cool, “user experience” friends. That’s okay. It’s only a natural part of growing up, I guess.

 

Although I brought you into this field, gave you your first job, supported you, and brought you respect and recognition, I guess you’re ready to go out on your own now, and you need to establish your own identity. That’s understandable, but I must admit I was a little hurt when even my most loyal child, the Usability Professionals Association changed its name to the User Experience Professionals Association. Okay, actually that one hurt a lot.

Don’t get me wrong, I do admit that “user experience” makes sense. There’s more to what people experience than just usability. I realize that. But don’t ever forget that usability is still very important. In fact, I’m probably the most important of the elements that make up user experience. If something isn’t usable, then it can’t really be useful, desirable, or valuable can it?

In fact, most of what you and your friends do under the name “user experience” today is what we did back in my day, under the name “usability.” So I don’t really see the big difference.

I think I deserve a little respect, though. I spent many years making a name for myself and getting people to think about the needs of the user. The current popularity of user experience wouldn’t be possible without the trail I blazed first. At least people know my name, usability, and what it means. Try finding a consistent definition of user experience, ha!

So after all I’ve done for you, this is the thanks I get? People declare that usability is old, tired, boring, uncool, not innovative, and even claim that I’m dead? Just wait until you have offspring of your own. See how you feel when they move on from user experience to the next buzz word.

Already I can see it beginning. Everyone’s jumping on the bandwagon and calling themselves a user experience professional these days. The term user experience is getting too broadly defined and overexposed. I can feel the pendulum starting to swing back. At this point, I’m so uncool that I’m actually becoming cool again. Soon I’ll be able to say, “I’m back!” Just you wait and see!