Skip to content

What’s the Formula for Success in Recruiting User Research Participants?

If you believe that designing an effective user experience requires involving users throughout the design process, then you must admit that finding and recruiting participants for user research and usability testing is the most crucial step. After all, if you can’t get participants, you can’t do research or usability testing. A recent, difficult recruiting effort left me ruminating about what it is that makes recruiting either easy or difficult. Is there something like an equation that can account for the ease or difficulty of recruiting participants? Yes, there is: RE = ((K + S) x RM) x ((C+A) x PM) Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation) Okay, I just kind of made that up, but I think it’s a good way to visualize the various factors that influence how easy or difficult it will be to recruit participants. Let’s look at the components of the formula in more detail.

The Recruiter

The first part of the equation involves attributes of the recruiter: Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation) By recruiter, I mean whoever is doing the recruiting. It could be you, a recruiting company, your client, a salesperson, or someone else. Ideally, the researcher would also recruit the participants, because that person knows the most about who to recruit. But sometimes you have to rely on other people to do the recruiting, usually because they have a better connection to participants.

Knowledge

Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation) The first factor is the recruiter’s knowledge of the research and the type of people to recruit.

  • Does the recruiter know the characteristics of the people that are desired? Can the recruiter identify people who match those characteristics?
  • How well can the recruiter describe the research to the participants?

Without this knowledge, recruiters may select the wrong types of people or give them the wrong impression about the research. If you’re not doing the recruiting, it’s important to give the recruiter a clear definition of the types of people you’re looking for and a simple description of the research that they can use when contacting participants.

Skills

Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation)  Effective recruiting also depends on the recruiter’s interpersonal, communication, and organizational skills.

  • Is the recruiter personable and persuasive enough to convince people to participate?
  • Can the recruiter clearly and concisely communicate with the participants?
  • Is the researcher organized and coordinated enough to contact, schedule, and keep track of many participants?

Ensure that the person doing the recruiting has these skills. If not, have them delegate this task to someone who does. Administrative assistants often excel at recruiting because they are already well versed in communicating and coordinating schedules.

Recruiter Motivation

Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation) When finding participants becomes difficult, recruiter motivation is the most important criteria.

  • How much does the recruiter care about finding participants?
  • Will there be positive or negative consequences for the recruiter if participants are found or not found?

If you’re doing the research and the recruiting, you’re usually highly motivated to find participants, but when you have to rely on someone else, try to make sure that person has a stake in the recruiting. The negative consequences of not finding participants and derailing the project are highly motivating. Once a project starts and dates are set, there’s a ticking clock counting down the days to the research sessions. Delays in finding participants will delay the project.

With deadlines looming and money on the line, there’s a lot of pressure for whoever’s responsible to find participants. You can bet that most of the time, it gets done. Professional recruiting companies have a financial motivation to find participants, since they get paid a certain amount for each participant they recruit. However, if the recruiting turns out to be more difficult than they expected, they will begin to lose money as they spend more time finding each participant. When that happens, they’ll usually try to get you to loosen up the recruiting requirements to make it easier to find people, or they may say that they’ve exhausted their efforts and give up.

The Participants

The second part of the equation involves the recruiter’s relationship to the participants: Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation)

Connection

Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation) When people have a connection with the recruiter, they’re more likely to agree to participate. The strength of a connection can take various forms:

  • The participant may know the recruiter personally.
  • The participant may share a common affiliation with the recruiter. For example, they may be fellow employees or members of the same organization.
  • The participant may have an affiliation with an organization that the recruiter represents. For example, they may be a customer of the company the recruiter represents.

An email or phone call to participate in a study, often raises the questions about whether it’s legitimate, whether it’s worth considering, and whether it’s worth the time to participate. The closer the connection the person has with the recruiter, the more likely they will take the time to consider the request. Unless you already have this relationship, you might have someone else, who the potential participants know, do the recruiting. They’re more likely to listen to, consider, and respond to someone they know rather than a complete stranger.

Access

Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation) When you have access to lists of potential participants (such as employees, organization members, or customers) it’s much easier to recruit participants, especially when those lists contain characteristics that you can use to filter and narrow down those lists to the right types of people. It’s very difficult when you and your client have no access to the types of people you need. That’s when it makes sense to use a recruiting company.

Participant Motivation

Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation) Once they’ve been contacted and have considered the pitch, motivation determines whether a person will volunteer to participate. We can look at participant motivation as its own sub-equation: PM = A – D Participant Motivation = Advantages of participating – Disadvantages of participating To make it worth participating, the advantages have to be greater than the disadvantages. A > D = Participate A < D = Decline The advantages of participating can include things like:

  • Incentives – A $150 check, $50 Starbucks gift card, free software, etc.
  • Personal benefit from the improvements that come out of research – They’re going to improve this horrible software that I have to use every day at work!
  • Feelings of altruism
  • Feeling good about having your ideas heard and being able to improve something
  • Novelty of doing something different and interesting
  • Getting praise for participating and avoiding getting in trouble for not helping out

The disadvantages of participating can include things like:

  • The difficulty and effort it takes to participate
  • The time it takes to participate
  • Discomfort, anxiety, and fear of the unknown

Add it Up

Actually, something this complex can’t be distilled into an easy formula, but I think this equation provides a good visualization of the important factors. The quality and motivation of the recruiter, combined with the ease of finding participants, and the way people perceive the benefits they’ll receive versus the hassles they’ll have to go through, determines how easy or difficult it will be to recruit participants.

Recruiting Ease = ((Knowledge + Skills) x Recruiter Motivation) x ((Connection + Access) x Participant Motivation)

 

Mathematics image courtesy of Tom Brown

Are Consent Forms Always Necessary?

Are consent forms always necessary? We’re told that consent forms are an indispensable part of ethical user research. Consent forms are the vehicle to give and get informed consent – they inform the participants of what the study will entail and they allow the participant to indicate consent – with a signature and date.A consent form

Yet consent forms can conflict with the informal, friendly rapport that we try to establish with participants. Anything you present for people to sign immediately looks like a legal document or liability waiver. It puts them on guard.

That’s ironic because consent forms are the opposite of legal waivers. Legal documents are created to protect the interests of the company that creates them, while consent forms are created to protect the rights of the people signing them. Yet most participants assume they are signing a typical legal waiver.

Consent forms seem acceptable in more formal user research situations, such as usability testing and focus groups, but they seem odd and even off-putting when used in more informal situations. I’ve found them to be especially awkward when doing field studies at people’s offices. You strive to set up an informal situation, such as asking someone to show you how they create reports or asking them to try out a new design for an expense report application. But when you show up with a consent form for them to sign, it shatters the informal, comfortable rapport you tried so hard to establish. I’ve had people react to consent forms in this kind of situation with, “Hey! I thought we were just talking here.” How many times in the course of your work-life have you had someone show up to a meeting with a legal document for you to sign?

So I say use your judgment. When a consent form feels like it would be overly formal, don’t use it (unless your legal department requires it). Instead, get informed consent informally by email. “Inform” with your email describing what will take place, and get “consent” from their reply email agreeing to participate. At the start of the session, you can inform them again with a summary of what you’ll be doing. They will then give consent by continuing to participate in the session.

A good guideline is how comfortable or uncomfortable you feel when giving participants the consent form. If you feel uncomfortable, you’re probably breaking a group norm. So you should find a more acceptable way of getting informed consent.

Two Rarely Used Research Methods

Observing in a public areaMy two most recent articles in UXmatters have been about two research techniques that are very common to anthropology and the social sciences but are rarely used in design research. Covert naturalistic observation and participant observation certainly require more work and time than we usually have in UX projects, but it’s worth taking a look at these two methods to see what we can adapt to design research.

Covert Naturalistic Observation
This type of study is known in psychology, anthropology, and other social sciences as covert naturalistic observation. It is the opposite of the techniques we typically use, which are forms of overt naturalistic observation. Being covert means observing behaviors in their natural contexts without any intervention or influence by the researcher and without participants knowing that they’re being observed.

Participatory Observation
Anthropologists and sociologists often practice participant observation, in which they join a group as a participating member to get a first-hand perspective of the group and their activities. Instead of observing as an outsider, they play two roles at once—objective observer and subjective participant…

User Experience to be Thankful For

The First Thanksgiving, painting by Jean Louis...

The First Thanksgiving, painting by Jean Louis Gerome Ferris (Photo credit: Wikipedia)

This Thanksgiving, before you carve the turkey, you may take a moment to think about the things you’re thankful for. But what about the things you’re thankful for in your work? After all, work takes up a large part of your life. If you’re a user experience geek like me, you may want to think about all the things that you’re thankful for in user experience. So let’s count our UX blessings:

The user experience of most products, websites, and applications is better than ever.

People are having a better experience than ever with most products, websites, and applications. Things are far from perfect, but we’ve made a big difference in improving the overall user experience over the last 20 years.

UX professionals are needed more than ever.

Although the overall user experience has definitely improved, it hasn’t improved enough that we’re not needed. There will always be a need for someone to focus on user research and designing the user experience. As we move from software and websites to mobile devices, wearable computing, and ubiquitous computing, there are many interesting challenges ahead.

We don’t have to do as much educating and selling UX as we used to.

More people than ever have a general idea of what user experience is and believe in its importance. They may not be able to define user experience, but they are primed to understand what it means with a little explanation. More people than ever know what usability (or at least “ease of us”) is. Even if they don’t know it by the term “user experience,” they can already sense it and value it. It’s easier to explain to people and we are less often challenged to prove its value.

There are more user experience jobs than ever before.

I don’t know this for sure, but it does seem like there are more user experience jobs than ever before, and it seems like that will only increase. There’s room in this field for generalists and specialists.

User experience is a very inclusive field.

User experience is still a very multidisciplined field. We welcome many different specialties and perspectives.

We don’t fight as much as we used to.

Compared to a few years ago, there doesn’t seem to be as much fighting about job titles, definitions, and which field or organization really represents the user experience.

User experience professionals are more connected than ever before.

We have more opportunities to connect with other UX professionals, either in-person or virtually, than ever before. The list of conferences and local events seems to continue to grow with more networking opportunities than ever before. With Twitter and LinkedIn, we can communicate and connect with people that we would never have met or heard from just five years ago.

There are more UX resources to learn from than ever before.

If you want to keep up to date and continue learning, there are more opportunities than ever before. There is a lot of great UX-related content out there, from web magazines, blogs, publications from UX organizations and conferences, and books. UX books have become shorter, more specialized, and aimed at experienced professionals.

We’ve never had better tools.

We’ve never had better tools for user research, prototyping, and design. Tools for remote usability testing, unmoderated testing, card sorting, tree testing, and eyetracking have expanded the type of research we can do. Prototyping and design tools make it easier than ever to create prototypes that we can evaluate with clients and test with users.

We do good work. We are part of a noble calling.

We’re lucky to be in a line of work where we solve interesting problems and help make people’s lives a little better. It may sound corny, but our work is a noble calling. We create better experiences for people. That may be as minor in the grand scheme of things as creating an easy, pleasant online shopping experience; or it may be as serious as preventing major safety errors that could endanger lives. Most of the time it’s somewhere in between, but it’s nice to know that to some degree we’re making the world a better place.

Effectively Communicating User Research Findings

I presented at UXPA 2013 today on Effectively Communicating User Research Findings.

This is the reason I’ve been way too busy to blog lately. I’ve been working non-stop on this presentation and also my UXmatters article published this week on Creating Better UX Research Videos: http://www.uxmatters.com/mt/archives/2013/07/creating-better-ux-research-videos.php

So check either of these out.

UX Testing?!!

Old Man Usability

Old Man Usability

Okay, now wait just a goddamn minute! UX testing? U-X testing?!! Now that’s just going too far!

You think you’re all better than me and don’t need “usability” anymore? “User experience” is a more inclusive and descriptive term about the aspects we’re interested in these days. Yeah yeah, fine. It’s more than just usability. Okay, I get it.

But keep your damn UX hands off my usability testing!!! That’s my signature method. I invented that! Talk about kicking a man when he’s down.

What am I talking about, you say? I’ve begun to notice this disturbing trend of you UX creeps stealing my method and calling it “UX” testing. Just look at this recent article from those fancy-pants, “digital marketers” over at eConsultancy: A Case for UX Testing and Agile. And then I noticed this article from last year: UX Testing and Cultural Preferences. Even User Zoom has gotten into the act with this article: 17 Questions Answered About UX Testing and Agile. And it doesn’t stop there. I just Googled “ux testing” and got 28,300 results!

Usability testing has been providing more than just usability for a long time. So in some ways I see your point that perhaps the word “usability” only describes part of what this method provides insight into.

But usability testing is the one method that’s still primarily about usability. Put participants in a lab (or test them remotely), give them tasks to perform, observe their behavior, and ask them to tell you what they’re thinking – that’s usability testing. In addition to assessing usability, it can provide information about satisfaction, emotions, and opinions, but it doesn’t give you a true depiction of the user experience. Other UX research methods give you a better picture of the user experience by observing people in their natural contexts of use. You can test usability, but you can’t really test the user experience.

And what are these people who are doing “UX testing” really doing? You guessed it! Usability testing. It’s nothing different. Just a name change.

So, I agree that user experience makes sense, but that doesn’t mean you should do a global find and replace, turning every instance of “usability” into “user experience.”

So keep your damn hands off my usability testing! It will always be “usability testing” no matter what you want to call it.

By the way, Googling “usability testing” brings up 2,110,000 results. So there!

- Old Man Usability

How is usability testing like beer?

Glass of beer

Flickr: HeadCRasher

My manager asked me recently, “Do you think usability testing has become a commodity?” He was referring to the fact that in the last year or two we’ve seen clients go with cheaper usability testing companies. He was questioning whether clients have decided that there’s no difference between usability testing companies except price. Quality isn’t a differentiator to them anymore (if it ever was).

It does seem that there are companies out there doing usability testing at increasingly lower prices. How they cut corners to get their costs low enough to still make a profit amazes me, and it makes me wonder what kind of quality the clients receive.

Then an analogy dawned on me – usability testing, as a consulting service, is like beer. There are many people out there who are perfectly happy drinking cheap beer. It’s cheap, it’s bland, but it does the trick in the end – it gives you a buzz. But that doesn’t mean everyone is satisfied with cheap beer. There are still those out there who appreciate and will pay more for craft beer with quality, taste, and a better buzz.

If you want cheap usability testing, you can get it. It won’t taste good or be the best quality, but if all you’re looking for is a cheap buzz, it will do the trick in the end. On the other hand, if you have a sophisticated enough palette, you’ll be able to tell the difference between cheap usability testing and craft usability testing, and you’ll be more satisfied in the end. The bottom line is: you get what you pay for. There’s usability testing out there for all tastes and budgets.

Quicken Extortion

Apparently Intuit has found a good way to get Quicken customers to upgrade to the latest version – extortion. Bloating the software with new features is no longer enough to get customers to upgrade, so instead they effectively disable the software by taking away critical features, such as the ability to download transactions and balances from your financial institutions.

I recently received the email shown below that my Quicken 2010 software was expiring and “critical services” will be discontinued. I can understand canceling support and help for an older piece of software, but downloading your financial information is a major part of using Quicken. It’s like a car company trying to get you to buy a new car by taking away your power steering after three years.

Planned obsolescence is pretty common, but it’s usually achieved these days by putting out a better product that makes the product you own pale in comparison. Quicken 2010 works just fine, and the 2013 version doesn’t seem to offer anything new and worthwhile. On Amazon, it gets an average of 2.8 stars out of 5, with most reviewers giving it one star.

If feature bloat no longer does the trick, how do they get people to upgrade? Extortion and monopolizing the personal finance software market. Maybe it’s time to use something else instead. Microsoft Money? Oh yeah, that’s gone now. Mint.com maybe? Sounds good, but guess who owns Mint now? Intuit.

Oh well, I guess all I can do is give in and upgrade to 2013. At least I’ll get a $20 off coupon and free shipping. How considerate!

Quicken2013

What it’s like to receive user research findings

As a user researcher, I often deliver bad news to clients – a long list of problems with their application or product and an intimidating list of recommended changes. It’s easy for me to think that the client should immediately set about making all these recommended changes. And when some of those changes aren’t made, I shake my head and can’t understand why they just don’t get it.

It’s not often that I’ve put myself in the client’s shoes to think about what they’re going through when they get the news. I’ve never been in the position of being the owner of an application or product who has to listen to a long list of problems that need to be fixed. So how can I empathize with what the client goes through?

I am, however, a homeowner who bought an older house after getting a home inspector to inspect the house. His inspection gave us a long list of problems to fix. It was intimidating and depressing to see how much work would be needed even after the high cost of buying the house.

What prevented us from giving up on the house, was that he prioritized the list of problems to show which items were critical to fix before purchasing the house, what would need to be done within the next year, which things could wait a few years, and what were optional but recommended fixes. That made his recommended changes seem more doable. We knew what fixes it would be reasonable to ask the seller to make, which items to focus on first after buying the house, and what we could wait on.

With user research findings and recommendations, it’s important to give your audience a sense of the severity of the problems and the priority of what should be addressed first. Like a home buyer, few clients have the time and money to fix all the problems right away. Receiving a large list of problems and recommended fixes can lead to a defeated feeling and a desire to ignore the problem or just give up. Instead, give them a sense that the problems are manageable and that it’s possible to focus on fixing a few items first and then gradually address additional problems over time.

A UX Researcher’s New Year’s Resolutions

New Year

Flickr: RLHyde

Try new things

In 2013, I’ll try new techniques instead of relying on the same routine research activities. For each project, I’ll step back and think about what research activities make the most sense based on the situation. Trying new things and inventing new techniques keeps things interesting.

Work faster

This year, I’ll do some things faster, to be more agile and lean where possible so that research continues to be included in projects. We’ve done a good job of selling clients and project team members on the value of including user research in projects. The remaining hurdle is that it often takes longer than they would like. There definitely are areas that can be sped up.

Work slower

In 2013, I’m going to use the time I save on working faster to spend more time on the activities that provide the most value. Some things shouldn’t be rushed. Analysis of research data, for example, is the most important, but least understood part of user research. No one ever seems to understand what analysis involves, how long it can take, and how important it is. In 2013, I’m going to fight for the time needed for analysis and do a better job explaining what it involves, why it takes so long, and why it’s so important to give it the time needed.

Get better participants

In 2013, instead of aiming high and settling for what we’re able to recruit, I’m going to create better screeners and spend more time making sure that the people who have been recruited, match the type of people we want to get. I’m going to be especially careful when clients are doing the recruiting of their own customers, employees, or members. I’ll give them better instructions in findings and recruiting people, and I’ll evaluate the types of people that they’ve scheduled.

Publish and present

In 2013, I’ll continue to publish articles in UXmatters and elsewhere. I’ll try to present at a conference. Publishing and presenting are great ways to share your knowledge with others in the field. For more tips on publishing and presenting, see my series of articles on Publishing and Presenting in UXmatters.

Attend more UX events

In 2013, I’ll attend more local UX events. In Philadelphia, we have a very active CHI group, PhillyCHI, but I always find it difficult to get motivated to go out after a long work day and attend their events. Every time I do attend, however, I find that it’s a great way to meet others in the field, and I always learn something useful.

Read more

In 2013, I’d like to read more UX-related books and articles. That’s easier said than done when you’re really busy at work. And after a day practicing UX, it’s difficult to get motivated to read UX in your spare time. Usually, I want to read anything else. Fortunately, a great trend in UX books is towards shorter, more practical books that can be read quickly (for example, the Rosenfeld Media books).

Be thankful for what I have

It can feel good to complain and think about what could be better about your job, but I find that I often don’t think about how good I have it. I’m doing a job I enjoy, that’s challenging, and usually interesting. In 2013, I’d like to focus more on the positive and appreciate what I have. If I find that I have only complaints and nothing to be thankful for, I’ll know that it’s up to me to change things.

Those are my resolutions. I hope I can achieve most, if not all of them, in 2013.

Follow

Get every new post delivered to your Inbox.