Skip to content

Two Rarely Used Research Methods

Observing in a public areaMy two most recent articles in UXmatters have been about two research techniques that are very common to anthropology and the social sciences but are rarely used in design research. Covert naturalistic observation and participant observation certainly require more work and time than we usually have in UX projects, but it’s worth taking a look at these two methods to see what we can adapt to design research.

Covert Naturalistic Observation
This type of study is known in psychology, anthropology, and other social sciences as covert naturalistic observation. It is the opposite of the techniques we typically use, which are forms of overt naturalistic observation. Being covert means observing behaviors in their natural contexts without any intervention or influence by the researcher and without participants knowing that they’re being observed.

Participatory Observation
Anthropologists and sociologists often practice participant observation, in which they join a group as a participating member to get a first-hand perspective of the group and their activities. Instead of observing as an outsider, they play two roles at once—objective observer and subjective participant…

User Experience to be Thankful For

The First Thanksgiving, painting by Jean Louis...

The First Thanksgiving, painting by Jean Louis Gerome Ferris (Photo credit: Wikipedia)

This Thanksgiving, before you carve the turkey, you may take a moment to think about the things you’re thankful for. But what about the things you’re thankful for in your work? After all, work takes up a large part of your life. If you’re a user experience geek like me, you may want to think about all the things that you’re thankful for in user experience. So let’s count our UX blessings:

The user experience of most products, websites, and applications is better than ever.

People are having a better experience than ever with most products, websites, and applications. Things are far from perfect, but we’ve made a big difference in improving the overall user experience over the last 20 years.

UX professionals are needed more than ever.

Although the overall user experience has definitely improved, it hasn’t improved enough that we’re not needed. There will always be a need for someone to focus on user research and designing the user experience. As we move from software and websites to mobile devices, wearable computing, and ubiquitous computing, there are many interesting challenges ahead.

We don’t have to do as much educating and selling UX as we used to.

More people than ever have a general idea of what user experience is and believe in its importance. They may not be able to define user experience, but they are primed to understand what it means with a little explanation. More people than ever know what usability (or at least “ease of us”) is. Even if they don’t know it by the term “user experience,” they can already sense it and value it. It’s easier to explain to people and we are less often challenged to prove its value.

There are more user experience jobs than ever before.

I don’t know this for sure, but it does seem like there are more user experience jobs than ever before, and it seems like that will only increase. There’s room in this field for generalists and specialists.

User experience is a very inclusive field.

User experience is still a very multidisciplined field. We welcome many different specialties and perspectives.

We don’t fight as much as we used to.

Compared to a few years ago, there doesn’t seem to be as much fighting about job titles, definitions, and which field or organization really represents the user experience.

User experience professionals are more connected than ever before.

We have more opportunities to connect with other UX professionals, either in-person or virtually, than ever before. The list of conferences and local events seems to continue to grow with more networking opportunities than ever before. With Twitter and LinkedIn, we can communicate and connect with people that we would never have met or heard from just five years ago.

There are more UX resources to learn from than ever before.

If you want to keep up to date and continue learning, there are more opportunities than ever before. There is a lot of great UX-related content out there, from web magazines, blogs, publications from UX organizations and conferences, and books. UX books have become shorter, more specialized, and aimed at experienced professionals.

We’ve never had better tools.

We’ve never had better tools for user research, prototyping, and design. Tools for remote usability testing, unmoderated testing, card sorting, tree testing, and eyetracking have expanded the type of research we can do. Prototyping and design tools make it easier than ever to create prototypes that we can evaluate with clients and test with users.

We do good work. We are part of a noble calling.

We’re lucky to be in a line of work where we solve interesting problems and help make people’s lives a little better. It may sound corny, but our work is a noble calling. We create better experiences for people. That may be as minor in the grand scheme of things as creating an easy, pleasant online shopping experience; or it may be as serious as preventing major safety errors that could endanger lives. Most of the time it’s somewhere in between, but it’s nice to know that to some degree we’re making the world a better place.

Effectively Communicating User Research Findings

I presented at UXPA 2013 today on Effectively Communicating User Research Findings.

This is the reason I’ve been way too busy to blog lately. I’ve been working non-stop on this presentation and also my UXmatters article published this week on Creating Better UX Research Videos: http://www.uxmatters.com/mt/archives/2013/07/creating-better-ux-research-videos.php

So check either of these out.

UX Testing?!!

Old Man Usability

Old Man Usability

Okay, now wait just a goddamn minute! UX testing? U-X testing?!! Now that’s just going too far!

You think you’re all better than me and don’t need “usability” anymore? “User experience” is a more inclusive and descriptive term about the aspects we’re interested in these days. Yeah yeah, fine. It’s more than just usability. Okay, I get it.

But keep your damn UX hands off my usability testing!!! That’s my signature method. I invented that! Talk about kicking a man when he’s down.

What am I talking about, you say? I’ve begun to notice this disturbing trend of you UX creeps stealing my method and calling it “UX” testing. Just look at this recent article from those fancy-pants, “digital marketers” over at eConsultancy: A Case for UX Testing and Agile. And then I noticed this article from last year: UX Testing and Cultural Preferences. Even User Zoom has gotten into the act with this article: 17 Questions Answered About UX Testing and Agile. And it doesn’t stop there. I just Googled “ux testing” and got 28,300 results!

Usability testing has been providing more than just usability for a long time. So in some ways I see your point that perhaps the word “usability” only describes part of what this method provides insight into.

But usability testing is the one method that’s still primarily about usability. Put participants in a lab (or test them remotely), give them tasks to perform, observe their behavior, and ask them to tell you what they’re thinking – that’s usability testing. In addition to assessing usability, it can provide information about satisfaction, emotions, and opinions, but it doesn’t give you a true depiction of the user experience. Other UX research methods give you a better picture of the user experience by observing people in their natural contexts of use. You can test usability, but you can’t really test the user experience.

And what are these people who are doing “UX testing” really doing? You guessed it! Usability testing. It’s nothing different. Just a name change.

So, I agree that user experience makes sense, but that doesn’t mean you should do a global find and replace, turning every instance of “usability” into “user experience.”

So keep your damn hands off my usability testing! It will always be “usability testing” no matter what you want to call it.

By the way, Googling “usability testing” brings up 2,110,000 results. So there!

– Old Man Usability

How is usability testing like beer?

Glass of beer

Flickr: HeadCRasher

My manager asked me recently, “Do you think usability testing has become a commodity?” He was referring to the fact that in the last year or two we’ve seen clients go with cheaper usability testing companies. He was questioning whether clients have decided that there’s no difference between usability testing companies except price. Quality isn’t a differentiator to them anymore (if it ever was).

It does seem that there are companies out there doing usability testing at increasingly lower prices. How they cut corners to get their costs low enough to still make a profit amazes me, and it makes me wonder what kind of quality the clients receive.

Then an analogy dawned on me – usability testing, as a consulting service, is like beer. There are many people out there who are perfectly happy drinking cheap beer. It’s cheap, it’s bland, but it does the trick in the end – it gives you a buzz. But that doesn’t mean everyone is satisfied with cheap beer. There are still those out there who appreciate and will pay more for craft beer with quality, taste, and a better buzz.

If you want cheap usability testing, you can get it. It won’t taste good or be the best quality, but if all you’re looking for is a cheap buzz, it will do the trick in the end. On the other hand, if you have a sophisticated enough palette, you’ll be able to tell the difference between cheap usability testing and craft usability testing, and you’ll be more satisfied in the end. The bottom line is: you get what you pay for. There’s usability testing out there for all tastes and budgets.

What it’s like to receive user research findings

As a user researcher, I often deliver bad news to clients – a long list of problems with their application or product and an intimidating list of recommended changes. It’s easy for me to think that the client should immediately set about making all these recommended changes. And when some of those changes aren’t made, I shake my head and can’t understand why they just don’t get it.

It’s not often that I’ve put myself in the client’s shoes to think about what they’re going through when they get the news. I’ve never been in the position of being the owner of an application or product who has to listen to a long list of problems that need to be fixed. So how can I empathize with what the client goes through?

I am, however, a homeowner who bought an older house after getting a home inspector to inspect the house. His inspection gave us a long list of problems to fix. It was intimidating and depressing to see how much work would be needed even after the high cost of buying the house.

What prevented us from giving up on the house, was that he prioritized the list of problems to show which items were critical to fix before purchasing the house, what would need to be done within the next year, which things could wait a few years, and what were optional but recommended fixes. That made his recommended changes seem more doable. We knew what fixes it would be reasonable to ask the seller to make, which items to focus on first after buying the house, and what we could wait on.

With user research findings and recommendations, it’s important to give your audience a sense of the severity of the problems and the priority of what should be addressed first. Like a home buyer, few clients have the time and money to fix all the problems right away. Receiving a large list of problems and recommended fixes can lead to a defeated feeling and a desire to ignore the problem or just give up. Instead, give them a sense that the problems are manageable and that it’s possible to focus on fixing a few items first and then gradually address additional problems over time.

A UX Researcher’s New Year’s Resolutions

New Year

Flickr: RLHyde

Try new things

In 2013, I’ll try new techniques instead of relying on the same routine research activities. For each project, I’ll step back and think about what research activities make the most sense based on the situation. Trying new things and inventing new techniques keeps things interesting.

Work faster

This year, I’ll do some things faster, to be more agile and lean where possible so that research continues to be included in projects. We’ve done a good job of selling clients and project team members on the value of including user research in projects. The remaining hurdle is that it often takes longer than they would like. There definitely are areas that can be sped up.

Work slower

In 2013, I’m going to use the time I save on working faster to spend more time on the activities that provide the most value. Some things shouldn’t be rushed. Analysis of research data, for example, is the most important, but least understood part of user research. No one ever seems to understand what analysis involves, how long it can take, and how important it is. In 2013, I’m going to fight for the time needed for analysis and do a better job explaining what it involves, why it takes so long, and why it’s so important to give it the time needed.

Get better participants

In 2013, instead of aiming high and settling for what we’re able to recruit, I’m going to create better screeners and spend more time making sure that the people who have been recruited, match the type of people we want to get. I’m going to be especially careful when clients are doing the recruiting of their own customers, employees, or members. I’ll give them better instructions in findings and recruiting people, and I’ll evaluate the types of people that they’ve scheduled.

Publish and present

In 2013, I’ll continue to publish articles in UXmatters and elsewhere. I’ll try to present at a conference. Publishing and presenting are great ways to share your knowledge with others in the field. For more tips on publishing and presenting, see my series of articles on Publishing and Presenting in UXmatters.

Attend more UX events

In 2013, I’ll attend more local UX events. In Philadelphia, we have a very active CHI group, PhillyCHI, but I always find it difficult to get motivated to go out after a long work day and attend their events. Every time I do attend, however, I find that it’s a great way to meet others in the field, and I always learn something useful.

Read more

In 2013, I’d like to read more UX-related books and articles. That’s easier said than done when you’re really busy at work. And after a day practicing UX, it’s difficult to get motivated to read UX in your spare time. Usually, I want to read anything else. Fortunately, a great trend in UX books is towards shorter, more practical books that can be read quickly (for example, the Rosenfeld Media books).

Be thankful for what I have

It can feel good to complain and think about what could be better about your job, but I find that I often don’t think about how good I have it. I’m doing a job I enjoy, that’s challenging, and usually interesting. In 2013, I’d like to focus more on the positive and appreciate what I have. If I find that I have only complaints and nothing to be thankful for, I’ll know that it’s up to me to change things.

Those are my resolutions. I hope I can achieve most, if not all of them, in 2013.

Writing and Publishing is Hard Work

word

Ugh! It’s December, and I feel I have to write something. I haven’t published anything since November, and I feel like I should have at least one post a month. But it’s hard to write a blog. Finding the time to write something worthwhile is difficult. When you also publish elsewhere, as I do, it’s especially difficult. Your blog ends up as the recipient of whatever energy and ideas are left after publishing elsewhere.

Several recent developments have also made me realize how difficult it is to publish a Web magazine. These magazines are run by volunteers and are rarely money making ventures. They depend on volunteer writers and editors, who each have their own work and personal responsibilities competing with their side project – working on the Web magazine.

Last week, for example, Johnny Holland, one of the leading Web magazines focused on interaction design, announced that it was ending its run (at least for now). In November, Boxes and Arrows, perhaps the oldest UX Web magazine, announced that it was Not Dead Yet and asked for the next generation of volunteers to take over running the site.

There are still several great UX Web magazines out there. I highlight these in my latest article on UXmatters – Publishing and Presenting, Part 2: Publishing. If you’ve ever had an interest in writing or getting involved in publishing, I recommend reading my advice about the writing process and where and how to get published.

If you need motivation to start publishing or presenting, check out part one – Publishing and Presenting, Part 1: Yes, You Can! And look for part 3 coming out in the next few weeks, about presenting at conferences.

A Researcher Observing User Research

I had an interesting experience recently observing another user researcher conducting user research. I was overseeing a project with a more junior researcher who was actually doing the work. She was conducting contextual inquiries, and I was the secondary person on the team, simply observing her lead the session. The experience made me realize several things.

What you notice as an observer

It’s difficult to let someone else lead

Sitting back and simply observing instead of leading the session was a very odd experience for me. I’m used to taking the active role and leading the session. It was difficult to restrain myself from asking questions and taking over the session. It took me a while to relax and just let the session unfold. Once I did that, I realized that I noticed different things.

You notice different things

When you’re simply observing and you don’t have to facilitate, you’re able to observe more. Since you don’t have to focus on taking notes, assessing the information that you’re taking in, thinking of the next question to ask, and maintaining a positive rapport with the participant; you can focus more on the context in which the tasks are performed. Since I knew my colleague was recording the session and taking detailed notes on what the participant was doing and saying, I found myself taking more notes on the work environment, the participant’s desk, the things posted on the cubicle wall, and tools that the participant used.

You notice how well the research works

As an observer, you notice much more about how well the primary researcher facilitates the session, you see how the participant reacts to the situation, and you pick up tips for your own facilitating. You can sit back and focus on the interaction between the researcher and participant with a critical eye.

What is the role of a secondary researcher?

What can we learn from this, and what’s the role of a secondary researcher on field research? The secondary researcher:

  • Should stay out of the way and let the primary researcher lead the session
  • Can note additional questions and ask them at appropriate intervals, without throwing off the direction of the session
  • Can focus on the environment and contextual cues, taking detailed notes and asking specific questions about these elements
  • Can be the note-taker who takes detailed notes, when there won’t be enough time to listen to the recordings later
  • Can evaluate how well the primary researcher facilitates the session, using that knowledge to provide constructive feedback later and also to improve his/her own skills in facilitation

Two is the magic number of people who should attend field studies. One is the primary researcher. The other can be another researcher, but it’s often better to include a designer to allow him/her to see the research first hand. So in most cases, a second researcher is not the best person to attend research. Although when a second researcher does attend, use the techniques I’ve discussed in this post to make the most of the experience.

Amateur Job Interviewers

Flickr: joelogon

Because asking questions is one of the main things I do as a user researcher, it often amazes me that I’m not better at interviewing job candidates. But a job interview is very different from a user interview. In a job interview, you’re trying to assess the qualifications and suitability of a person who’s actively trying to impress you and may not be telling you the full truth. You’re constantly making judgments. That’s completely different than a user interview, in which you try to remain impartial and neither you nor the interviewee has a vested interest in the outcome, besides trying to understand the person and the tasks he or she performs.

I’m an amateur job interviewer. Expert interviewers, such as the hiring manager and the recruiter or HR person, have the most at stake in interviewing the candidates. Amateur interviewers are usually potential co-workers who are there to assess the qualifications of the candidate and how well the person would fit in with the company.

Most interviewing advice focuses on how to interview with experts, but they don’t give you much insight into the minds of the amateurs. Here are a few things to know about amateur job interviewers.

They’re busy and your interview may be a hassle

Amateur interviewers are often busy. They get a meeting invitation to do a job interview and they accept it without thinking. They may then forget about the interview until a few minutes beforehand. So it’s likely that your interview is an interruption in their workday, which may affect their mood. They may just want to get the interview over with.

You may interview with someone who has been called in at the last minute

Since people are busy, last minute cancellations are common. Project work usually takes precedence over job interviews. So the person you were scheduled to meet with may change to someone else at the last minute. That person who was called in at the last minute is probably even less prepared to interview with you.

They may not have read your resume

Amateur interviewers may take a quick look at your resume when they first get the meeting invitation, but they have probably forgotten the details by the time of the interview. Often, they simply glance over the resume just before or during the beginning of the interview. “Tell me a little about yourself,” is often the first question because they haven’t read your resume.

They don’t necessarily know how to evaluate you

Amateur interviewers often have no interview training. They’ve been on job interviews themselves, but they don’t always know what questions to ask or how to evaluate the responses. Other than trusting your gut opinion, it’s often very difficult to judge someone from just a 30 minute interview.

It’s safer not to make a decision

Since it’s difficult to judge people, it’s often safer to either reject a candidate or recommend additional interviews. Sure, there are clear times when you feel that someone would be a great fit for the job and times when you definitely feel that a candidate should be rejected. But very often, the decision isn’t that clear. In those cases, it may seem safer to turn a candidate down or recommend additional interviews.

They may not want to hire someone

Unlike the HR person or the hiring manager, amateur interviewers may not want to hire another person. They may see an additional employee as a threat. If you appear to be too good, they may feel threatened by you – that you will surpass them or compete for promotions and attention. The interviewer may not see the need for additional employees. You may be seen as someone who will take away their work.

They don’t know the big picture

HR people and hiring managers know about all the candidates that have applied. They’ve likely read the resumes and picked out the best people to interview. They know about the level of talent that their job posting has been attracting and can judge between the possible candidates for the job. Amateur interviewers rarely have all this information. So it’s difficult for them to judge the one or two people they’ve interviewed, because they can’t compare them to the other options.

They won’t take much time to look at samples

Work samples are helpful, but busy people have very little time to look at work samples outside of the interview itself. This is especially true for writing samples, which take more time to read and evaluate.

They may be uncomfortable asking difficult questions

Amateurs are often not comfortable asking tough questions. That may sound like a good thing, but just because a question isn’t asked, doesn’t mean that the question isn’t in the interviewer’s head. Often the biggest doubts about hiring someone seem impolite to ask, which doesn’t give the candidate the ability to address those issues.  For example, “You have no experience in this field, so how do you think you can succeed in this job?” is a great but difficult question to ask. If it’s not addressed, that doubt will still linger in the interviewer’s mind and may be the reason to recommend not hiring you.

How to make the most of an amateur interview

By knowing what an amateur interviewer is thinking, there are a few things you can do to make the most of one of these interviews:

  • Bring extra copies of your resume to hand out to interviewers who don’t have one.
  • Ask them for their name and what they do, since the person you interview with may not be the original person scheduled for the session.
  • Prepare a good summary about yourself when asked, “So, tell me about yourself.”
  • Present your best self, but don’t be too good. You don’t want the interviewer to feel that you’re a threat to their job. You want them to picture you as a friendly co-worker, not a ruthless competitor.
  • Ask the interviewer questions about him/herself, the job, and the company. Most people feel much more comfortable answering questions about themselves and the company rather than asking questions. Amateur interviewers are often the best sources of information about the job, the people who work there, and what it would be like to work for the company.
  • Even if you’ve already asked all your questions of previous interviewers, ask the same questions again with the person you’re currently interviewing with. You don’t seem prepared if you don’t have any questions to ask.
  • Conduct your own assessment of the amateur interviewer. Would you want to work with that person?
  • Bring work samples to show during the meeting. It gives you something to talk about, taking the burden off the interviewer. Also, it’s probably the only time the interviewer will have to look at them.
  • Make sure you address the important points about why you would be a good fit for the position, and proactively answer any possible doubts. The interviewer may be relieved that he or she doesn’t have to ask these uncomfortable questions, and you may relieve their doubts. It also shows that you have good judgment and gives the impression that you’re honest and realistic.
  • Thank them for taking the time out of their busy schedule to interview you.
  • Send a brief and informal thank you email after the interview.

And lastly, remember that once you get the job, someday you’ll end up as an amateur interviewer too. So remember what the situation was like on the other side of the interview table.

1 4 5 6 8