When experience matters (and when it doesn’t)

Leisa Reichelt3 Comments

A little while ago I wrote this, which turned out to be more controversial than I intended.

I wrote it after having several encounters, in close succession, where one of these had happened:

  1. Recruiting scenarios where people assumed that previous domain experience was more important than the overall research experience, and capability, of the researcher.
  2. Working with a researcher who had domain experience on a particular topic who kept shutting down team conversations based on experience they’d had in previous projects, which the rest of the team had not been involved in.

These are both anti patterns of domain experience in user researcher and should be avoided.

Recruiting user researchers

When you’re recruiting for a user researcher, there are three things that teams should be looking for:

  1. Firstly, ensure that the user researcher has experience in the methods of user research you expect you’ll be using on the project. For example, don’t hire someone who has a decade of experience doing usability testing when you need someone who is strong in contextual research to work on the discovery phase of your project.
  2. Secondly, be confident that the user researcher cares about the quality of the service that their team ultimately delivers, will be able to hold their own in the team and will be effective in communicating research findings in a way that compels the team to act on them. This is the hardest criteria to meet. Too many user researchers have had a great time doing research and making reports, then leaving the teams to do with it what they will. You want a user researcher who wants to have ‘skin in the game’, and wants to see their research valued and used in making service design decisions and in setting priorities for the team.
  3. Finally, if the researcher has previous experience in the subject domain of the project, this can be an advantage. But only once the first two criteria have been met. These first two criteria are much more important for user researchers than subject matter domain expertise, and they’re also pretty hard to find.

When you hire a user researcher, the domain they should be most expert in is user research.

In teams, diversity is a strength

For most projects, having some subject matter expertise is essential. In most cases, that would be at best inefficient and at worst dangerous to do otherwise.

What is equally problematic, though, is a team full of people who have extensive experience working on the problem space they are just about to tackle.

Teams like this have so many shared mental models and assumptions about how things work, what things mean, where the constraints are, and how people think and work, and, despite their experience, not all of these things are right.

The CivicPatterns website calls this the Clean Room pattern and says:

If your project operates within any bureaucratic system, ensure that the person responsible for its design knows as little as possible about how the existing system works.

Most people hate dealing with bureaucracies. You have to jump through lots of seemingly pointless hoops, just for the sake of the system. But the more you’re exposed to it, the more sense it starts to make, and the harder it is to see things through a beginner’s eyes. Therefore, when building a system that helps someone bypass bureaucracy, start by designing how the system should be, with as little pre-knowledge as possible, and then, when you need to add any complexity, work as hard as you can to hide that from the user.

Lack of diversity in experience-levels (and lack of diversity in general) in the team will reduce their ability to consider a full range of service design options that can streamline the experience for users. This will limit the potential for transformation.

There are some roles where experience the domain of the project is essential and teams would be foolish not to include them. Designers and user researchers are not those roles.

Design your teams so they have diversity, in gender, in age, in backgrounds and in subject matter expertise and you’ll design better experiences for your users.

Anatomy of a good sticky note

Leisa Reichelt

Originally published on the GDS User Research Blog

For many of our teams, the ‘unit of data’ in qualitative user research is the sticky note – or, more precisely, units of information captured onto sticky notes. People often think this is amusing, given we’re always talking about digital by default.

Using sticky notes to analyse research helps us take a fast yet thoughtful and collaborative approach to analysis, and in a way that no digital tool I’ve come across allows. There’s just something about moving bits of paper around…

As a result of making and analysing so many sticky notes, we’ve become a bit opinionated about what a ‘good’ user research data sticky note looks like.

There are good, bad and downright useless ways of recording data onto sticky notes.

Here’s what we’ve found works best:

  1. Capture verbatim quotes or direct observations. Don’t make analysis notes on the sticky note. Capture exactly what you’ve seen or heard the participant say. Let the sticky note become the voice of the user.
  2. One data point per sticky note. No bullet points. During analysis, you need to be able to independently move quotes and observations around, so don’t group them on one note.
  3. Code sticky notes with a participant ID. Mark each sticky note with a code that will help you identify which participant said or did the thing you noted. An easy way to do this is to number participants in the order you see them. Our sticky note example shows a number 5 – this note refers to the 5th participant in the study.
  4. Use colour thoughtfully. Colour coding can be very useful when you’re doing analysis, so think about how to use it meaningfully. We capture all observations on one colour sticky note, then use a different colour to indicate findings and another colour for actions. Some people like to use different colour sticky notes to indicate things that went well and things that were problematic. Other people use colour to indicate different types of participants, for example, ‘confident online’ and ‘not confident online’.
  5. Thick pens. Match the thickness of your pen to the size of your sticky note. Never use a biro. We usually use 3×5 inch sticky notes and a pen with 0.5mm width.
  6. Encourage observers to write in capital letters. Although it’s slower, it makes everyone’s handwriting more legible, which is important when you’re doing analysis.
  7. Decent quality sticky notes. There’s nothing more annoying than having your sticky notes rain from the walls while you’re trying to do analysis. When it comes to sticky notes, the glue really matters and you get what you pay for.

Follow these guidelines, and you’ll be set for a great research analysis session.

Sample size and confidence: how to get your team to trust qualitative research

Leisa Reichelt

Originally published on the GDS User Research Blog

If you’re doing qualitative design research, don’t worry about sample size. Sample size and statistical significance don’t matter*.

The only thing that matters is how confident your team is about the next decision they need to make** 

You might get that confidence from watching 2 people.

You might need to watch 20.

You might need to watch 20, then check your hypothesis at a larger scale with a survey or analytics.

It all depends on the confidence of your team.

There are only 2 things you need to do to start building confidence in your next decision:

  1. Start watching end users using the thing you’re making. (Do research in every iteration.)***
  2. Make sure enough of your team observe the research. (Research is a team sport, help your team get their exposure hours.)

As soon as you start doing this, you’ll be so busy working out what to do with everything you’re learning, you’ll have no time to worry about statistics.

You’ll know what is broken and what is not, and you’ll know what you need to spend more time learning about.

It’s as simple as that.

Footnotes

* Obviously it’s better to see a lot of people over very few

We recommend you see a handful in each iteration – keep iterating, and you’ll gradually get a larger and broader sample. Also, the ‘Is 5 people enough?’ question often discussed by usability people is irrelevant when you’re seeing 5 people in every iteration.

** Your next decision might be…

  • How do we fix the question in this form so that people can answer it better?
  • Is this service meeting the user needs?
  • What is the next most important thing we should prioritise in the backlog?
  • Does this work better than what we’ve currently got live?

*** Who do you watch?

It’s tempting to launch into a complicated segmentation exercise. Don’t do that. There will be some obvious segments – just start with those. Start talking to people and the segmentation that really matters will emerge from those discussions.

The important segments will sometimes be things you couldn’t have guessed at before you started meeting people. Start talking to the easiest people. Pick the people who are most likely to be able to (and want to) use your service and test with them. If your service doesn’t work for them, it won’t work for anybody.

As you begin to solve problems, make it harder for yourself by inviting people who are going to be more difficult – those with edge case and the complex user needs and those with lower (or no) digital literacy.

What user researchers do when they’re not researching

Leisa Reichelt

Originally published on the GDS User Research Blog

A lot of teams aren’t used to having a user researcher embedded full time. When we suggest that they should, we’re often asked:

But what will the researcher do when they’re not conducting research?

We’ve discovered that our user researchers are only actually researching about 30% of their time.

Whilst it is our job to understand end users and how they interact with our service, we think our most important job is this: to make sure that everyone in our team understands end users with the same empathy, accuracy and depth as we do.

Here are some of the things that embedded user researchers do when they’re not researching:

We work with the team to improve the service

  • Analyse research with the team, understand the key insights and work out what actions to take as a result.
  • Work with designers, content designers and developers to make prototypes for testing something new. We sit with the designer and developer so we can chip in and answer questions about what’s being made and why.
  • Spend time with the business analyst or product manager to prioritise features and stories. We also work through questions thrown up, as a result of research, around user stories and acceptance criteria.

We communicate what we know

  • Give research ‘playbacks’ and share key research insights with the project team and stakeholders.
  • Compile video clips from the research so those who couldn’t attend can see real users for themselves.
  • Write blog posts on our research findings.
  • Prepare and deliver presentations, demos and show-and-tells.
  • Make things to post on walls  to keep people thinking about our users, their needs and the challenges we need to overcome to make a great service for them.

We prepare for research

  • Organise upcoming research participants by either writing screeners for agencies or contacting potential participants directly.
  • Work with the design and development teams to work out future user research needs and to create research hypotheses.

We look at site analytics to…

  • See how, and if, we can use analytics to quantify findings we’ve observed qualitatively.
  • Spot patterns in the data and identify future research needs.

These activities are essential in making sure that the time and effort put into user research isn’t wasted. Research is only valuable if your project team understands it and acts on it. These activities help make that happen.

At least 3 days a week

We recommend that user researchers are embedded in a team at least 3 days a week. In fact, most researchers find that just 3 days a week leaves them stretched for time in getting everything done.

It’s a different approach to research than most are used to, but in agile projects, we’re finding it’s the only way that works.