Category: research

Originally published on the GDS User Research Blog

If you’re doing qualitative design research, don’t worry about sample size. Sample size and statistical significance don’t matter^*.

The only thing that matters is how confident your team is about the next decision they need to make^** 

You might get that confidence from watching 2 people.

You might need to watch 20.

You might need to watch 20, then check your hypothesis at a larger scale with a survey or analytics.

It all depends on the confidence of your team.

There are only 2 things you need to do to start building confidence in your next decision:

1. Start watching end users using the thing you’re making. (Do research in every iteration.)^***
   
2. Make sure enough of your team observe the research. (Research is a team sport, help your team get their exposure hours.)

As soon as you start doing this, you’ll be so busy working out what to do with everything you’re learning, you’ll have no time to worry about statistics.

You’ll know what is broken and what is not, and you’ll know what you need to spend more time learning about.

It’s as simple as that.

---

Footnotes

* Obviously it’s better to see a lot of people over very few

We recommend you see a handful in each iteration – keep iterating, and you’ll gradually get a larger and broader sample. Also, the ‘Is 5 people enough?’ question often discussed by usability people is irrelevant when you’re seeing 5 people in every iteration.

** Your next decision might be…

• How do we fix the question in this form so that people can answer it better?
• Is this service meeting the user needs?
• What is the next most important thing we should prioritise in the backlog?
• Does this work better than what we’ve currently got live?

*** Who do you watch?

It’s tempting to launch into a complicated segmentation exercise. Don’t do that. There will be some obvious segments – just start with those. Start talking to people and the segmentation that really matters will emerge from those discussions.

The important segments will sometimes be things you couldn’t have guessed at before you started meeting people. Start talking to the easiest people. Pick the people who are most likely to be able to (and want to) use your service and test with them. If your service doesn’t work for them, it won’t work for anybody.

As you begin to solve problems, make it harder for yourself by inviting people who are going to be more difficult – those with edge case and the complex user needs and those with lower (or no) digital literacy.

Originally published on the GDS User Research Blog

A lot of teams aren’t used to having a user researcher embedded full time. When we suggest that they should, we’re often asked:

But what will the researcher do when they’re not conducting research?

We’ve discovered that our user researchers are only actually researching about 30% of their time.

Whilst it is our job to understand end users and how they interact with our service, we think our most important job is this: to make sure that everyone in our team understands end users with the same empathy, accuracy and depth as we do.

Here are some of the things that embedded user researchers do when they’re not researching:

We work with the team to improve the service

• Analyse research with the team, understand the key insights and work out what actions to take as a result.
• Work with designers, content designers and developers to make prototypes for testing something new. We sit with the designer and developer so we can chip in and answer questions about what’s being made and why.
• Spend time with the business analyst or product manager to prioritise features and stories. We also work through questions thrown up, as a result of research, around user stories and acceptance criteria.

We communicate what we know

• Give research ‘playbacks’ and share key research insights with the project team and stakeholders.
• Compile video clips from the research so those who couldn’t attend can see real users for themselves.
• Write blog posts on our research findings.
• Prepare and deliver presentations, demos and show-and-tells.
• Make things to post on walls  to keep people thinking about our users, their needs and the challenges we need to overcome to make a great service for them.

We prepare for research

• Organise upcoming research participants by either writing screeners for agencies or contacting potential participants directly.
• Work with the design and development teams to work out future user research needs and to create research hypotheses.

We look at site analytics to…

• See how, and if, we can use analytics to quantify findings we’ve observed qualitatively.
• Spot patterns in the data and identify future research needs.

These activities are essential in making sure that the time and effort put into user research isn’t wasted. Research is only valuable if your project team understands it and acts on it. These activities help make that happen.

At least 3 days a week

We recommend that user researchers are embedded in a team at least 3 days a week. In fact, most researchers find that just 3 days a week leaves them stretched for time in getting everything done.

It’s a different approach to research than most are used to, but in agile projects, we’re finding it’s the only way that works.

Originally published on the GDS User Research Blog

At least a dozen times a week I find myself saying to someone:

We call it user research not user testing. We test our design, our words and our ideas. We don’t test our users.

It’s a little thing, some might call it pedantry, but we think it matters.

The way we talk about things shows the way we think about things. It’s important to remember that people come to participate in user research to help us learn whether the current approach we’re taking to service design is meeting user needs or not.

It’s us being tested, not our users, and that’s a good thing.

It’s how we learn and continue to improve the quality of work we do, whether we’re designers, developers, researchers, product owners, security people or policy makers.

If people can’t use the thing we’ve made, that’s a reflection on us, not our users.

This is particularly true for us. We’re people who work in government and have a responsibility to make things work for everyone who wants to do things online, no matter their level of digital literacy.

You can, if you like, call it usability testing. That’s okay, if the thing you’re doing is testing the usability of the interface design. Most often though, it’s user research: a mixture of usability testing and more generally trying to better understand our end users so we can make better services for them.