It’s user research, not user testing

Originally published on the GDS User Research Blog

At least a dozen times a week I find myself saying to someone:

We call it user research not user testing. We test our design, our words and our ideas. We don’t test our users.

It’s a little thing, some might call it pedantry, but we think it matters.

The way we talk about things shows the way we think about things. It’s important to remember that people come to participate in user research to help us learn whether the current approach we’re taking to service design is meeting user needs or not.

It’s us being tested, not our users, and that’s a good thing.

It’s how we learn and continue to improve the quality of work we do, whether we’re designers, developers, researchers, product owners, security people or policy makers.

If people can’t use the thing we’ve made, that’s a reflection on us, not our users.

This is particularly true for us. We’re people who work in government and have a responsibility to make things work for everyone who wants to do things online, no matter their level of digital literacy.

You can, if you like, call it usability testing. That’s okay, if the thing you’re doing is testing the usability of the interface design. Most often though, it’s user research: a mixture of usability testing and more generally trying to better understand our end users so we can make better services for them.

Strategy fast and slow (or culture IS strategy for breakfast, lunch and tea)

This is the third post in a series of rambles around the topic of strategy in the general vicinity of user experience which I’m posting as a kind of obituary to the book I almost finished writing then realised was pretty much completely wrong. This is some of the stuff  I probably should have written instead. You can see the first post here and the second one here.

I’m a big Peter Drucker fan. He was a User Experience guy way before there was such a thing as User Experience Guys. (I wrote up some of my favourite things about Peter Drucker in the earliest stages of trying to write my Strategic UX book. You can read them here.)

Drucker is the guy who apparently said ‘culture eats strategy for breakfast’. I think he’s right, but I also think it’s not as simple as that. I think that culture is strategy in action.

The culture you see reveals the strategies that people are actually employing in their day to day work – not the ones that are proclaimed on posters by the lift or in the annual business plan.

And, as Joshua Porter would say, the behaviour you’re seeing is the behaviour you’ve designed for. (Granted he was referring to the behaviour of people in social systems on web application, but I think the wisdom holds).

So, culture and current strategies are dependent. Culture and proclaimed strategies are often competitors. And there are probably things that we do that encourage the culture we currently have and discourage the culture (and day to day strategy) that we want.

This makes me think of Stewart Brand’s great book ‘How Buildings Learn: What Happens After They’re Built’ (based on a BBC series you can watch in full on YouTube).

In considering the life of buildings once they’ve been designed and come into use, Brand observes the way that we ‘iterate’ buildings so that they better suit user needs. (At least, the good buildings can be iterated rather than bulldozed). There is a great diagram in the book that shows the different ‘layers’ of buildings and the relative speed at which those layers are able to/likely to change.

This makes me think about layers of strategy – firstly that these different layers exist and also that the rate that those layers should change will be different. Some strategic layers should be able to change quickly and others ideally move more slowly. In the same way that you don’t want to change the structure of your house every couple of months (probably), it’s also less than idea for some of the strategic foundations of your organisation to change substantially on a regular basis.

Things like your value proposition, you don’t really want to move that unless you really have to, it should be a slow moving strategy. For somewhere like the Government Digital Service, the Digital by Default Service Standard should be a slow moving strategic document.

This is a document that asks the whole of the UK government to change the way that it creates and maintains digital services in a fairly substantial way. A lot of people are doing a lot of work to get into alignment with this strategy.

The Service Standard should be able to be iterated (to be improved and to respond to change in the world) but it shouldn’t change substantially at a rapid pace. If it changed all the time it would massively compromise it’s ability to be successful.

Neil Addison thought that my use of the word ‘enshrined’ when talking about strategy was indicative of a problematic mindset (and I think he was right to pick it up as a red flag), but I think that it’s indicative of a slow moving strategy. If you’ve got a big ship to turn around, you need some consistency in your approach.

Slow moving strategies can help you get lots of people to get lots of activities into alignment. If you want to do some big things, you need some slow moving strategies.

On the other hand, fast moving strategies can be very useful for helping to shift culture.

Strategies that are smaller, more malleable, and able to be influenced by people at all levels of the organisation. Strategies that form a part of the daily tasks of people in the organisation.

Agreeing and applying design patterns might be a good example of fast moving strategy. Conversations to form and iterate design patterns involve lots of different people in the organisation (or they should), including designers, front end developers, user researchers, security specialists – just for starters.

The goal of the design pattern is to try to get everyone working to a consistently high standard of interface design (and ideally with some consistency in the interface), but obviously as the organisation learns more and more, the quality of the design patterns and their implementation will continue to iterate and improve.

Even more important, the discussions that people have on a daily basis as they implement and challenge the design patterns encourages discussions around design, user research, and technical implementation that help to build a culture where design, user research and technology well implemented is valued.

Things get cool when your fast moving patterns integrate well with some slower moving strategies, like design patterns (faster) integrating with design principles (slower). For example, plenty of discussions about design patterns reference the design principle ‘do the hard work to make it easy‘.

This is what you’re aiming for: layers of strategy from fast moving ones that people ‘work’ on every day that integrate with the next layer out so that  ultimately work that you are doing day to day has an evident link to the big, slow moving strategies.

Fast moving strategies that give people small ways to align their work to the bigger picture – letting people ask ‘what’s the user need?’ or  to think ‘publish don’t send‘ as they decide whether to send an email, write a blog post or update the wiki. (yes, some people do update wikis).

Being about to routinely, every day, every hour, align the choices you make to the strategies of the organisation you work for, that helps build culture. Especially if you’re all using the same words when you do it. 

Its just one ingredient of culture, but being empowered to make tiny small choices to align and sometimes even adjust the strategy of the organisation – the fast and the slow layers, eating strategy for breakfast, lunch and tea helps ensure that you build a culture that’s not just ‘healthy’ but is aligned and supportive of the strategic goals of your organisation.

or something like that.

Strategy doesn’t live in a silo (or, there is no such thing as UX Strategy)

This is the second post in a series of rambles around the topic of strategy in the general vicinity of user experience which I’m posting as a kind of obituary to the book I almost finished writing then realised was pretty much completely wrong. This is some of the stuff  I probably should have written instead. You can see the first post here.

I started trying to write a book about UX Strategy not long after my youngest son was born. This all started because Bruno Figueiredo, who does a wonderful job organising the great UX-LX conference, had asked me whether I thought I could run a workshop on UX Strategy for him. I figured I did strategic type stuff whilst doing user experience work all the time, so how hard could it be?

Turns out, pretty hard.

I spent the following years trying to write about UX Strategy and find myself, with my youngest son about to start school, still not knowing what UX Strategy is. I’ve met plenty of people who are employed as UX Strategists, and I’m still not clear that what they do is specifically a part of the discipline of user experience.

Certainly, there is user experience work that is done more or  less strategically but there is isn’t a role for a person in the UX team to come up with the UX Strategy.

(Of course, along the way I have also been convinced that it’s not legitimate to use the label ‘UX’ when defining a job role or a team either, so perhaps I’m just going through my nihilistic phase?)

Of course, corporate strategies are incredibly influential when it comes to impacting the overall user experience, but they only really effective when initiation and ownership comes from the highest level of the organisation.

Let’s take the my current workplace as an example. If you asked people who work at the Government Digital Service what our strategies are you will almost certainly be referred to one or more of the following:

  • simpler, clearer, faster. You can find this headlining an actual government policy and you fill find it as the tagline on GOV.UK This is not something that a UX team came up with, but it is something that the entire organisation and all of the different disciplines within that organisation use as guiding principles for deciding what to do and how to do it.
  • our design principles. These are called ‘design principles’ and I’m pretty sure they mostly came from the design team (although, GDS people, feel free to correct me), but they are now used by people to make important strategic decisions, like whether or not a project should even be started (is there a user need?)
  • similarly, what the UK Government now considers to be essential practice for teams building digital services is now enshrined in the Digital by Default Service Standard. Again, parts of this might read like what could be a UX strategy, but even those parts are also important guiding principles for people who are doing very different things that making wireframes and journey maps.
  • Probably the most important strategy of all is delivery. Doing less hand waving, white paper writing, frameworking making, and just actually making things and letting people use them so that we can start learning and iterating and actually improving the experience that people are having every single day. This might be a strategy that radically improves the user experience, but it is certainly not a UX Strategy. It’s a statement of intent that informs every decision made across the organisation every day.

This is not just true for UX Strategy. It is the same for technology strategy, security strategy, social media strategy, content strategy. None of these can be effective in isolation, the only strategy that works is cross-disciplinary, multi-disciplinary, integrated across the organisation.

Users win when the whole organisation orients itself around the users so that the technical team, the security team, the content team, the social media team and the designers and researchers all make decisions in the users best interest. Otherwise, all you get is handwaving and frameworks.

5 ways to help user research work better in agile

Originally published on the GDS User Research Blog

User research doesn’t have the best reputation for fitting well into agile project teams. Agile requires speed – getting useful insights to an agile team quickly can be difficult to achieve.

There are 5 things that have helped us make user research more effective in agile:

  1. Dedicate a user researcher to each project team
  2. Do research in every sprint
  3. Evaluate the now/explore the future
  4. Gather both tactical and strategic insights every sprint
  5. Communicate research to your team in many ways

It’s taken us a while to settle on those 5. I’m going to explain what we used to do compared to what we do now.

1. Dedicate a user researcher to each project team

We used to treat user research like an internal consultancy. Teams would brief a researcher to do a piece of work and the researcher would either run the research themselves or procure the work from an agency.

In those days, a 4-week turn around was considered fast for user research. By the time the research came back, chances were high that the team had moved on a sprint (or more) and the research was irrelevant.

Also, because the researcher knew little about the project when they were called, there was no guarantee they were researching the right thing.

What we do now

We now aim that researchers have at least 3 days each week sitting with the team.

This lets us design better research projects and we’re able to learn useful things at the right time for the project’s needs. We can do more than just usability test interface elements and, because we’re now sitting with the team, we can make sure that designers and product owners are able to make practical use of what we’ve learned.

2. Do research in every sprint

We think that agile sprints are one of the best things to have happened to user research.

When a user researcher joins a project team, the first thing they do is make sure a round of research happens in every sprint.

We mostly do lab-based, 1-on-1 depth sessions – it’s the easiest way for everyone in the team to observe the research and get exposure hours. We use other research methods in addition to this regular ‘heartbeat’ of research.

User research is a team sport – an important team mantra.

The wins

Researching in every sprint helps the team maintain a learning attitude throughout the project. This is the best and most valuable outcome of integrating researchers into agile teams.

It also allows us to experiment at low cost with a range of different approaches to service design and we’re continually learning about our audience – what’s important to them and how best to design the service to their needs.

3. Evaluate the now/explore the future

We don’t worry about how many sprints we should be ahead of the developers – we believe researchers need to be all over the service all of the time or at least traversing the ‘now’ and the ‘future’ on a very regular basis.

Evaluating the now

The role that people seem most familiar with in digital teams is evaluative.

Evaluation asks, through usability testing, whether the thing we’ve designed is doing a good job of meeting user needs and is easy to use or not.

Exploring the future

Regular experimental research (using a light-weight prototyping environment) gives the team the opportunity to explore a range of different approaches to design and content and to understand whether the effort to bring those things to reality would be worthwhile.

This is critical in the earliest project phases (alpha and beta) but also supports the continuous improvement required by the Digital by Default Service Standard.

4. Gather both tactical and strategic insights every sprint

Agile research doesn’t restrict you to ‘tactical’ insight only.

Research in each sprint should be both tactical (answering design hypotheses) and strategic (understanding the audience and what’s significant to them in the context of the service).

The trick is to make sure you’re asking strategic questions regularly then making time to update your models and communicate them back to the team so they’re useful.

Our researchers create models, personas, journeys etc. to capture strategic insights, which they update and share with the team as they learn.

… which forms a nice segue to the last point.

5. Communicate research to your team in many ways

Different teams have come up with different ways of communicating key strategic insights to their team, from posters to slide decks.

We don’t produce research reports in agile teams. They take too long to deliver and, when you’e involved in day-to-day design decisions and prioritisation of the backlog, they’re not necessary.

We have another mantra:

Don’t do research for researchers, do it for the team.

Communication is our way of delivering on that.

Sit with the team

Communication happens because the researcher is co-located with the team and can represent the end user when the sprints are being planned and stories are being prioritised.

Communication also happens when the researcher is sitting next to the designer while the interface is being designed – they can provide feedback then and there based on what they know from research.

Present and share

Researchers present at regular show-and-tells (where we show videos from research) and find opportunities to ‘play back’ the research to team members who weren’t able to attend.

We also share the results of strategic research on walls – personas, journey maps and interfaces annotated with research findings.

In conclusion

These 5 points have made a big difference to the effectiveness of user research in our agile teams. We hope you’ll find them useful too.