When more is not a lot better
This post, by Renee Ho, originally appeared here on Feedbacklabs.org on December 22, 2015.
Collecting feedback is smart for business. It allows you to know what’s working directly from product users, your consumers. Fixing problems and improving usability—whether of a product or service—gives you a competitive edge in the market.
For products and services in aid and philanthropy, the same logic applies: collect feedback so you can rapidly iterate, improve, and provide value to your constituents (“beneficiaries”).
Many people feel that to improve usability you need lots of cases, lots of feedback. Casually, people will say, “Is your ‘n’ big enough?” By “n” they are trying to refer to what, in the language of statistics, is the number in a trial or sample. Yes, for quantitative studies to get statistically significant numbers you need a minimum number of cases. Otherwise, your finding is seen as just an anecdote.
But feedback for improving usability is not about statistical significance. You’re trying to get quick insights that can help you rapidly adapt your product or service.
You might only need a few test users. Five of them, actually.
Jakob Nielsen, from the Nielsen Norman Group, has long argued this:
“Elaborate usability tests are a waste of resources. The best results come from testing no more than 5 users and running as many small tests as you can afford.”
Beyond five test users, the returns or insights are rapidly diminishing— you learn less and less. It’s simply not really worth the time or resources to do more.
If you have budget to do more testing, don’t do more of the same test. Do different tests each with a different set of five users. Nielsen reminds us that whenever a product or service is changed, you’ll anyway have to do a test over again and with different test users.ss. It’s simply not really worth the time or resources to do more.
A few years ago, he looked across 83 usability consulting projects to see if the number of user tests was positively correlated with the number of usability findings.
He writes, “There is a small correlation, but it’s really tiny. Across these many projects, testing more users didn’t result in appreciably more insights.”
In other words, the line is pretty flat. Having 28 test users didn’t provide a lot more benefit than having five.
Nielsen’s research has been long used by the UX/UI world. And over the last few years, human-centered design enthusiasts in the non-profit sector have also used this kind of insight-driven approach too.
What’s even more exciting is an emergent “Adaptive Management” movement (see here, here, and here) that might further promote this “small n” user testing.
According to USAID,
“Adaptive Management is an approach to implementing the Program Cycle that seeks to better achieve desired results and impacts through the systematic, iterative, and planned use of emergent knowledge and learning throughout the implementation of strategies, programs, and projects.”
The focus on iteration and learning through implementation is, in fact, quite reminiscent of the UX/UI’s world of prototyping.
To prototype doesn’t presume a read-made solution; it suggests an idea that, with iteration and adaptation, can improve to become that solution.
It’s important to note that this kind of “small n” approach to collecting feedback is good for improving usability for products or services that most people interact with in more or less the same way. The interfaces of software applications are a good example.
Put an iPhone in the hands of five people and you’re probably fine. But in aid and philanthropy, the way people interact with a product and technology is likely to be less universal. If you prototype a financial savings product, five user tests may simply reflect five very different financial backgrounds. There will still be insights but they may not aggregate neatly.
To solve this problem, it may be a question of market segmentation. What particular population do you want to be using your financial savings product? If you’re able to be more specific (i.e. women below the poverty line in rural western Kenya), then you could find test users from this specific segment and probably get insights that aggregate better.
The “small n” approach isn’t for every case of feedback collection. However, it’s a helpful way for even the most budget-constrained organizations to start thinking about incorporating constituent feedback into their work.