The Value of Learning from Data in the Field
This piece is cross-posted from PotaVida's blog. Click here to learn more about PotaVida's work.
“The purifiers aren’t working!” was the first thing the users in the Ugandan refugee settlement of Kyangwali told us when we got out of our truck. My heart sank as my mind raced through possibilities. In partnership with World Vision in October 2014, we had distributed a handful of our Smart Solar Purifiers a week earlier, piloting a small batch before handing out dozens more for extended testing. I had taught the recipients to use the purifiers entirely through a translator, using a relatively ad-hoc script. None of the users spoke English, so I was relying on the combination of indirect verbal instructions and pictograms in our printed directions.
What could have happened? The users had seen the red light indicating in-progress, but never the green light indicating completion of the disinfection process. At a loss to even come up with follow-up questions to assess this problem, I instinctively fell back on our usage-tracking technology, for which this was the first live deployment. When I asked to see one of the purifiers and downloaded its usage data, I was at first confused to see that it had apparently been turned off (aborting a disinfection cycle) many times in just one week. The trend continued with a couple more purifiers, leaving me wondering if it was a glitch. Confused, I put faith in the data and asked, through my translator: “Did you turn it off? If so, why?”
“Yes,” everyone I asked said: “we turned it off when we put it away, and we put it away whenever the sun went down or it started raining.” In this forehead-slapping moment I realized what had happened: whenever I demonstrated the usage of a purifier, at the end I would turn it off and put it away. After this breakthrough, the fix was of course easy. I simply added two sentences to the instructions: “You must leave it on until you see the green light. If you turn it off, it starts over from the beginning.” The resultant “Ah-ha” responses needed no translation. A day later, I saw the first green blinking light – the user was so excited to be participating and to have used it correctly that he waited for me to stop by his house to show me.
Over the next few days, I went back to as many people as I could who’d received sample purifiers, to give them the two-sentence refinement to the usage instructions. I also handed out more samples, using the new instructions. The good news is that people continued to use the sample Smart Purifiers, and two weeks later the data – collected by a volunteer, after I’d left Uganda - showed that more than 60% of the samples were being correctly used to purify water. (Note: field studies show only 5-10% correct usage of other water purification solutions, so 60% is an amazingly good number!) Plus, 90% of the samples had seen attempted usage – some users hadn’t received the updated instructions but were still trying to purify water.
How long would it have taken me to diagnose the usage instruction miscommunication without electronic usage tracking? Just one of several initial users reported seeing the green light, but the usage data told me they had not: intentionally or not, they’d misreported their own usage. How would I have known, in the end, whether I’d successfully gotten people to use the purifiers, or just taught them what to say to elicit another smile from this strange and entertaining foreigner? Although I got on a plane thinking my field test had narrowly avoided complete failure, I realized, once I had the data, that we’d succeeded with a level of confidence that isn’t even possible for any other household water treatment solution.
Today, I clearly recognize the power of collecting accurate usage data in the field, taking the time to learn from what the data says, and adapting plans accordingly. My co-founder and I constantly think about ways to empower our partners with better data and help them to use it to achieve better development outcomes. My initial dismay in Kyangwali has turned into a touchstone when discussing the power of accurate usage data and stopping to learn from situations that may appear to be failures: it’s indispensable for solving the challenges you didn’t even know existed.