Measuring the Unmeasured: Assessing the Value of Evidence for Social Impact

Over the course of five years and to the tune of more than a million dollars in funding, three research teams from Kenya and Tanzania worked to catalogue the DNA sequences of endangered animals (here and here) and plants. The purpose of this endeavor was to use DNA evidence in prosecuting wildlife traffickers who have become highly skilled at disguising poached goods. This is seen as a necessary step to preserve biodiversity in the face of a multi-billion dollar a year illegal trading industry. However, try to find evidence of this work in traditional outlets and you’ll likely come up empty handed. Even now, these teams haven't published any findings or registered for any patents. So you might be surprised to learn that they accomplished exactly what they had intended to. 

International development agencies, like USAID, invest heavily in research capacity and evidence generation with the goal of establishing self-sufficiency for local knowledge creation and to use the generated evidence to inform policies and programs that address specific challenges and improve socio-economic outcomes. These investments have proven instrumental in improving capacity within lower and middle income countries to respond to unexpected shocks like the ongoing COVID-19 pandemic.  However, despite significant investments in research, there are few metrics for measuring successes in social impact, which often escape traditional academic publication.

The Program and Policy Change (PPC) Framework is a sector agnostic tool designed by the Research Division in the Innovation, Technology, and Research Hub at USAID. It was developed to evaluate the social impacts of scientific research; in particular, the increased use of evidence to inform policies, programming, and behavior change. A detailed look at the design and practical applications of this framework was recently published in the journal Research Evaluation.

How it Works: The PPC Framework uses qualitative program data and assigns quantitative values as a measure of social impact using a two-dimensional scoring rubric. IMPLEMENTATION asks the extent to which research is being meaningfully applied in practice to policy, programming, or behavior change. INFLUENCE roughly measures the reach or scale of implementation (Figure). 

The Research Division applied the PPC framework to two major programs: Partnerships for Enhanced Engagement in Research (PEER) and Higher Education Solutions Network (HESN). Hundreds of projects can be plotted simultaneously on the PPC framework for comparing program portfolios. Here, the number of projects is denoted by the size of the circle. As research evidence is generated, activities may advance along the implementation scale and into the ‘area of impact’, which indicates stakeholders are putting evidence into practice; a measure of success in research-for-development programming. 

The value of the framework is threefold: 1) Each scored activity is reported on regularly allowing funders to apply adaptive management to an activity (and overall program) and note specific events that led to increases in an implementation score. Patterns in these types of events can offer valuable insight towards improving future program design. 2) It can be applied to any sector. Many international development agencies have diverse research portfolios. This diversity will only continue to expand as we link multidisciplinary teams. 3) Quantitative scoring allows for multiple lines of real comparison. Once individual activities are scored, we can ask questions about effectiveness: amongst research sectors or regions; as a result of the research team characteristics; and between activities and programs. It allows one to quickly visualize performance metrics and hone in on which design elements are moving the needle (although this comes with limitations). 

So how do  the research teams that set out to curb poaching and the sale of bushmeat in East Africa measure up? They hold no patents. They have no publications. They are all but invisible on the international academic field. But in Kenya and Tanzania, they are trailblazers and changemakers, nationally recognized in genomics, conservation, and natural resource law enforcement. Poachers are being prosecuted using DNA evidence for the first time and the myriad evidence generated has impacted an entire ecosystem. The PPC framework gives this evidence generation meaning and a relative value amongst hundreds of other research activities that are also generating evidence in the blind spot of traditional metrics.