Cracking the Evidence Conundrum: Four Ideas to Get People to Use Evidence

Dec 18, 2017 by Matt Baker, Monalisa Salib Comments (0)

This visual summarizes findings from a recent literature review on evidence-based decision-making. This work is supported by the Global Development Lab (the Lab). We will be sharing the full report here once it is available. If you are interested to learn more, please contact the authors or Jessica Lucas at the Lab.

It is fairly obvious that using evidence to inform our decisions makes sense. USAID’s update to its operational guidance includes evidence-based decision-making as one of its key principles and collaborating, learning and adapting (CLA) as an approach to operationalize this. But using evidence in our daily lives is easier said than done, so what is stopping us from using evidence?

“USAID’s decisions about where and how to invest foreign assistance resources should be based on analysis and conclusions supported by evidence.” (ADS 201 series, pg. 11)

  • Do we even have the "right" evidence to inform our decisions? A 2017 GAO study found that only around 15 percent of USAID managers said they agreed to a very great extent that they had access to the performance information they need to manage their operations or technical work. This is actually in line with the rest of the federal government agencies surveyed, but arguably it could be a lot better.
  • Does the evidence conform to our pre-existing beliefs? Even when the evidence is available, if it doesn’t conform to what we already believe to be true, we may not find it credible and might be less likely to use it.
  • Do political or organizational considerations outweigh the evidence? By and large, evidence isn’t the only input into a decision. What political or organizational imperatives take priority over the evidence? Sometimes those imperatives are in conflict with the evidence (example: working in a certain geographic area because of a local government request even though that area has less need from a development perspective).

Despite these barriers, there are some insights from research on evidence-based decision-making that can help us be more intentional about applying evidence to our decisions:

  • Insight 1: Engage and collaborate with decision makers in the process of generating and analyzing evidence. The literature suggests that while including decision-makers early and often in the process can be useful, outputs should be tailored to the decision-maker in mind. It suggests understanding the motivations of decision-makers, as well as the costs and benefits of interactions between evidence generators and users, to ensure engagement is feasible and effective. Moreover, there is evidence that techniques such as social influence, mentoring, and online engagement can be effective engagement tools.

What typically happens

A better approach

Decision-makers are presented with information after it has been gathered and synthesized and asked to make a decision and/or take action.

Decision-makers are engaged in determining what evidence should be gathered from the outset and reflect on the evidence as it is generated to inform decision-making.

  • Insight 2: Embed the use of evidence into existing organizational structures and processes. One way is to tie or integrate interventions together, such as integrating decision-makers in the evidence gathering process as well as providing communications about and access to evidence. Also, the literature suggests linking evidence to policy priorities and goals, and linking the use of evidence to planning, budgeting, and reporting processes, in order to integrate the use of evidence into how organizations operate. For inspiration, the OMB’s Analytic Perspectives, Budget of the U.S. Government, Fiscal Year 2018 (Washington, D.C.: 2017), as an example, has recommended that federal agencies “adopt a “learning agenda” in which they collaboratively identify the critical questions that, when answered, will help their programs to be more effective, and to plan to answer those questions using the most appropriate tools.”

What typically happens

A better approach

Evidence is generated by separate (often external) people and is only used in an ad hoc fashion or selectively incorporated into daily work processes.

Evidence is systematically integrated into work processes. In the USAID context, this may include requiring that proposal submissions reference the evidence base for their proposed interventions. Likewise, during implementation, using pause and reflect moments to internalize and apply evidence.

  • Insight 3: Share the evidence: proactive in transmitting, accessing and translating the evidence to decision-makers, key influencers, and their networks; via customized, accessible products with tailored messaging that avoids jargon; at the appropriate time. Understanding audiences through segmentation (where you divide a large group of users into smaller sub-groups of users (known as segments) based on some type of shared characteristics), as well as providing multiple modalities for access to evidence (such as printed documents, online platforms and in-person events) were highlighted. The literature focuses on being persistent with regular communication, as well as maintaining flexibility in determining the best approach and means to communicate.

What typically happens

A better approach

Evidence, framed in non-accessible language,  is passively stored away with limited awareness among potential users. Dissemination is very top-down and often only shared once.

Evidence is shared, tailored and disseminated frequently in a variety of forms with special attention to how decision-makers can readily find and understand the evidence.

  • Insight 4: Build the capacity of individuals to understand and see the value of evidence. Awareness-raising interventions alone have little evidence of success in improving the use of evidence, but building the skills and motivation to use evidence showed promise. In particular, efforts to build positive attitudes towards evidence use are highlighted as an important complement to awareness-raising work. Marketing the benefits of using evidence in work was also emphasized. This includes social marketing, professional recognition, and norm setting which have been shown to improve evidence uptake.

What typically happens

A better approach

Decision-makers are assumed to have the awareness, familiarity and interest in evidence that is produced.

Decision-makers are provided with the necessary skills and motivation to use evidence by demonstrating its utility in making their work more effective and efficient.

Yes, these more collaborative, user-centered, and intentional approaches can take more time. But implementing an approach that doesn’t have an evidence base or designing programs without reference to the evidence base is potentially far more costly to the U.S. Government and program participants in the long-term.

How have you integrated evidence into your decision-making processes? What barriers have you faced? And what approaches have worked in your organization?