Reflections on a USAID Development Journey #5: Ex Post Evaluations
Introduction to these informal notes
On Halloween 2022, I retired from working with USAID after almost 40 years, as a technical officer, a Project Design Officer and supporting the Agency in what was the former Bureau of Policy and Program Coordination (PPC) and then the Bureau for Policy, Planning, and Learning (PPL). I also had a lot of fun working on renewable energy programs as well as Natural Resource Management (NRM) policy programs throughout Africa, as well as part of the reengineering team developing the Agency’s programming guidance and co-designed the Results Framework. I also had a lot of work on training, knowledge management and a wide range of other things.
One of my most fun jobs over the last several years has been working with a number of the PPL Communities of Practice (COPs), including the Collaborating, Learning, and Adapting COP and the Program Cycle Implementation COP, which are internal knowledge sharing spaces for USAID staff. I decided to leave by summarizing some of my thoughts over the…decades, through a series of posts. Here are the posts, with just some irrelevant timing logistics deleted and grammar and typos corrected, as well as reducing the level of USAID-centric situations and acronyms.
Please remember that they are informal posts, and represent my personal perspective, and do not represent USAID policy.
Reflections on a USAID Development Journey #5: Ex Post Evaluations
An interesting question, re ex post. I think the formal answer would be "yes, ex post evaluations are the type of evaluations I am suggesting.” But a couple of problems with that: "ex post" to me always sounds funereal somehow. Our investment might be done, and hence ex post, but the problem in a broader sense may continue. I wish we had a phrase, one which didn't have built into it the sense of completeness. While this may happen in many cases (such as when we are building something), in other situations, things haven't ended but just evolved. The problem with "ex post" is it sounds so forensic.
To some extent, it may be more like thinking of life as a question of scale: are we looking at the map at one scale, or at a distance, over a broader reach and a longer distance? When we zoom out, is the activity we are involved with the end point or in fact a middle ground to something longer/bigger?
One reason we like the smaller, focused scale is it is more likely to be contractable. Our business model to some extent is both our salvation and our trap. It is an effective way to spend money, do things, and track impact, but sometimes, in order to put these in motion, we need to limit the scope and timeframe, and focus on the doable and trackable. We do this in part by reducing the scope and putting aside some of the complexity, focusing on what we can do and fund in five years or so, and less on what is happening regardless of, and after, our money has been spent.
One area of focus might be to consider using theories of change (TOC), but more than one and at different scales. One which fits into the procurement mechanism - time frame and scale, and another one, with less precision and more uncertainty, which describes the way the broader world works, and evolves, and which we have less control over. In a way, one could say we have it, between the Country Development Cooperation Strategies (CDCS) and corresponding Results Frameworks and Development Hypotheses, and then the TOC used to define a scope of work. But to some extent, the broader view is even beyond/above the CDCS.
In the 1980s, USAID funded a series of massive programs to improve agricultural research. The foundational TOC could be summarized as: bringing the Green Revolution in South Asia to Africa by building up applied crop research and extension services, especially related to cereal crops, would lead to improved yields, more small farmer income and improved nutrition.
These programs often included substantial support from an American university (through what was called a Title XII cooperative agreement), training and strengthening of staff back in the States, the building of infrastructure needed to support an expanded program, and then eventually the research itself.
But then sometime in the mid-80s, the senior economists in the Bureau concluded that these investments were simply not cost-effective, and over a short period of time, the programs were canceled, in favor of a more direct focus on addressing value-added weaknesses in the agricultural value chain that could be tackled in the short(er) term. Then, about 10 years later, the University of Michigan looked at these programs, similarly to the way FAO looked at their agroforestry investments.
What they concluded was that USAID may or may not have cut their programs too early, but that USAID DID look for final impact too soon. The overall timeline for change was probably a decade too short, and success really came from governmental and other local investments, as well as extension and outreach. In other words the actual TOC of final change was far more complicated, and less related to USAID investments, than we assumed when we designed the initial interventions. But one of the downfalls of those programs was not in getting the TOC wrong, but in being wildly expansive and overstated in terms of our role, and in terms of the time it would take. By the time Michigan finished their analysis, while the Bureau agreed with this analysis, the programs were long gone and couldn't be revived.