Systems and Capacity: Two Measurement Challenges in Search of Progress Event Materials
Report on Measurement Event
The development community faces some daunting measurement challenges. On August 27, USAID convened a day-long discussion of two of them: measuring systems change and measuring organizational capacity development. Ninety-seven people representing 47 organizations joined the conversation.
The first session focused on USAID’s current thinking on measuring systems change. The need to capture how systems change over time is essential for gauging if development interventions are supporting sustained development. As noted in USAID’s Local Systems Framework, achieving and sustaining any development outcome depends on the contributions of multiple and interconnected actors, actors that interact within a (local) system.
Yet donors and their partners are only able to make a limited number of interventions into these systems. And while traditional monitoring methods can measure the effects of each discrete intervention, capturing how those interventions are contributing to strengthening the local system and promoting sustainable results requires monitoring approaches that are grounded in systems thinking and responsive to the complexity of systems dynamics.
To make headway on this challenge, USAID has developed and begun to test a framework designed to focus attention on five key attributes of systems: the “5-Rs” of rules, resources, roles, relationships and results. In addition, USAID has begun to identify measurement approaches that can help identify change along these five dimensions; approaches that fall into one of three groups:
- Iterative application of systems visualization methods, such as organizational systems analysis
- Methods that aggregate narratives of actors within the system, such as Most Significant Change
- Identification and use of relevant indicators
However, monitoring systems change will inevitably entail assembling multiple approaches into a measurement portfolio.
The second session saw presentations from six organizations sharing their experiences in measuring systems. Representatives from Impact LLC, John Snow, Inc., Root Change, Global Knowledge Initiative, the International Republican Institute, and Plan International described approaches that ranged from ex-ante receptivity to change ex-post evaluation of the extent to which results were sustained.
The morning presentations sparked spirited discussion among participants during Session 3. The feedback during the synthesis session strongly supported the direction this work is going, the shift in emphasis from claiming attribution to documenting contribution for the realization of results, and the utility of the 5-R framework. Additional feedback from that session is available here.
With Session 4, the conversation shifted to the question of organizational capacity measurement. To kick-it off, USAID shared the work of a cross-Agency working group that generated draft recommendations for capacity development measurement as well as a package of associated explanatory documents. The recommendations identify five key principles for measurement of capacity development programming:
- Measure Performance • Define Performance Holistically
- Measure at Organization and System Levels
- Trace Contribution Between Levels
- Account for Complex Pathways of Change
In Session 5, participants provided feedback on the recommendations, broadly endorsing the principles for getting at the “so what” of capacity development and helping to put the relationship between organizations and systems at center stage. They also highlighted concerns around clarity in language, better articulation of what is new and different about it, helping staff digest the implications into their theories of change and project designs, lag time before changes might become visible, and ensuring local ownership of and buy-in to the metrics of a given effort, and suggested various ways to address those concerns in revisions and in planned next steps for incorporation of the principles into policy and training.