MEASURING SYSTEMIC CHANGE IN MARKET SYSTEMS DEVELOPMENT – A STOCK TAKING

Background

In 2018, USAID/Honduras’ Monitoring & Evaluation Support for Collaborative Learning and Adapting (MESCLA) activity had the opportunity to support the USAID/Honduras Transforming Market Systems (TMS) Activity management teams to develop the Activity ME&L system. The TMS Activity has the goal of fostering competitive, resilient, and inclusive market systems that provide increased economic opportunities that incorporate poor, marginalized Hondurans and reduce incentives to migrate.

Supported by MESCLA, the USAID/Honduras and ACDI/VOCA Activity Management teams identified a learning need – how other actors working in facilitating market systems development were implementing ME&L to monitor and evaluate market systems change. In particular, the teams were interested in learning from and building relationships with actors outside of USAID programming to expand their learning networks. We were lucky to have the opportunity to partner with Marcus Jenal in this learning effort. In partnership with Marcus, TMS, USAID/Honduras and MESCLA began a 6-month learning journey: reviewing the literature, interviewing donors, implementers, and ME&L specialists.

The learning consultations and research revealed most implementers working in market systems development take one or more of several different perspectives, each of which has implications on what is measured and the determinationof when change is systemic. 

The learning consultations and research revealed most implementers working in market systems development take one or more of several different perspectives, each of which has implications on what is measured and the determination of when change is 'systemic'. 

 Table 1. Four perspectives of systemic change and implications for monitoring and evaluation

Perspective

What is Measured

Methodological Implications

Assessment Implications

Systemic change through innovation diffusion and adoption

Assessment of the extent to which an innovation is diffused across a system

·    Relies on a pre-defined threshold

·    Sampling – representative or census

·    Measure direct and indirect impacts, such as using multipliers

·    Is the innovation adoption catalyzing changes in how the system fundamentally works? 

·    Is the innovation uptake at a system-wide scale?

Systemic change through structural change

Assessment of the extent to which system structures have changed, and extent to which structural changes altered behaviors

·   Relies on a pre-defined threshold

·   Sampling – representative or census

·   Assessment of relationship between changes in structures and behaviors

·   Are the structural changes facilitating behavioral changes?

·   Is the innovation uptake at a system-wide scale?

·     

·     

Systemic change as a change in state

Assessment of the extent to which a gap was closed between a pre and post activity / intervention state

·  Dependent on defining a hypothetical state and measures to assess

·  Relies on pre- and post-assessment

·  Is systemic change achieved if the gap is not closed / targets not achieved as defined?

·  Is achievement of targets the result of fundamental systemic change?

·     

·     

Systemic change as a change in trajectory

Assessment of the extent to which system actors change the evolutionary trajectory of a market system

·  Relies on iterative, ongoing identification of changes

·  Retrospective analysis

·  Lack of pre-defined outcomes makes baselines challenging

·  Is the current trajectory of the system different than what the trajectory of the system would have been without the Activity?

Key Conclusions and Questions

• Subjectivity in Assessing the Systemic Extent of Change. The various perspectives on systemic change, and the approaches used to assess when change is systemic all introduce an element of subjectivity into the assessment, which begs the question – how is the concept of “systemic change” most useful and best used?
• Disconnect between beneficiary and system levels. Whether taking an innovation or structural change-oriented perspective on systemic change, both face challenges in linking change at the beneficiary level to change at the system level
• Pre-identified vs. emerging change. Key informants note that those designing and implementing MSD projects do not know in the begining how exactly systemic change will look in reality, and therefore, the relevant measures for systemic change. This makes developing baselines challenging, and systemic change is often assessed in a retrospective way. Additional mechanisms are needed to strengthen validity.
• Methods for collecting and interpreting data. The researchers did not find any strong alignment between different perspectives and approaches used to assess when change is sytemic, and data collection and interpretation methods. Most projects rely on data collection tools and methods which are commonly used in most research - surveys, focus groups, interviews, and case studies
• Ad-hoc use of complexity aware monitoring tools. When and where complexity aware monitoring tools are used, they are used ad-hoc, and not systematically integrated into the core ME&CLA system.
• Responsibility for measuring systems change. Some key informants argued systemic change is beyond the scope -- both in breadth and time -- of any individual project, and thus should be part of a broader, multi-initaitive strategy, and measured at this level by a seperate entity or collaboration of entities. 

For more information, please read the full stocktaking report. Additionally, please contact Jacob Gray at [email protected], or Marcus Jenal at [email protected].