Learning in Real Time: Three Ways that Implementation Research Adds Value to Public Health Programs
This post was written by Anne La Fond, JSI Center for Health Information, Monitoring and Evaluation and Yasmin Chandani, JSI Center for Health Logistics
Ensuring community health workers can receive continuous supplies of life saving drugs and commodities can, literally as well as figuratively, be an uphill battle. These last-mile workers often feel isolated from the health system because they have no facility to work from and have to navigate difficult terrains - such as dirt trails and waterways that even motorized vehicles cannot traverse - to provide services and receive medicines.
Scarce resources for community-level supply chain strengthening can be most effectively allocated when we have sound data on different strategies that can guide how we allocate time and budget. Investors and practitioners in global health are increasingly recognizing the importance of implementation research (IR) as a means to generate that data to improve the effectiveness and impact of programs. IR studies how research findings and best practices are applied in the field to contribute to evidence-based services delivery system improvements.
Undertaking IR as part of a program should help us understand how and why implementation is working or not within a particular context, rather than focusing only on evaluating program outcomes and impact. By drawing from a structured set of methods, and linking findings with those of impact evaluation, we can ensure that the IR data collected is sound and analytically generalizable to inform program scale-up.
The results can be illuminating. As part of the IR findings from JSI’s Supply Chains for Community Case Management of Childhood Illness (SC4CCM) project, our team found that improving data visibility and information systems had a much bigger impact on product availability than focusing on storage capacity and conditions. Similarly, the project demonstrated that transportation barriers at the last mile could be overcome with effective management interventions. These results provide valuable information to guide future investment.
Here are three lessons that we at JSI have learned about the application and value-add of IR:
- IR is most successful when implementers and measurement staff work as a team. IR is best done when implementers and measurement staff work together as a team, with both sides committed to working towards a common learning agenda and goal. If done effectively, IR brings together implementers and evaluators through a structured process to plan programs, identify key implementation strategies and metrics to measure the performance of the implementation strategies, and then test and refine these strategies based on evidence discerned from the metrics during implementation.
- Real time data allows for continuous improvement. IR gives you strategic information that allows you to make evidence-based decisions about implementation strategies applied within a certain context. IR continues providing this information along the way, enabling course correction throughout the implementation process, rather than being structured to provide single “point in time” snapshots of performance to get logged in a performance monitoring plan and submitted in an annual report.
- IR helps identify strategies that maximize health impact. IR can help us prioritize activities to get the biggest bang for your buck. It can improve investment strategies by comparing one system strengthening intervention with another.
The SC4CCM IR Approach
As part of our SC4CCM learning grant, our team conducted implementation research in Ethiopia, Malawi and Rwanda. This research aimed to understand how to strengthen supply chains for products managed at the community level and significantly improve availability of these products among community health workers.
The team compared the effectiveness of two different implementation strategies in each country against these overall objectives. The interventions included implementing a technology solution (the cStock mobile logistics management information tool), opportunistic and affordable training methods, and establishing quality improvement teams in Malawi and Rwanda that took different approaches. In both countries, multi-level teams used a structured process, tools and data to problem solve and identify actions to improve supply chain performance, facilitated peer-to-peer learning and recognized achievements. In Malawi, the objectives were predefined and a standard set of indicators were reviewed routinely. In Rwanda, teams identified priority objectives and actions on a quarterly basis, and reviewed performance against new targets each time. JSI designed the IR strategy to identify contextual factors that contributed to the successful strategies as well as drivers for scale and sustainability.
As illustrated in the framework below, the team identified three key phases of the journey towards sustainable implementation across the three countries, anchored on collecting and using information about program implementation along the way. Taking an IR approach, diving into the details of how and why pilots were working or not, and creating an environment supporting real-time data use generated the information necessary to make concrete recommendations based on the program’s learning.
At its core, IR can be framed as a method for understanding why something may be working or not within a particular context. This is particularly important as we try to adapt and implement best practices and innovations from the private sector or other settings to increase the impact of our development programs. IR can produce the necessary evidence to inform implementation, and help us along the journey from pilot to scale to sustainable systems strengthening.