Supporting Local Partners with MEL: Lessons from the USAID/Serbia Local Works Experience
A recent internal evaluation of the Local Works program in Serbia conducted interviews with USAID/Serbia staff and local partners to understand the enablers and constraints to local partnerships. Among the topics discussed was monitoring, evaluation and learning (MEL). While strong MEL processes can facilitate useful learning for both partners and USAID, MEL continues to be a challenge for many of USAID’s local partners around the world. In the words of one USAID/Serbia team member, MEL becomes a practice of “feeding the dragon once a year” during performance reporting, rather than a tool to better understand programmatic outcomes and adapt programming based on that learning.
The internal evaluation asked USAID/Serbia staff and Serbian LW partners about their experiences and recommendations regarding MEL. Their responses led to the following four lessons.
Lesson 1: Provide clear direction on MEL requirements and expectations. The USAID/Serbia LW team provided flexibility to partners to structure their MEL plans in a way that worked well for them. While partners appreciated the flexibility, it left them confused about USAID’s requirements and expectations for MEL plans. In turn, it created a scenario in which USAID had to ask partners to edit their submitted plans.
LW Missions should provide clear communication to partners about which elements are required in a MEL plan, and which elements allow for flexibility. The USAID/Serbia LW team recommends doing this through a workshop at the beginning of an activity to provide guidance on standards, templates, and examples, then conducting individual follow-up with each partner. Partners felt that examples would be especially helpful. One partner remarked,
“It would be great, before we set up the MEL [plan], to see successful examples from other projects and understand whether those were good or bad, costly, or hard. That would help us set up our MEL [plan].” - Local Works partner
If working with a MEL contractor to provide support to partners, it is critical to communicate MEL expectations to the contractor. In Serbia, a gap in the Mission MEL Specialist role meant that the local MEL contractor received little guidance on what USAID wanted from partners, which made it challenging for that organization to support LW partners effectively. Remember that USAID - not the MEL contractor - is responsible for determining requirements and expectations.
Lesson 2: Work in partnership to choose indicators and measurement approaches. LW supports devolving power to local partners to choose what they want to measure. However, keep in mind that many new, local partners are formally approaching monitoring, evaluation, and learning for the first time. Too often, handing partners a template and a list of standard indicators leaves them confused, and they choose indicators that are not a good fit for their activity. They may also choose indicators that require more resource-intensive data collection efforts (such as large-scale surveys), which are unfeasible for some activity budgets and timelines.
LW partners in Serbia felt that they needed not only training, but mentorship and opportunities to talk through indicator selection. One partner related,
“We chose the wrong indicators - some were good, but some were on a more intellectual level that were hard to track, like ‘level of trust’’...so we have to keep explaining why we don’t use it. We didn’t set it up well.” - Local Works partner
Missions can help partners choose appropriate indicators through sitting down with them and helping them think through their intended process or output-level results (i.e. what they did) and outcome-level results (i.e. what happened as a result of the activity in the short-term and long-term). Depending on each partner’s existing level of familiarity with performance monitoring, Missions might encourage them to brainstorm indicators that they feel would capture these intended results. Alternatively, Missions could suggest possible indicators that are a good fit for the partner’s intended results (either from the Standard Indicator Framework, or custom indicators specific to the development challenge and sector), then discussing them with partners to help them choose which ones are most useful. Remember, MEL plans can include both qualitative and quantitative indicators. Qualitative measures may be a better fit for programming in certain sectors (e.g. democracy and governance) in which the most important outcomes cannot be counted without great effort and expense.
It is also important to think about the data collection requirements of selected indicators. For example, does an indicator require a large-scale survey to be measured? If the partner does not have resources for that type of data collection effort, an indicator that relies on data collected through observation or a limited number of interviews with key stakeholders may be a better choice. It’s a good idea to involve Mission MEL Specialists in these discussions.
Under the LW program, each Mission has a dedicated point of contact (POC) in Washington, who is always available to help make a plan for mentoring local partners in this area. LW MEL staff in Washington can help Missions think through tricky measurement challenges, like measuring outcomes that are challenging to quantify, or assessing contribution of activities to development results.
Proactively initiating conversations with partners to select appropriate indicators and plan data collection serves both USAID and local partners. It increases the likelihood that partners will be equipped to collect high-quality data for upward reporting. At the same time, it gives partners an opportunity to understand how monitoring can be useful for their own learning, rather than an exercise conducted only to fulfill USAID requirements.
“We need someone to talk to us, not just show us a paper.” - Local Works partner
Lesson 3: Help partners find value in MEL throughout the life of an activity. Missions can help partners find value in their MEL processes by continuing to provide support throughout the life of an activity. For example, in Serbia, the Mission’s MEL Specialist helps partners revise their activity MEL plans as they discover new indicators or measurement approaches that are a better fit, develop surveys, plan for evaluations, and set aside time to reflect on programmatic learning. Through this effort, the Mission has found that most LW partners have improved their MEL capacities; for example, one partner reported gaining an improved understanding of how to measure changes in norms and attitudes through both quantitative and qualitative methods. Additionally, some of the partners have improved their perception of the value of MEL in itself (beyond just reporting to USAID). Three of the partners independently sought support from the Mission to commission an external evaluation of their programming.
Missions should look for opportunities to support partners in improving their measurement approaches and data quality. For example, areas for additional support highlighted by LW partners in Serbia included:
- Defining and measuring concepts that are hard to quantify, such as ‘improved trust between public and private sectors’
- Conducting measurement for indicators that are hard to measure reliably, such as ‘number of citizens reached through a media campaign’
- Communicating activity successes when outcomes are complicated and very technical, and hard to make appealing to a broader audience
- Assessing the contribution of an activity to observed results
- Measuring systems-level change
Partners in Serbia also felt that pause-and-reflect sessions, facilitated by the Mission MEL Specialist and the MEL contractor alongside AORs, were very helpful to think about accomplishments and consider what adaptations may be helpful moving forward. One partner noted that their team is so focused on the day-to-day operations of the activity, so having someone schedule a time to reflect on learning is very useful.
Lesson 4: Hire a local MEL contractor tasked with supporting LW partners. Supporting local partners new to USAID - both during activity start-up and throughout implementation - is time consuming! Hiring a MEL contractor to take on this work can help. Though the Mission should take the lead in making minimum standards clear - and in setting expectations for mentoring partners in a way that helps them find value in MEL - an external contractor can take on much of the mentorship work that may stretch Mission staff bandwidth. Missions should contract a local MEL provider - rather than a U.S.-based company - whenever possible.
In Serbia, the Mission had already identified capable local MEL service providers, and had set up a local-only IDIQ through which they procured support for LW partners. Though some staff expected the local MEL contractor to exhibit lower capacity than that of international providers, they found that that was not the case. The contractor had a strong command of relevant MEL concepts, and brought fresh ideas on how to conduct monitoring. One USAID respondent felt that the local MEL providers (under the IDIQ) offered higher quality than that offered by international partners, as they were more “dialed in” to the local context. One partner noted that the contractor was very open and available, and could provide support without the need to schedule a formal meeting - something that is often not the case with international contractors. Finally, local MEL contractors can offer high quality support at a lower cost than U.S.-based providers. In the rare instances in which it is not possible to partner with a local MEL provider, Missions should consider these alternative options for drawing on local MEL expertise.
Lastly, to support MEL activities of interest to partners themselves, Missions should consider including a budget for small grants within the MEL provider’s contract budget. This would allow the MEL provider to support emerging opportunities for partner learning, such as analyses that require additional data collection to assess community-level opinions, or pilots of methodologies discussed at the annual MEL training (such as contribution analysis or beneficiary feedback mechanisms).
Do you have additional ideas on how to work with local partners to make monitoring, evaluation, and learning processes more meaningful? The Local Works team wants to hear them! Email [email protected].
These perspectives on supporting local partners with MEL arose from an internal evaluation (USAID internal only) of USAID/Serbia’s Local Works program. The evaluation examines the enablers and constraints to locally led development practices in the Mission. If you have questions, comments, or reactions to this product, we’d love to hear them! Email Amanda Satterwhite ([email protected]) and the Local Works team ([email protected]).