Evidence supports the claim that CLA contributes to improved development outcomes.

• Bottom-up approaches contribute to better development results: A recent study analyzing about 10,000 development projects found that aid agencies achieve better results when using bottom-up approaches that empower frontline workers and organizations to make decisions using their local knowledge and relationships. The study finds that we are more likely to miss the mark on our development goals when we lead with a headquarters-driven, top-down management approach. While some may perceive that bottom-up approaches incur more risk due to a loss of centralized control, the study demonstrates the opposite: in most scenarios, top-down fails more often. Why? Because overly prescriptive rules and controls meant to curtail bad behavior can also curtail good behavior, making it difficult for staff to apply locally relevant knowledge and adapt programs (i.e., “navigate by judgment”).

• M&E are positively and significantly associated with achieving development outcomes when incorporated into program management and designed to support learning and decision-making: A June 2016 World Bank study analyzed large sets of data to determine if there was a correlation between the quality of M&E and project outcomes. It found that good-quality M&E that informs decisions during and after implementation is positively and significantly associated with project outcomes. In addition, several cases in the literature pointed to the importance of using evaluation for learning to enable adaptive management and improve performance

An analysis of CLA Case Competitions from 2015 also shows that a number of cases across multiple technical sectors reported improved organizational and development outcomes as a result of their CLA approach. The main findings linking CLA to these outcomes include:

1. Collaboration leverages resources for collective benefit: The cases that shaped this finding describe how collaborating helps development actors identify their respective comparative advantages around a common goal. Then, the stakeholders decide on next steps and divvy up responsibilities. Based on the final agreement each stakeholder provides funding, human resources and/or materials—among other potential contributions—toward the mutual desired outcome. This collaboration then leads to organizational and/or development outcomes that may not have occurred otherwise. Cases that support this finding demonstrate how strategic collaboration supported the dairy sector in Bangladesh, access to water in Lebanon, integration of humanitarian and development funding in Somalia, and savings groups in Uganda.

2. Local engagement leads to local ownership and, ultimately, improved development outcomes: Several of the cases demonstrate that engaging with local stakeholders contributes to their increased ownership, which in turn can lead to better development outcomes. When implementing partners invite local stakeholders to participate in development processes, they become motivated and engaged in solutions to community challenges. In the analysis, we see how local engagement contributed to improved sanitation in Zambia, savings groups in Uganda, and livelihoods in global programming.

3. Intentional knowledge management generates standard good practices for broader application: The cases contributing to this finding demonstrate how capturing knowledge and sharing best practices derived from that knowledge can contribute to improvements at the organizational level. Moreover, knowledge dissemination can also lead to scale-up of good practices. From the analysis we learned how intentional knowledge management supported improvements to safe male circumcision in Uganda and preventing mother to child transmission of HIV in Tanzania.

4. Feedback loops increase the likelihood that evidence will inform decision-making: The cases illustrating this finding describe how specific tools and processes for creating feedback loops provide continuous learning to inform decision-making. In generating feedback, teams and organizations analyze learning, make decisions based on that learning and then follow through on decisions reached. Seven additional cases support this finding and include examples of how real-time data led to greater responsiveness in the aftermath of the 2015 Nepal earthquake, a midterm evaluation resulted in changes to an activity’s theory of change and overall approach, and routine data collection in Haiti informed where best to allocate resources.

5. CLA begets CLA and sometimes leads to scale-up: Some cases highlight how personally experiencing a CLA approach can lead to increased CLA uptake among staff within an organization and thereby lead to potentially improved organizational and/or development outcomes. Other cases show that there is a “demonstration effect” when development stakeholders learn about the benefits of a successful CLA approach implemented by another actor, and they then adapt this approach and scale it up in their own context. Nine additional cases support this finding and include examples of how implementing partners applied CLA approaches that USAID/Uganda and USAID/Malawi promoted and scale-up of an effective monitoring approach for gender programming.

A growing body of evidence from the CLA literature review also emphasizes the link between CLA and improved organizational effectiveness, demonstrating that:

• Strategic collaboration improves performance

• Success of donor staff is linked to using locally-led approaches

• Pausing and reflecting contributes to learning and improved performance

• Continuous learning is linked with job satisfaction, empowerment, employee engagement and ultimately, improved performance and outcomes

• Quality knowledge management (KM) systems have a significant impact on project performance

Lastly, a review of the Federal Employee Viewpoint Survey (FEVS) data revealed that the relationships between CLA and employee empowerment, engagement, satisfaction and perceived organizational effectiveness are strong, positive, and significant.

The five key findings from 2015 Case Competition Analysis described above also illustrate how these CLA approaches contributed to organizational or development outcomes:

Finding 1: Collaboration leverages resources for collective benefit.

Finding 1 in Action: Multi-Stakeholder Collaboration Supports Cross-border Vaccinations in the Horn of Africa

A 2013 polio outbreak in the Horn of Africa led the CORE Group Polio Project (CGPP) to focus on cross-border transmission and work to tackle the spread of polio in the region’s most unstable areas. To address this complex crisis, CGPP employed the Secretariat Model, which convenes civil society, in-country and such international actors as UNICEF and the World Health Organization, to coordinate efforts to stop the spread of poliovirus in cross-border areas. This collaboration, based on the respective value add of each stakeholder, produced positive outcomes, including expanding vaccination activities to other remote areas in the Horn of Africa

Finding 2: Local engagement leads to local ownership and, ultimately, improved development outcomes.

Finding 2 in Action: Local Engagement Helps Curtail the Spread of Ebola in Liberia

When Ebola struck Liberia and eventually spiraled into an epidemic, Global Communities realized it needed to help curtail the epidemic via improved dead body management and safe burials. However, communities were skeptical of Global Communities and even attacked staff members and their vehicles. Residents wanted to know: “Why do you only come when someone has died? Why do you not come to help when someone is sick?”

As a result, Global Communities had to quickly shift tactics and meaningfully engage community members, particularly traditional leaders, to support safe burial. Because of this intentional relationship building, traditional leaders were willing to educate their community members about Ebola transmission and accompany burial staff for safe and peaceful burials. As a result the number of safe burials increased and Ebola transmission rates declined.

Finding 3: Intentional knowledge management generates standard good practices for broader application.

Finding 3 in Action: Systematic knowledge management contributes to scale up of improved safe male circumcision

After an external assessment revealed inconsistencies in quality of service for safe male circumcision in Uganda, the USAID Applying Science to Strengthen and Improve Systems project (ASSIST), with other implementing partners, was asked to make improvements in 30 health facilities.

USAID ASSIST established a process to continuously identify gaps in the health system and then co-create local solutions to address them. To implement this process, the program trained health workers on managing adaptively: in peer-to peer sessions, they learned how to regularly monitor quality, identify areas for improvements, and adopt best practices. The program also created a guide and best practices document that was shared with all 30 sites and can be used in other regions and countries. The approach spread from 30 sites to now 165 sites in Uganda and has now been introduced by ASSIST in three other countries. It also led to the development and adoption of standardized national tools and indicators for monitoring the quality of services

Finding 4: Feedback loops increase the likelihood that evidence will inform decision-making.

Finding 4 in Action: Real-Time Feedback Supports Adaptive Management in Uganda

To improve literacy rates and reduce HIV transmission among primary and secondary school students, the Ugandan government designed an integrated education and health strategy. USAID/Uganda supports this approach through the School Health and Reading Program (SHRP), implemented through the Research Triangle Institute (RTI). The mission awarded a Performance and Impact Evaluation mechanism (P&IE) to Panagora Group with the goal of providing monitoring, evaluation, and CLA advisory services to RTI. Panagora Group developed a multi-stage approach to continuous learning by providing RTI with real-time performance information needed to inform adaptive management decisions and actions. The example from the case shows how feedback was used to improve a training.

Finding 5: CLA begets CLA and sometimes leads to scale-up

Finding 5 in Action: Effective M&E for Learning Practices Scaled Globally

To increase farmers’ access to food and improve nutrition, CARE worked to change the behaviors of rural farmers in Bangladesh through its Strengthening the Dairy Value Chain activity. However, behavior change is notoriously difficult to measure, particularly when an activity involves tens of thousands of program participants. As a result, CARE created a Participatory Performance Tracker (PPT) to track and discuss behavior change data relevant to the activity’s objectives. It also resolved some of CARE’s operational challenges: relying on the PPT required community groups to hold regular meetings to review their behaviors. These data could then be aggregated and shared with program staff, who would analyze the data and suggest course corrections. Assessing their own progress also ended up motivating community groups to take more agency in adopting behaviors promoted by CARE. It also led to CARE expanding the use of the PPT to an additional 8 country offices working with 5,000 community groups. In both cases, effective M&E for learning had a ripple effect on communities using the tool and CARE’s offices.

The CLA literature review  also documents the specific conditions under which CLA contributes to improved organizational effectiveness and development outcomes:  

• Using evidence to make decisions is more likely to occur when decision makers themselves demand, define and interpret evidence. The literature identifies a number of principles for ensuring the use of evidence when making decisions. These include assessing the needs and identifying specific demands of users, understanding and engaging with target audiences throughout, and ensuring ongoing engagement with and between users and producers of evidence. Even when good-quality, relevant and reliable research is available, straightforward application is difficult. Several studies suggest that successful implementation of research necessitates the interest and involvement of decision makers and an explicit focus on ideas, practices and attitudes specific to the context of users.

• Individuals who are curious, have “growth mindsets,” and are able to empathize with their colleagues are generally better able to adapt to changing circumstances. Ultimately, it is individuals who take on the work of collaborating, learning and adapting within organizations and across partner organizations. Individual personality traits, habits and competencies can affect who is more likely to take on these behaviors. The literature reviewed found the ability to be flexible and adaptive is highly related to individual personalities, which in turn drive office culture and institutional appetite for change. Across sectors, the literature found that hiring those with “adaptive mindsets” (inquisitive by nature, able to ask the right questions, flexible skillsets) and those that show sensitivity to the feelings and needs of their colleagues had a direct impact on a team’s ability to learn and adapt to effect change.

• Adaptive management contributes to sustainable development particularly when it has leadership support, public support, and an adequate investment of time: Evidence suggests that aid agencies are most successful when they are able to operate flexibly and manage adaptively. In many ways, insights from the business and natural resource management sectors parallel much of the debate in development practice. One study recently found that companies that apply more data-driven and adaptive leadership practices perform better compared to those that focus less on those practices. Another study found that change brought about by adaptive management can be achieved, but it can only be achieved slowly, with an adequate investment of time and it requires key ingredients of: leadership, data, patience and public support

• Leaders are essential to creating a learning culture, the foundation of learning organizations. The literature discusses how organizations that encourage honest discourse and debate and provide an open and safe space for communication tend to perform better and be more innovative. Leaders are central to defining culture and “learning leaders” are generally those who encourage nonhierarchical organizations where ideas can flow freely

• Teams that have high levels of trust and are considered safe for interpersonal risk-taking tend to be better at learning and adapting. Managing adaptively requires a level of group tolerance for risk-taking, which by extension is contingent on teams having trusting relationships. The literature reviewed found that high trusting teams generally tend to be high-performing. Why are high trusting teams higher performing? Because they also tend to have high levels of “psychological safety,” which is the shared belief that the team is safe for interpersonal risk-taking. This means they are more likely to participate in risk-taking learning behavior, and by extension proactive learning-oriented action, which positively impacts results.

• Managing adaptively is more likely to improve outcomes when decision-making autonomy is placed as close to frontline staff and local partners as possible:  This evidence also echoes findings from the broader public management literature that decentralized authority is associated with better performance. Evidence from aid agencies and developing country governments supports this conclusion, suggesting that greater autonomy helps project adaptability and flexibility

The next iteration of the CLA Case Competition Analysis will also seek to address the “under what conditions” question by focusing specifically on barriers and enablers to CLA. This evidence will be available in 2019.

Development outcomes:

Finding 1: Collaboration leverages resources for collective benefit.

Finding 1 in Action: Multi-Stakeholder Collaboration Supports Cross-border Vaccinations in the Horn of Africa

A 2013 polio outbreak in the Horn of Africa led the CORE Group Polio Project (CGPP) to focus on cross-border transmission and work to tackle the spread of polio in the region’s most unstable areas. To address this complex crisis, CGPP employed the Secretariat Model, which convenes civil society, in-country and such international actors as UNICEF and the World Health Organization, to coordinate efforts to stop the spread of poliovirus in cross-border areas. This collaboration, based on the respective value add of each stakeholder, produced positive outcomes, including expanding vaccination activities to other remote areas in the Horn of Africa

Finding 2: Local engagement leads to local ownership and, ultimately, improved development outcomes.

Finding 2 in Action: Local Engagement Helps Curtail the Spread of Ebola in Liberia

When Ebola struck Liberia and eventually spiraled into an epidemic, Global Communities realized it needed to help curtail the epidemic via improved dead body management and safe burials. However, communities were skeptical of Global Communities and even attacked staff members and their vehicles. Residents wanted to know: “Why do you only come when someone has died? Why do you not come to help when someone is sick?”

As a result, Global Communities had to quickly shift tactics and meaningfully engage community members, particularly traditional leaders, to support safe burial. Because of this intentional relationship building, traditional leaders were willing to educate their community members about Ebola transmission and accompany burial staff for safe and peaceful burials. As a result the number of safe burials increased and Ebola transmission rates declined.

Finding 3: Intentional knowledge management generates standard good practices for broader application.

Finding 3 in Action: Systematic knowledge management contributes to scale up of improved safe male circumcision

After an external assessment revealed inconsistencies in quality of service for safe male circumcision in Uganda, the USAID Applying Science to Strengthen and Improve Systems project (ASSIST), with other implementing partners, was asked to make improvements in 30 health facilities.

USAID ASSIST established a process to continuously identify gaps in the health system and then co-create local solutions to address them. To implement this process, the program trained health workers on managing adaptively: in peer-to peer sessions, they learned how to regularly monitor quality, identify areas for improvements, and adopt best practices. The program also created a guide and best practices document that was shared with all 30 sites and can be used in other regions and countries. The approach spread from 30 sites to now 165 sites in Uganda and has now been introduced by ASSIST in three other countries. It also led to the development and adoption of standardized national tools and indicators for monitoring the quality of services

Finding 4: Feedback loops increase the likelihood that evidence will inform decision-making.

Finding 4 in Action: Real-Time Feedback Supports Adaptive Management in Uganda

To improve literacy rates and reduce HIV transmission among primary and secondary school students, the Ugandan government designed an integrated education and health strategy. USAID/Uganda supports this approach through the School Health and Reading Program (SHRP), implemented through the Research Triangle Institute (RTI). The mission awarded a Performance and Impact Evaluation mechanism (P&IE) to Panagora Group with the goal of providing monitoring, evaluation, and CLA advisory services to RTI. Panagora Group developed a multi-stage approach to continuous learning by providing RTI with real-time performance information needed to inform adaptive management decisions and actions. The example from the case shows how feedback was used to improve a training.

Finding 5: CLA begets CLA and sometimes leads to scale-up

Finding 5 in Action: Effective M&E for Learning Practices Scaled Globally

To increase farmers’ access to food and improve nutrition, CARE worked to change the behaviors of rural farmers in Bangladesh through its Strengthening the Dairy Value Chain activity. However, behavior change is notoriously difficult to measure, particularly when an activity involves tens of thousands of program participants. As a result, CARE created a Participatory Performance Tracker (PPT) to track and discuss behavior change data relevant to the activity’s objectives. It also resolved some of CARE’s operational challenges: relying on the PPT required community groups to hold regular meetings to review their behaviors. These data could then be aggregated and shared with program staff, who would analyze the data and suggest course corrections. Assessing their own progress also ended up motivating community groups to take more agency in adopting behaviors promoted by CARE. It also led to CARE expanding the use of the PPT to an additional 8 country offices working with 5,000 community groups. In both cases, effective M&E for learning had a ripple effect on communities using the tool and CARE’s offices.

The CLA literature review  also documents the specific conditions under which CLA contributes to improved organizational effectiveness and development outcomes:  

• Using evidence to make decisions is more likely to occur when decision makers themselves demand, define and interpret evidence. The literature identifies a number of principles for ensuring the use of evidence when making decisions. These include assessing the needs and identifying specific demands of users, understanding and engaging with target audiences throughout, and ensuring ongoing engagement with and between users and producers of evidence. Even when good-quality, relevant and reliable research is available, straightforward application is difficult. Several studies suggest that successful implementation of research necessitates the interest and involvement of decision makers and an explicit focus on ideas, practices and attitudes specific to the context of users.

• Individuals who are curious, have “growth mindsets,” and are able to empathize with their colleagues are generally better able to adapt to changing circumstances. Ultimately, it is individuals who take on the work of collaborating, learning and adapting within organizations and across partner organizations. Individual personality traits, habits and competencies can affect who is more likely to take on these behaviors. The literature reviewed found the ability to be flexible and adaptive is highly related to individual personalities, which in turn drive office culture and institutional appetite for change. Across sectors, the literature found that hiring those with “adaptive mindsets” (inquisitive by nature, able to ask the right questions, flexible skillsets) and those that show sensitivity to the feelings and needs of their colleagues had a direct impact on a team’s ability to learn and adapt to effect change.

• Adaptive management contributes to sustainable development particularly when it has leadership support, public support, and an adequate investment of time: Evidence suggests that aid agencies are most successful when they are able to operate flexibly and manage adaptively. In many ways, insights from the business and natural resource management sectors parallel much of the debate in development practice. One study recently found that companies that apply more data-driven and adaptive leadership practices perform better compared to those that focus less on those practices. Another study found that change brought about by adaptive management can be achieved, but it can only be achieved slowly, with an adequate investment of time and it requires key ingredients of: leadership, data, patience and public support

• Leaders are essential to creating a learning culture, the foundation of learning organizations. The literature discusses how organizations that encourage honest discourse and debate and provide an open and safe space for communication tend to perform better and be more innovative. Leaders are central to defining culture and “learning leaders” are generally those who encourage nonhierarchical organizations where ideas can flow freely

• Teams that have high levels of trust and are considered safe for interpersonal risk-taking tend to be better at learning and adapting. Managing adaptively requires a level of group tolerance for risk-taking, which by extension is contingent on teams having trusting relationships. The literature reviewed found that high trusting teams generally tend to be high-performing. Why are high trusting teams higher performing? Because they also tend to have high levels of “psychological safety,” which is the shared belief that the team is safe for interpersonal risk-taking. This means they are more likely to participate in risk-taking learning behavior, and by extension proactive learning-oriented action, which positively impacts results.

• Managing adaptively is more likely to improve outcomes when decision-making autonomy is placed as close to frontline staff and local partners as possible:  This evidence also echoes findings from the broader public management literature that decentralized authority is associated with better performance. Evidence from aid agencies and developing country governments supports this conclusion, suggesting that greater autonomy helps project adaptability and flexibility

The next iteration of the CLA Case Competition Analysis will also seek to address the “under what conditions” question by focusing specifically on barriers and enablers to CLA. This evidence will be available in 2019.

The CLA Initiative for Measurement (CLAIM) learning network launched in the fall of 2016 to answer the following questions: 

• Does an intentional, systematic and resourced approach to collaborating, learning and adapting contribute to development outcomes? If so, how? And under what conditions?

• How do we know? 

The CLAIM network was also meant to help address the question of how to measure the contribution of CLA to better outcomes.  When it comes to the measurement question, the learning network members emphasized:

• Defining what CLA practices are, how they are interrelated, and which practices will be measured are key to understanding the power of your findings and comparing with others’ findings. Generating a general theory of change and a detailed theory of change can help to minimize the difficulty of comparing across research agendas.

• Methods that analyzed decision-making processes were most successful in connecting CLA to development outcomes but come with limitations, including recall and confirmation bias as well as an inherent focus on actions taken rather than those not taken.

• Flexibility of research design is necessary due to the complexity of the research agenda. Network members recommended, to increase the likelihood of producing robust findings, including multiple partners and using varying tools to increase the adaptability of the research agenda

All in all, the CLAIM network has advanced our understanding of how to measure CLA, but more research is needed to definitely answer the question of measurement.