Failing Fast and Small: Lean Testing to get to Better Learning Outcomes
Morgan Benson is a senior program associate at Results for Development, focusing on R4D's evaluation and adaptive learning approach.
Lean testing is a method of rapidly experimenting with and validating new approaches, or evidence-based approaches that are new to a given context. At Results for Development (R4D), we use lean testing when you have a particular intervention in mind to address your problem, but there is still uncertainty about whether it will lead to its intended outcomes in your context, based on the evidence available. You may also have multiple intervention options you want to test, but you’re not ready to invest in a larger-scale experiment.
Our evaluation and adaptive learning practice has been inspired by scientists like Peter Murray and Steve Ma, who use the term “lean experimentation,” a process similar to what we call lean testing (see figure below). According to Murray and Ma, those who use lean experimentation “run many small tests and adjust their efforts after discovering what works (and what doesn’t work). Done well, lean helps organizations innovate more efficiently, build new services that meet the needs of their constituents, and develop disruptive solutions to seemingly intractable problems.”
(Source: The Promise of Lean Experimentation, Peter Murray & Steve Ma, 2015)
Lean testing can also be a valuable tool for implementers within international development, particularly when working toward an innovative solution. R4D has been using it with education partners like Rising Academy Network in Sierra Leone to design and test a remedial literacy program, and more recently in Liberia with four school operators that are part of the Partnership Schools for Liberia (PSL) initiative.
Lean testing with Partnership Schools for Liberia
In 2016, the Ministry of Education of Liberia invited eight private operators who aim to improve learning outcomes for children to manage 93 public primary schools under the PSL initiative. In 2017, our team was invited to work with four PSL operators to use monitoring and evaluation methods, including lean testing, to generate learning that operators can use to improve their schools and accelerate student progress in literacy and numeracy.
During Theory of Change workshops focused on year two of PSL, each operator identified a major challenge in achieving their intended outcomes: that PSL teachers consistently face difficulties in reaching students of varying ages and ability levels within classrooms. However, the operators were uncertain about the best way to support teachers in overcoming this challenge. So, we co-developed and lean tested two evidenced-based strategies to support teachers drawing on interventions highlighted in R4D’s Early Learning Toolkit:
- Teacher coaching is an approach to teacher training in which teachers receive ongoing one-on-one coaching tailored to their strengths and areas in need of improvement. Teachers develop improvement plans with their coach and participate in regular observation and feedback sessions.
- Differentiated instruction is a teaching method that aims to create a learning environment where the needs of each individual student are met and performance expectations are not guided strictly by age or grade level.
Our approach to lean testing in five steps
We spent one week in each of two schools managed by two different PSL operators to lean test multiple iterations of both interventions, and to learn about what worked well and what didn’t. Here is briefly how we did it:
- Research the interventions. We worked closely with operator leadership to co-develop versions of the two interventions which might be feasible to use within their schools.
- Prepare multiple options for activities and resources. Come with more options than you are likely to test in case, e.g., you need to modify the difficulty level of the content once you learn more about your audience. Through Global Reading Network resources, we identified several existing, validated classroom observation forms and other materials to test and use to generate feedback.
- Test at a small scale and iterate. Roll out small-scale versions of the interventions, taking into account as much as possible what you need to learn from the testing, and what a sustainable large-scale version would need to include. For teacher coaching, we worked closely with teacher supervisors (five total) at both schools to train them in a few different teacher coaching strategies and to use a few different types of classroom observation forms aimed at guiding feedback conversations with teachers. For differentiated instruction, we both: 1) worked with the supervisors to train teachers to use differentiated instruction techniques (grouping students by ability level and tailoring instruction accordingly) within the classroom, and 2) trained multiple school stakeholders to run mock remedial literacy and numeracy classes with students outside of their regular classrooms.
- Observe and collect feedback. We collected feedback on the interventions through direct observations, as well as through interviews with teachers, students, and school management.
- Be willing to learn and adapt. Some seemingly great ideas will not work — that is fine. Failures provide valuable information to use to adapt and improve.
So, what did we learn and where will we go from here?
We learned supervisors have a large untapped capacity to coach teachers, but they need support to do so. While most supervisors were initially unfamiliar with observing and providing feedback to teachers on their classroom strategies, they improved the quality of their feedback during the brief time that we worked with them. Teachers were also receptive to and expressed interest in participating more in a coaching program focused on their professional development.
In addition, we found tailoring instruction by ability within the classroom was a challenge, even for the most experienced teachers. Their capacity should be built in more fundamental skills before introducing this intervention (also due to constraints around re-configuring classrooms).
Finally, less is more for remedial classes. No more than one subject should be covered in a single class period (approx. 50 minutes) due to the need to focus on fundamental skills. Community members identified by the schools will likely have the skills to lead these classes if needed.
Outcomes from this lean testing will help the operators and our team co-design an experiment (e.g., a rapid randomized controlled trial) to compare variations of these interventions to each other and to a control group. Our aim is that, based on this lean testing, the interventions we more rigorously test will be sensitive to local contextual factors and more likely to yield solutions for the operators that generate sustainable, positive change in student learning.