Twaweza’s external evaluation aims at rigorously analyzing impact, Twaweza’s contribution to change, and discussing and developing Twaweza’s theory of change, providing feedback throughout.
Evaluations is part of our monitoring and evaluations framework, which can be seen here. To see how this component fits into the other work we do, view our Metric Framework of Goals and Benchmarks here.
The evaluation of Twaweza focuses on outcomes and impact - How do we know what we achieved? Why did we/did we not succeed in a certain area? Does Twaweza’s theory of change work? It is built alongside Twaweza’s programs, from the beginning, following processes as they unfold. The evaluation is carried out by independent, external evaluators from top research institutions.
We have chosen a jigsaw approach: the idea of having one team in charge of the entire evaluation was abandoned in favour of committing a number of different research teams, each contributing unique expertise to cover the depth and breadth (sectors) of Twaweza’s work. The evaluation teams use mixed approaches and methods (RCT, econometrics, qualitative, participatory, video).
LPT (Lieberman, Posner, Tsai): Evaluation of Uwezo in Tanzania and Kenya (Jan 2011-Dec 2013)
Uwezo aims to increase the agency and participation of citizens, and to improve the quality of education in East Africa, as measured by school age literacy and numeracy. The principal investigators are Prof. Evan Lieberman, Princeton University; Prof. Dan Posner, MIT; and Prof. Lily Tsai, MIT. Read the LPT Evaluation proposal here.
LPT’s research allows for a rigorous assessment of the Twaweza/Uwezo core interventions: literacy/numeracy assessments administered to students in villages and urban locations, as well as various follow-up communication campaigns. The study will provide evidence of the direct impact of the assessments and the immediate follow-up information provided by Uwezo on parent attitudes and participation in their child’s education, and the impact on student performance in the short-term; and the effects of different communication campaigns in generating broader social mobilization and feelings of efficacy both in the communities in which Uwezo worked and in adjacent communities via spill over.
The findings will also provide us with a rich combination of qualitative and quantitative data on the processes that may be activated by these campaigns and how these processes may be shaped by pre-existing contextual and institutional factors. The research proposal reflects Twaweza’s interest in a multi-method analysis and an assessment that incorporates a variety of measurement strategies and analyses and has the following main components:
- Tanzania: Analysis of baseline survey data (2011)
- Kenya: Phase 1 (fieldwork June to September 2011)
- Kenya: Phase 2 (fieldwork during 2012)
AIID (Amsterdam Institute for International Development): The Amsterdam Institute for International Development (AIID) aims at rigorous evaluation of policy interventions in developing countries. AIID uses a battery of techniques to address impact evaluation questions while at the same time striving at rigorous statistical analysis. Locations (e.g. villages) are used as the unit of observation, while survey questionnaires are applied at the household level. Read the AIID Evaluation proposal here.
Principal Investigators are Prof. Jan Willem Gunning and Prof. Chris Elbers. The evaluation has three main components: an econometric analysis based on changes over time; high-frequency (‘real time’) monitoring of Twaweza campaigns at the village level through monthly telephone interviews with 250 village informants over three years; and a package of qualitative methods to support interpretation of the results. Read more about this evaluation here, and the baseline survey findings are available here.
Georgetown University: Twaweza has partnered with the GUIDE (Georgetown University Initiative on Innovation, Development and Evaluation) to evaluate the impact of motivational material provided on videos in inducing citizen agency and collective action in Mukuru slum in Nairobi, home to about 500,000 people. The broad aim is to give individuals and communities the confidence and power to make a difference in their lives, by emboldening them to demand what is rightfully theirs, to work together to achieve common goals, or to exercise agency in creating alternative opportunities when existing arrangements (e.g., for service delivery) prove ineffective.
Do women who are exposed to motivational materials both report and engage in more citizen action than women who are exposed to placebo materials? In other words, can mass media serve as that motivational nudge, so that people will more actively partake in opportunities provided to them (or, even create their own opportunities)? Read more about this work here, and see also this powerpoint presentation that summarises the results of the study.