Evaluation
Twaweza’s external evaluation aims at rigorously analyzing 
impact, Twaweza’s contribution to change, and discussing and developing Twaweza’s theory of change, providing feedback throughout.
Evaluations is part of our monitoring and evaluations framework, which can be seen here. To see how this component fits into the other work we do, view our Metric Framework of Goals and Benchmarks here.
The evaluation of Twaweza focuses on outcomes and impact - How do we know what we achieved? Why did we/did we not succeed in a certain area? Does Twaweza’s theory of change work? It is built alongside Twaweza’s programs, from the beginning, following processes as they unfold. The evaluation is carried out by independent, external evaluators from top research institutions.
We have chosen a jigsaw approach: the idea of having one team in charge of the entire evaluation was abandoned in favour of committing a number of different research teams, each contributing unique expertise to cover the depth and breadth (sectors) of Twaweza’s work. The evaluation teams use mixed approaches and methods (RCT, econometrics, qualitative, participatory, video).
LPT (Lieberman, Posner, Tsai): Evaluation of Uwezo in Tanzania and Kenya (Jan 2011-Dec 2013)
Uwezo aims to increase the agency and participation of citizens, and to improve the quality of education in East Africa, as measured by school age literacy and numeracy. The principal investigators are Prof. Evan Lieberman, Princeton University; Prof. Dan Posner, MIT; and Prof. Lily Tsai, MIT. Read the LPT Evaluation proposal here.
LPT’s research allows for a rigorous assessment of the Twaweza/Uwezo core interventions: literacy/numeracy assessments administered to students in villages and urban locations, as well as various follow-up communication campaigns. The study will provide evidence of the direct impact of the assessments and the immediate follow-up information provided by Uwezo on parent attitudes and participation in their child’s education, and the impact on student performance in the short-term; and the effects of different communication campaigns in generating broader social mobilization and feelings of efficacy both in the communities in which Uwezo worked and in adjacent communities via spill over.
The findings will also provide us with a rich combination of qualitative and quantitative data on the processes that may be activated by these campaigns and how these processes may be shaped by pre-existing contextual and institutional factors. The research proposal reflects Twaweza’s interest in a multi-method analysis and an assessment that incorporates a variety of measurement strategies and analyses and has the following main components:
- Tanzania: Analysis of baseline survey data (2011)
- Kenya: Phase 1 (fieldwork June to September 2011)
- Kenya: Phase 2 (fieldwork during 2012)
AIID (Amsterdam Institute for International Development): The Amsterdam Institute for International Development (AIID) aims at rigorous evaluation of policy interventions in developing countries. AIID uses a battery of techniques to address impact evaluation questions while at the same time striving at rigorous statistical analysis. Locations (e.g. villages) are used as the unit of observation, while survey questionnaires are applied at the household level. Read the AIID Evaluation proposal here.
Principal Investigators are Prof. Jan Willem Gunning and Prof. Chris Elbers. The evaluation has three main components: an econometric analysis based on changes over time; high-frequency (‘real time’) monitoring of Twaweza campaigns at the village level through monthly telephone interviews with 250 village informants over three years; and a package of qualitative methods to support interpretation of the results. Read more about this evaluation here.
Wananchi Survey, managed by Uwazi at Twaweza and expected to be launched in 2012, will regularly contribute large amounts of systematic real-time evaluative information through mobile phones. About 10 questions will be asked in each round from a sample of about 1,250 households in each country. Information collected will include both data on situation as well as public opinion. Topics will include health, education, water and citizen agency, and other issues of public concern. The findings will be quickly analysed, shared with media and published online and through short briefs.
Citizen monitoring is supported through Twaweza’s programs, and is carried out on a more ad hoc basis by volunteers and small groups. The sampling scope of these activities tends to be smaller, and process kept simple. Examples of monitoring undertaken in the past include a survey of the functionality of water points and price of water, checking whether funds have reached schools and the availability of medicines. Results are published in simple briefs and shared with media and online.
Maweni Farm is working with a participatory video evaluation, visiting one community each in Kenya, Tanzania and Uganda over the period of four years, to build an understanding for citizens’ experiences regarding health, water, education and citizen agency at the local level, as well as changes over time and how these are reflected in the public debate.
International Law and Policy Institute (ILPI), Oslo: ILPI is planning a Deep Monitoring Project in Tanzania, and has offered to collaborate with Twaweza on research design, information sharing, and analysis. The purpose is to establish an ongoing local level monitoring process to assess the impact of institutional and socio-economic reforms. ILPI proposes to build and maintain a network of long-term monitors/researchers at village and street level in all regions. The project is a combination of ethnography, based on observation and open-ended questions, alongside a more structured study and quantitative analysis. Should the ILPI project take off, Twaweza will seek active collaboration.
Experiments Twaweza plans to conduct controlled experiments to test possible solutions for enhancing the quality of education in East Africa. These will serve as evaluation components, as well as inform Twaweza’s programmes. Initial discussions have been held with researchers from the international evaluation community (including Abdul Latif Jameel Poverty Action Lab (J-PAL), Innovations for Poverty Action (IPA) and the Center for Effective Global Action (CEGA)). Between 2012 and 2014 the following experiments are planned: capitation grants to schools in Tanzania; local cash on delivery for teachers, whose students perform well according to independent tests in Tanzania; and cash on delivery in Uganda.
Further components for the evaluation will be conceived in 2012. These will be built around the emerging network of Twaweza partnerships and initiatives across East Africa, as well as linked to the development of global knowledge. The aim is to generate a composite evaluation, where the three countries and four sectors are covered by different approaches and methods. At present there are still gaps in Kenya and Uganda (where programs are now emerging), as well as in health and water. We are looking for specialists in rigorous impact evaluation and randomized control trials (RCTs) for experiments on education, and for researchers with extensive experience with qualitative and participatory methods for exploring mechanisms of citizen agency.
