Twaweza’s internal monitoring aims at documenting what we do and why, and following up on what works, and what enables learning and informed decision-making.
Monitoring at Twaweza aims at enhancing our understanding of what works under which conditions and at being transparent and accountable. Monitoring is an important part of our learning loops and closely linked to learning, communication and evaluation. Exploring new ways of working bears the need to document what we do and follow up. Monitoring generates information that allows for learning, how to do things better and make informed decisions about the next initiative. Monitoring is also crucial for informing evaluation and for our communication with partners, public and donors. Both internal monitoring and external evaluation are closely linked to Twaweza’s outcome indicators as formulated in the metrics framework.
There are three steps that comprise Monitoring at Twaweza, each with several components, illustrated below. This includes a range of different activities, mostly undertaken by the LME unit and acting as independent verification and learning exercises on the core Twaweza activities.
The production of outputs is monitored through the units overseeing the relevant contracts, including ensuring that outputs conform to internal quality standards (e.g., printing quality, sampling standards, etc.). These internal standards are developed and updated by units with the relevant specialization. Distribution of the outputs can be monitored by the contracting party (self-reports), although an independent check is usually required as well. This check can be performed by the overseeing Twaweza unit, or by LME; it is usually quantitative in nature. The main task of the LME unit is to provide a structure for developing Tier 1 monitoring plans, guidance on standards and tools, assistance in carrying out independent checks, and promoting the use of data and evidence to inform implementation decisions.
Twaweza’s interventions and initiatives will have the greatest chance of success if they are subjected to repeated testing, tweaking, adapting. An essential component to this is setting up feedback loops. These are tailor-made, small-scale measurement exercises with the following characteristics:
- They reach out to the target audience, and measure the perceived quality, relevance, and usefulness of the intervention/initiative
- The data collected through them is actionable – something that the organization can change through implementation
- Data is relatively easy to collect, easy to analyze – with either only internal, or limited external capacity needed
- They have very quick turnaround from data collection to application, from findings to learning
The most innovative ideas and greatest initiatives will not result in any significant effects if they are not implemented widely enough among the selected audience/population. Twaweza endeavours, wherever possible, to measure coverage (or reach) of its initiatives. Mechanisms for this can vary, depending on the type of initiative that is implemented, and can range from a nationally-representative survey to a tailor-made exercise following just a selected sample of individuals. In addition, we also measure how much coverage Twaweza gets in the media, and the quality of the coverage. This is one of the ways in which we can gauge whether we are influencing the tone and nature of national dialogue and debates, as presented through the media.
For some of our outputs an important measurement and validation exercise is the opinion or assessment by experts in a given field. These exercises are complementary to the feedback from target audience; the purpose is to obtain an objective view of the quality and relevance of a product. Twaweza conducts a limited set of such exercises, and each is tailor-made for the particular output.
Assessing the link between outputs and intermediate outcomes is an essential component of Twaweza’s internal measurement structure. This area of work, sometimes referred to as process evaluation, is conceptually and logistically demanding, as it goes beyond tracking the actual outputs, and into measuring the possible effects these outputs can have on the intended target audiences. In many cases, we envision using mechanisms useful for Tier 2 Feedback loops to also assess feedback related to intermediate outcomes. Similar to Tier 2, Twaweza’s LME unit will establish shared core concepts and guidance on how to think about and plan for assessing intermediate outcomes, and a set of standards, mechanisms and tools (including feedback loops) tools for implementation.
We want to make it easy to follow the results chain from Twaweza via partners to changes reflected in media and trace changes backwards, to see links and synergies, or to search by sector, network or goal. We are aiming to develop a web interface that will communicate monitoring information and allow the public, external evaluators, and our partners and donors to input and access data, generate reports, thereby facilitating others to combine different sources, create new knowledge and share lessons.
View our Metric Framework here.
See examples of our partner monitoring:
- Bring the light | The impacts of solar lamps on learning - Twaweza partnered with SolarAid in 2011 to subsidize and distribute 100,000 solar lights to secondary school students and teachers in Tanzania with the goal of providing a healthy source of light primarily for studying. After two years of the partnership, we wanted to find out if the lights were still functioning, what their primary use was, and whether there was any effect of the light on studying.
- The Effects of Minibuzz in the Tanzanian Media Landscape - This Monitoring Brief presents a summary of the results of an assessment of the Minibuzz TV show.
- Study: How parents recall the Uwezo assessment experience - This report presents results from the Lot Quality Assessment Survey, a monitoring tool to check on the implementation of certain key aspects of the assessment, implemented four weeks after the 2013 Uwezo assessment in Uganda.
- Do public officials listen to the voices of citizens? - In December 2013-January 2014, we conducted a qualitative monitoring exercise among a sample of key public figures and decision-makers to investigate the recall and engagement with the findings of Sauti za Wananchi. This new monitoring brief presents evidence from this exercise.
- Has Uwezo influenced key actors in 2013? Evidence from Kenya, Tanzania and Uganda - This Monitoring Brief summarises the findings from a recent qualitative assessment of the resonance and impact of the Uwezo data collection in Kenya, Uganda and Tanzania.
- Do interactive polls make for better radio content? - Trac FM uses mobile phone technology to collect listeners’ opinions through poll questions which are incorporated into partner radio station programs.Twaweza commissioned a study to monitor the delivery, coverage and quality of their service.
Twaweza uses media monitoring for two main purposes: to follow the reporting and public debate on areas of importance to Twaweza and its partners (education, citizen agency, governance etc.) and to document whenever media mentions Twaweza and its partners. At present this is done by an external media monitoring company in Tanzania on a routine and daily basis, and conducted internally by staff in Uganda and Kenya, and supplemented by external media monitoring company around the time of large media events (e.g., launch of Uwezo annual report).