There are three steps that comprise Monitoring at Twaweza (see diagram), each with several components. This includes a range of different activities, mostly undertaken by the LME unit and acting as independent verification and learning exercises on the core Twaweza activities.
The production of outputs is monitored through the units overseeing the relevant contracts, including ensuring that outputs conform to internal quality standards (e.g., printing quality, sampling standards, etc.). These internal standards are developed and updated by units with the relevant specialization. The distribution of the outputs can be monitored by the contracting party (self-reports), although an independent check is usually required as well. This check can be performed by the overseeing Twaweza unit, or by LME; it is usually quantitative in nature. The main task of the LME unit is to provide a structure for developing Tier 1 monitoring plans, guidance on standards and tools, assistance in carrying out independent checks and promoting the use of data and evidence to inform implementation decisions.
Twaweza’s interventions and initiatives will have the greatest chance of success if they are subjected to repeated testing, tweaking, adapting. An essential component to this is setting up feedback loops. These are tailor-made, small-scale measurement exercises with the following characteristics:
- They reach out to the target audience and measure the perceived quality, relevance, and usefulness of the intervention/initiative
- The data collected through them is actionable – something that the organization can change through the implementation
- Data is relatively easy to collect, easy to analyse – with either only internal, or limited external capacity needed
- They have a very quick turnaround from data collection to application, from findings to learning
The most innovative ideas and greatest initiatives will not result in any significant effects if they are not implemented widely enough among the selected audience/population. Twaweza endeavours, wherever possible, to measure coverage (or reach) of its initiatives. Mechanisms for this can vary, depending on the type of initiative that is implemented and can range from a nationally representative survey to a tailor-made exercise following just a selected sample of individuals. In addition, we also measure how much coverage Twaweza gets in the media and the quality of the coverage. This is one of the ways in which we can gauge whether we are influencing the tone and nature of national dialogue and debates, as presented through the media.
For some of our outputs an important measurement and validation exercise is the opinion or assessment by experts in a given field. These exercises are complementary to the feedback from the target audience; the purpose is to obtain an objective view of the quality and relevance of a product. Twaweza conducts a limited set of such exercises, and each is tailor-made for the particular output.
Assessing the link between outputs and intermediate outcomes is an essential component of Twaweza’s internal measurement structure. This area of work, sometimes referred to as process evaluation, is conceptually and logistically demanding, as it goes beyond tracking the actual outputs, and into measuring the possible effects these outputs can have on the intended target audiences. In many cases, we envision using mechanisms useful for Tier 2 Feedback loops to also assess feedback related to intermediate outcomes. Similar to Tier 2, Twaweza’s LME unit will establish shared core concepts and guidance on how to think about and plan for assessing intermediate outcomes and a set of standards, mechanisms and tools (including feedback loops) tools for implementation.
We want to make it easy to follow the results chain from Twaweza via partners to changes reflected in media and trace changes backwards, to see links and synergies, or to search by sector, network or goal. We are aiming to develop a web interface that will communicate monitoring information and allow the public, external evaluators, and our partners and donors to input and access data, generate reports, thereby facilitating others to combine different sources, create new knowledge and share lessons.