What actually will work, or works now, and how?

Evaluation: taking the longer-term, bird’s eye view. We have defined a limited number of “big bets” – that is, hypotheses stemming from the three main areas of our work (demonstrating citizen engagement locally, enabling citizen voice to be heard, protecting civic space). These are anchored by the baseline analyses, will span two or more years of implementation, and will use mixed methods to describe the trajectory of the implementation (what happened), capture the outcomes observed (results), and elucidate why the outcomes are (or are not) observed by situating the results in the context in which they occur.

The evaluation of Twaweza focuses on outcomes and impact – How do we know what we achieved? Why did we/did we not succeed in a certain area? Does Twaweza’s theory of change work? It is built alongside Twaweza’s programs, from the beginning, following processes as they unfold.

Evaluation outcomes are categorized according to our theory of change, which specifies 11 intermediate outcomes. Guided by these categories, we develop specific measures for each evaluation. For all our evaluations, methods follow purpose; we use and mix qualitative and quantitative methods; we strive to include more than one source of information and more than one round of data collection. Read the full evaluation framework [link to PDF from strategy].

HYPOTHESIS 1: Animators, community representatives and local government champions increase the Ability, Motivation and Opportunities of citizens (in selected communities) to participate in governance and development decisions.

HYPOTHESIS 2: Through advocacy and communication, Twaweza demonstrates that citizen participation in local governance (including service delivery) leads to improved outcomes (in services, and governance); and in turn use this to influence sector ministries (will specify) to promote meaningful participation and incorporate the needs and aspirations of citizens in plans and implementation. Once the efficacy and success of participation is (re) demonstrated, there will be increased appetite and uptake.

HYPOTHESIS 3: Enabling contextualised and systematically collected citizen voices to be heard will influence public policy debates at the national and sub-national levels consequently resulting in better services and governance.

HYPOTHESIS 4: Advocacy (soft and hard versions of advocacy) at national level will prompt government to respond by conceding elements of a more conducive civic space environment (our measures here need to be nuanced to track small but significant changes – e.g. not repealing an entire law, but clarifying it in a way that leaves space open).

Some evaluations are conducted internally, i.e. Twaweza’s LME unit is in charge of the entire process, including terms of reference, selecting implementing partners as needed (consultants, data collection firms, etc.), developing and vetting tools and methodologies, finalizing the results and reporting. Other evaluations are conducted in partnership with external high-quality research partners, in which the research design, tools and products are a result of a collaborative process. These partnerships are selected based on the overlap of interest in an evaluation (research) question between the research team and Twaweza, and tend to fill the dual role of providing Twaweza with an evaluation as well as providing the research team with raw material which furthers knowledge in their field.

Broadly we will apply comparative analytics to compare situations before and after introduction of our interventions and to the extent possible, we will compare citizen agency (collective action) between places that experienced our interventions and comparable locations that did not experience our interventions. Detailed contextual narratives of the independent variable/s (I.e, our interventions) will be combined with qualitative and quantitative statistical analysis of dependent variable/s (i.e. intended and unintended outcomes of our interventions). To be able to conduct these comparative analyses, we will conduct:

• Baseline of Tanzania / Uganda / Kenya current status of civic space, laws, regulations, and a sense of trend (i.e. what we are working with / against), by early 2019.
• Baseline of the status of the media, also early 2019 (content, regulations, actions).
• Baseline in the geographic spaces / areas where we will focus the local work. Should include status of services, citizens (men, women, younger, older, etc.), local authorities; current status of citizen agency, the Ability, Motivation and Opportunities to participate meaningfully.
• A deep follow up of few selected places. In a few areas, we commit to follow for the full four years. We do our annual immersions there; we build relationships. These should be a mix of communities in which engagement is both successful and less so.

All our evaluations are characterized by a commitment to transparency. We mean transparency of the results – whether they are positive or negative assessments of Twaweza’s initiatives. We also mean transparency of methods, tools, and the raw data.