Learning Note 1: Follow-up survey of Tanzanian communities

By Varja Lipovsek

As I write this, seven teams of survey enumerators are arriving in communities in Tabora, Morogoro, Mbeya, Lindi, Mwanza, Manyara and Kagera. They know exactly where they are going: similar teams visited the same communities five years ago, in 2010. Once the teams get there, they will first find a community leader (village executive officer, or similar) to introduce the teams, explain the exercise, and conduct an interview with the leader. Then, they will split up, with interviewers looking for the same ten households which were interviewed in 2010, during the baseline exercise. They will also visit the primary school and basic health facility which most people in the community use, and which were surveyed at baseline. By early June, the teams will have visited 250 communities across Tanzania, repeating the exercise in each one (see map below). The communities were selected at random; these four types of interviews together – the community leadership, school, health facility, and ten households – will provide rich information on how life in these communities, and in Tanzania overall, has changed since 2010.

Why are we doing this?

Back in 2010, Twaweza partnered with Amsterdam Institute of International Development (AIID) to design and begin implementing an innovative research study, with two main objectives. The first was to understand and describe Tanzanian citizen’s experience with public sector services, and their interaction with authorities, particularly around education, health and water. The second objective was to assess, some years later, whether there have been changes in citizen’s experience with the public sector, in the quality and quantity of available information, and in interaction between citizens and authorities. In order to detect changes, at least one other measurement must be made – so the survey currently in the field is the follow-up to the 2010 baseline. 

But the innovativeness of the AIID design wasn’t in having a baseline and follow-up surveys; it was in a unique combination of quantitative pre-post data with a continuous collection of “soft” qualitative data about life in the surveyed communities. This qualitative setup was called Sikiliza, or “listening device”. Under Sikiliza, graduate students from Dodoma University in Tanzania conducted monthly conversations with respondents from the same locations as had been sampled in the baseline study. Although data collection took some time to set up properly, Sikiliza ultimately collected nearly 2 years of rich contextual information from over 200 communities. This qualitative information has been entered into a database (Atlas.ti) and extensively coded (over 500 codes) on a variety of subjects focusing on citizen agency, the three public service sectors of interest, as well as on the interaction between citizens and authorities, and the result of the interaction.

The original study design (found here) hinged on the "book-ends" surveys combined with the rich coded qualitative data. It was also meant to be an evaluation of the overall Twaweza initiative over the five year period. It was designed at a time when Twaweza was charging full steam ahead under the vision that it could engage five existing and far-reaching networks (media, mobile telephones, teachers, religious leaders and fast moving consumer goods) to the extent where these networks became efficient and effective vehicles of transmitting essential information to citizens about basic services, and citizens would use this information in new and imaginative ways to address problems in basic services (whether by holding government to account, or coming up with a different solution altogether). In the intervening years, we have learned much, including that such sustained engagement of the five networks is much harder than anticipated, and that even when information does flow it, by itself, usually isn’t enough to drive problem-solving in basic services (for a summary of our soul-searching, see here).

So what to do with the research study which was meant to evaluate that vision? After much deliberation, we decided to press ahead with the follow-up survey, even though we realize that it is not likely to be an evaluation of Twaweza’s work in the first strategic period. Nevertheless, it is a deeply meaningful exercise, for a number of reasons:

  1. Without the follow-up, there is no story to tell. With the follow-up, the changes between 2010 and 2015 in the core indicators can be measured, and the Sikiliza data can be "threaded" in between these large surveys; the uniqueness of the research only has a chance to shine if the follow-up survey takes place.  
  2. While perhaps not evaluating Twaweza directly, the study will be able to test questions which were core to Twaweza’s first strategic period, many of which are still core today: i.e., is there an “echo” of Twaweza’s information spread (e.g., on learning outcomes); to what extent does more and better information available to citizens contribute to citizen agency; and to what extent does it contribute to greater responsiveness from authorities.
  3. It is a panel design. This means that not only is the follow-up to be conducted in the exact same communities, but the exact same facilities (schools, health centers) and the exact same households, to the extent possible, are to be interviewed again. Whether it’s about Twaweza or not, the panel design gives the study unique power (in a statistical, as well as conventional meaning) to tell the story of how life in Tanzanian communities has changed over nearly five years.

So the sampling plans have been reviewed and updated, questionnaires have been revised and piloted, the survey teams have been trained and re-trained, fieldwork plans have been put in place and, starting two weeks ago, the follow up data collection began. It is expected to continue until early June.

In the next learning note we’ll explore some of the devil-is-in-the-details of a conducting a panel design study. 

Read more: evaluation



You might also like...