Guest post by James Habyarimana and William Jack of Georgetown University Initiative on Innovation, Development and Evaluation (gui2de). See here for more details of Twaweza's partnership with gui2de.
In 2013-14, gui2de and Twaweza collaborated on a research project which asked a question fundamental to Twaweza’s theory of change: does compelling mass communication contribute to positive behavior change among citizens? This question has been asked before – but as with much social science, evidence tends to be mixed; moreover, Twaweza wanted to ask in relation specifically to a televised Kenyan program called Makutano Junction, which we had supported for a number of years. By all accounts, Makutano Junction is a high-quality show with a broad viewership (see e.g., this review in New York Times). But can we say more about whether and how this show – and others of its type – might manage to persuade individuals to undertake civic engagement action? The post below introduces the research and some of the rather puzzling first-line findings; a following post will look in more detail at the measures used in the research. A report detailing all the main results will be posted shortly.
Television has been used since its inception to deliver messages intended to education or influence individual behavior – be it by the BBC or the Coca Cola Company. The multi-billion dollar advertising industry is surely evidence that well-crafted communication can affect the way we think and act.
Recently, a growing literature has looked at the potential for videos and other forms of “edutainment” (an unfortunate word) to influence economic behaviors, aspirations, and outcomes. For example, Berg and Zia have measured the effectiveness of a soap opera on financial literacy and behaviors in South Africa. And Tanguy Bernard, Stefan Dercon, Kate Orkin and Alemayehu Taffesse identified a causal impact of documentary videos in Ethiopia showing how perseverance and determination could lead to improved economic outcomes on viewers’ aspirations in general and their children’s education in particular. Other studies have looked at the portrayal of women in soap operas affects viewers’ attitudes behavior.
While the role of the media in educating, informing, and inspiring individuals about ways to do better for themselves – be it by avoiding onerous debt or simply aspiring to reach greater heights – is beginning to be better understood, whether these channels can be used to effectively motivate collective action has received little attention. If individuals are to be motivated to solve coordination problems through civic participation, we face not only an educational challenge, but the arguably deeper problem of finding ways to encourage and sustain cooperation.
This hasn’t stopped some people from trying, like the producers of Makutano Junction, a Kenya soap opera, into the storylines of which issues such as income-generation, mental and physical health, and rights and responsibilities of good citizens, are incorporated. To test if the show was effective in prodding individuals towards civic engagement, we conducted an RCT in Mukuru slum in Nairobi, home to about half a million people living in single story dwellings in a 2-3 square kilometer area.
Our objective was to randomly assign women to be exposed to Makutano Junction and to a placebo – a local soap opera that we were told was reliably devoid of social messages and commentary called Mother in Law – and to compare various outcomes both across groups and with a control group who saw no videos. To limit spillovers and to minimize confusion, we clustered assignment on a geographical basis. To this end, using hand-held GPS units, we first mapped the perimeter of Mukuru, and any interior uninhabited areas (such as swamps, open fields, and garbage heaps). The resulting populated area was then segmented into 150 circles of radius approximately 50m. The GPS coordinates of the center of each circle were recorded, and each circle was then randomly assigned to treatment, placebo, or control status.
We then recruited up to 50 women from each circle, and invited them to view videos at specific times in specific venues that we had previously contracted (mostly churches). Women were required to share with us their national ID number, and used the ID as an entry ticket to the videos, ensuring compliance with the random assignment. Nearly 6,000 women – about 2,000 in each experimental group – were recruited.
Although there was some initial interest in the videos, 45 percent of those recruited attended no shows, 25 percent watched 4 or more, and only 4-4.5% from each group watched at least 10 of the 12. The small gifts that we gave to all attendees, such as hand lotion and soap, were not powerful enough to generate larger attendance rates.
One of our outcome measures was the willingness of women to sign an on-going petition that was being administered by a local NGO. The petition, which demanded better sanitation services for residents, was conducted independently of our research, but we were able to observe how participation was correlated with the assignment to different groups in our experimental. The idea was that signing the petition was a privately costly action, since a signatory was required to report her national ID number, and could potentially be exposed to retribution. But the marginal private benefit, and in particular the likelihood of being pivotal in any meaningful sense, was essentially zero.
To test whether motivational material or simply actionable information was more effective, we crossed the treatment and placebo arms with a simple community service message alerting viewers to the presence of the signature campaign and encouraging them directly to sign. This imperative was expected to have little effect on the other measures of agency that we documented in our end-line survey (the subject of a forthcoming post), but could conceivably have focused the attention of viewers on the signature campaign.
Our results were either disappointing, or intriguing, depending on how you think about it. Women in the control group had about an 80 percent likelihood of signing the petition if asked. Exposure to each of the video treatments – both treatment and placebo – appeared if anything to reduce women’s willingness to sign the petition. However the effects were statistically insignificant, and there was no difference between treatment and placebo groups. More importantly, those assigned to receive specific information about the signature campaign were about 10 percent less likely to sign (significant at the 5 percent level).
In this case at least, motivating and inspiring individuals to act in the public interest through storylines and group activities (watching either treatment or placebo videos) did not seem to have much of an effect. And the intervention that might have been more likely to move people – a simple informational imperative – may well have back-fired.
So we face a challenge – tangential motivation through general storylines might be ineffective, or at least sufficiently subtle to be difficult to detect in an experiment with low compliance rates, but direct instruction can lead to unintended consequences.
Sometimes, research highlights puzzling and unexpected results. Rather than brush them aside, we take the opportunity to think more and deeper about behavior change, which we know is conditioned by a range of factors, both internal to an individual as well as external (contextual). Among factors that behavioral interventions commonly try to address are individual attitudes and perceptions, based on the theory that these can promote or hinder desired behavior change; a following post will examine some of the nuances and challenges in influencing and measuring these key determinants of behavior in the context of the Makutano Junction study.
You might also like...
- Learning Note 4: Measuring impacts of Makutano Junction (31 Jul 2015)
- Learning Note 5: Citizen perspectives on politics — a qualitative study in context of national elections (part 1) (7 Sep 2015)
- Learning Note 2: A peek inside a panel survey (18 May 2015)
- Learning Note 1: Follow-up survey of Tanzanian communities (8 May 2015)
- Learning Note 8: Understanding Citizen Preferences for Political Candidates | An Experiment in Rural Tanzania (5 Feb 2016)
- Learning Note 6: “Politicians all make the same promises”: Citizen perspectives on politics from a qualitative study (part 2) (18 Sep 2015)
- Three experiments to improve learning (26 Jun 2012)