Case Study: How Talking Books Support Participatory Approaches for Impact Assessments
Updated: Jun 23
At IAIA22, Amplio's MEL advisor Lisa Zook shared how the Talking Book's user feedback function informed the evaluation process for a UNICEF/Ghana Health Service project
It's a best practice to include program participants in the evaluation process. However, project teams often face barriers such as illiteracy, power dynamics, and geography. The Amplio Talking Book offers a unique way to collect, analyze, and ampliyf the voices and perspectives of community members in impact assessments.
In May, Amplio’s monitoring, evaluation, and learning (MEL) advisor, Lisa Zook, presented at IAIA22, a conference organized by the International Association of Impact Assessment. Her presentation focused on Talking Book user feedback as a participatory approach for impact assessments. She discussed impact assessment methodologies in the context of a Talking Book pilot project Amplio implemented in the Upper West Region of Ghana in partnership with Ghana Health Service and UNICEF.
Watch a video of Lisa's talk or read a summary below.
Note: the video volume is quite low at the start but adjusts after a few seconds.
Watch a video of Lisa Zook's IAIA22 presentation on a user feedback case study.
Partnering on a Pilot Project
In 2021, Amplio partnered with UNICEF Ghana and Ghana Health Service to launch a Talking Book pilot project at 15 Community-Based Health Planning and Services (CHPS) compounds in Jirapa District. The project involved working with community health workers to strengthen access to accurate, consistent, and engaging health education materials, including maternal and child health, WASH, and child protection topics. Other goals were to generate demand for health services and to build capacity for the community health nurses and volunteers who deliver health education.
For the CHPS pilot, community health nurses and volunteers used Talking Books to deliver health messages, facilitate discussion, and record community feedback.
The team designed an impact assessment to look at how the Talking Book helped the community health workers create more impact. Were they seeing improvements in rural health outcomes? Did the technology support or improve their work? Was the device a scalable method for strengthening the health system?
Impact Assessment Data Sources
Lisa said the team identified four data sources for the impact assessment. The first three would be considered traditional impact assessment methods. These included usage statistics, community health worker surveys, and household surveys. The fourth component was Talking Book user feedback and that’s where the team wanted to capture emergent themes and build participation into the assessment.
Talking Book User Feedback
User feedback is generated when program participants record their questions or commentss directly onto a Talking Book. Most feedback is from adults but depending on the program we also get feedback from children and youth. User feedback recordings are stored on the device until someone goes to the field and connects that device to a laptop or smartphone to collect it. The data is automatically uploaded to Amplio’s server the next time their laptop or phone goes online.
Amplio and its partners can access the user feedback recordings to hear directly from program participants in their own words and voices.
At Amplio, we talk about prompted and unprompted user feedback. People can pick up the Talking Book and record feedback on anything they want to say. We can also build some prompts into the content.
"At Amplio, we talk about prompted and unprompted user feedback. People can just pick up the Talking Book and record feedback on anything they want to say. We can also build some prompts into the content," Lisa said.
"Let’s say a project has a message on hand washing behavior. At the end of the message, we could ask 'What are your thoughts on hand washing?' or 'What are your barriers to hand washing?' and that would be prompted user feedback because we would direct the program participant to record their thoughts about a topic."
1,200 User Feedback Messages
For the CHPS pilot, Talking Books collected 1,200 user feedback messages after about six months. People asked a wide range of questions about health services, the cost of health care, corporal punishment, child protection issues, and more.
At that point, the team needed to decide how to process all of the information.
"Keep in mind, the user feedback was in local languages. We had to work with local language speakers to sort and process the recordings, and make sure the data was fed into the impact assessment, and more importantly, into the program design."
To process the user feedback, Amplio's product team tried using existing software systems (Transana and Nvivo) but found that they were too cumbersome for audio messages and too complex for the local language speakers to navigate.
So Amplio developed a web-based user feedback processing system. Analysts can log in to access all of the messages that are collected by a project. The dashboard shows how many messages have been analyzed and how many still need to processing.
The system randomly selects a user feedback message for the analyst to listen to. A short series of questions helps them categorize the message and identify issues and trends. Multiple analysts can log in simultaneously, and the questions can be customized to align with a program's monitoring and evaluation strategy.
Want to give it a try? Log in to Amplio's user feedback processing demo. Enter "email@example.com" for the email address and "TalkingBook2021" for the password. (Note: The password is case sensitive.)
Lisa said the new user feedback processing system helped the team overcome some of the barriers to participation that are common with traditional impact assessments, including issues with gender discrimination, language, and geography.
"We had access to community members and perspectives that we never would have had access to previously. We were able to identify emergent themes that broke us out of the traditional impact assessment framework. Instead of a rigid framework, where we didn't know the right question to ask, all of a sudden we had an open door for people to give their perspective on anything that was on their minds."
We had access to community members and perspectives that we never would have had access to previously. We were able to identify emergent themes that broke us out of the traditional impact assessment framework.
Amplio's user feedback processing tool will continue to evolve as more Talking Book partners and programs use it and provide their feedback. Lisa said the team is looking at artificial intelligence and/or how voice recognition could be used, in particular for more common languages, to help process messages more quickly. They are working on how to use visualizations to better present qualitative data. They are also exploring more intentional work with the analysts who process the user feedback.
Got a question or comment?
If you're interested in learning more about user feedback processing or love to talk about data, connect with Lisa on LinkedIn or email her at firstname.lastname@example.org.