Beyond Coverage: Measuring Vitamin A Supplementation Program Effectiveness in Mauritania and Sierra Leone

Research Project

Beyond Coverage: Measuring Vitamin A Supplementation Program Effectiveness in Mauritania and Sierra Leone

Photo credit: UNICEF

Summary

UNICEF conducted strategic reviews of national vitamin A supplementation programs in Mauritania and Sierra Leone that explored how such programs can best evaluate the effectiveness of their efforts.

Mauritania and Sierra Leone

Vitamin A Supplementation

Promising Practices

Engage stakeholders in strategically reviewing a program’s effectiveness through participatory workshops.
Use a conceptual framework to guide stakeholder discussion on effectiveness parameters.
Select diverse strategic review participants who represent governmental and non-governmental entities.

Key Messages

To gather insight and recommendations on how vitamin A supplementation programs can evaluate their effectiveness and identify necessary improvements, the United Nations Children’s Fund (UNICEF) engaged stakeholders in strategic reviews of the national programs in Mauritania and Sierra Leone. Using a mixed-methods and participatory approach, researchers structured these reviews according to the parameters of the UNICEF delivery effectiveness framework.

During the strategic reviews, stakeholders provided input on the importance of each parameter and the perceived level of difficulty to improve it. They also discussed current performance, challenges, opportunities, and recommendations for improving program effectiveness.

By participating in the strategic reviews, stakeholders in both countries moved beyond using coverage as the sole metric for measuring effectiveness and considered additional effectiveness parameters.
The strategic reviews offered a forum for stakeholders to discuss locally relevant indicators, identify and begin to resolve obstacles to effective delivery, and agree on desired outcomes.
Context-specific recommendations for enhancing vitamin A supplementation effectiveness emerged from these strategic reviews.
These case studies of Mauritania and Sierra Leone can serve as a model for other public health programs to improve their effectiveness evaluation processes.
Processes and methodologies for strategic reviews are context-specific, and more research is needed to determine how to tailor approaches to other countries and programs.

Background

Context

In recent years, preventive health services like vitamin A supplementation have helped to facilitate improved health outcomes, including decreased child mortality rates and increased life expectancy. To maximize the positive effects of such interventions, it is critical to evaluate the effectiveness of these programs to identify needed improvements.

Problems or Gap

Evaluating the effectiveness of public health programs can be challenging due to overreliance on coverage data as the sole indicator of effectiveness. Differing stakeholder perspectives on what constitutes effectiveness can also complicate evaluation. Often, programs overlook the importance of reaching stakeholder consensus on desired program outcomes. In response to this challenge, this research project investigated ways in which programs can leverage available data to examine multiple aspects of effectiveness, as well as promote consensus among stakeholders on how to define effectiveness in specific settings.

Study Objectives

The objectives of this research project were to:

Develop and test approaches to strategically review the delivery effectiveness of vitamin A supplementation programs using the delivery effectiveness framework in two case-study countries (Mauritania and Sierra Leone).
Explore generic definitions and indicators for effectiveness parameters in the case-study countries.
Conduct desk reviews to map available data in the case-study countries.
In country-specific workshops, build consensus around the prioritization of effectiveness parameters and recommendations.
Collect and analyze lessons learned from the strategic review of vitamin A supplementation program effectiveness in the case-study countries.

View the slides and research brief for detail on the research methods.

Results

Summary

The findings of these two case studies suggest that a strategic review of program effectiveness can lead to better decision-making about program improvements and thus promote beneficial program outcomes. In both strategic review workshops, the conceptual framework and iterative workshop format enabled participants to discuss multiple effectiveness metrics and reach consensus on desired outcomes.

Key Findings

Participants’ responses regarding the importance and difficulty level of improving each parameter revealed the following findings:

Stakeholders in both countries viewed most of the UNICEF delivery effectiveness framework parameters as somewhat or very important.
Across both countries, participants characterized access, availability, service quality, and sustainability as having high importance.
Among these highly important parameters, both sets of stakeholders perceived the availability of health workers and supplies as being the easiest parameter to improve, whereas sustainability was considered difficult to improve.
Compared to participants from Mauritania, stakeholders in Sierra Leone characterized service quality and access as more difficult to improve.
Perceptions of the importance of equity differed noticeably by country. Stakeholders in Sierra Leone considered equity highly important and somewhat challenging to improve. Participants from Mauritania also perceived this parameter as somewhat challenging but viewed it as less important.
In Sierra Leone, stakeholders labeled clinical outcomes as less important and difficult to improve. Conversely, participants in Mauritania viewed clinical outcomes as both highly important and easy to improve. The countries follow different delivery approaches, which may contribute to the divergent viewpoints on the importance and difficulty of improvements to clinical outcomes.

In each strategic review, participants provided more than 150 recommendations on ways to enhance program effectiveness. Four broad themes emerged from their responses:

Increased human and financial resources
Expanded program capacity
Enhanced integration with other programs and campaigns
A more supportive political environment

Consensus findings in Mauritania

Consensus findings in Mauritania

Consensus findings in Sierra Leone

Consensus findings in Sierra Leone

“Service quality and equitable distribution will take a while to improve.”
—Sierra Leone study participant

Conduct a strategic review of a program’s effectiveness by engaging stakeholders in an interactive workshop to discuss and reach consensus on effectiveness indicators.
Prior to the workshop, allow sufficient time to gather, compile, and synthesize data on effectiveness for workshop participants to review and discuss.
Use a conceptual framework to structure the strategic review and guide discussion.
Recruit diverse stakeholders to participate in the strategic review, including community participants.
To address any gaps in a program’s data, review data from programs that follow similar delivery approaches.
Facilitate small group discussions for each effectiveness parameter to promote consensus-building among stakeholders.
Offer participants an opportunity to refute the validity or relevance of the data proposed by the project team.
To reach consensus, use simple methodologies for which results can be quickly calculated.

Photo credit: UNICEF

“[Vitamin A supplementation] outcomes depend directly on access, availability, and sustainability of service delivery. […] I think these are the most important factors that need to be improved.”
—Mauritania study participant

Research Project