The Way Forward:
A Renewed Agenda for Investments in Impact Evaluation and the Broader Evidence Ecosystem
Images of Empowerment
Images of Empowerment
This section provides an overview of the working group’s understanding of good practice in the design and use of impact evaluation for better public policy and programs. It offers recommendations on “what and how” to fund to deliver on the promise of impact evaluations and bolster the broader evidence ecosystem. Throughout, we highlight examples of good practice and innovations in the field. The working group directed its recommendations to the development community—government policymakers, funders, researchers, and NGOs.
Box 4. Understanding the Return on Investment
Throughout the report, we consider impact evaluation evidence, when used to inform a decision on the design and/or implementation of a public policy or program, as a development intervention in and of itself. Generating evidence entails a cost and produces a benefit in the form of increased or faster impact on outcomes or cost savings that could be deployed to other policy uses. This value-of-information approach serves as a guiding principle for the working group and the recommendations presented in this report.
An impact evaluation’s tangible value materializes when it informs a resource allocation or implementation decision, such as preventing expenditure on ineffective interventions or informing the scale-up of interventions shown to improve and save lives. The key ratio is the benefit of the evaluation’s information to the cost of the evidence.
There is a growing set of examples where the use of evidence from impact evaluations led to policy changes with tangible impacts on people’s lives. Impact evaluations have shifted global thinking and practice on reducing user fees for preventive health products like bed nets and deworming medicine; expanded a cash transfer program to reach more poor households in Ghana; and informed the timely design of a “low-tech” remote education intervention during COVID-19 in Botswana and subsequently other countries, among many more examples J-PAL 2018; 3ie 2020; Youth Impact; see Box 5 for additional details.
Policymakers, researchers, and funders should begin to use a value-of-information approach to proactively identify potential high-value cases of impact evaluation, including in areas such as climate. Programs that have the potential to have the greatest impact on lives and/or receive significant public resources and could easily be evaluated but have not yet been, such as COVID-19 vaccine delivery, should also be prioritized. Prioritizing evaluations of programs and interventions that are most likely to improve lives is sorely needed (Altshuler 2022). By embracing a value-of-information approach to prioritize studies and policymaker engagement efforts, funders can collectively harness greater benefits from impact evaluation.
Going forward, the cost-effectiveness of impact evaluation evidence as an intervention for better outcomes and/or savings should be modelled to quantify the benefits in terms that may resonate with a broad base of funders. The value-of-information literature by Carlson et al. (2013), Claxton and Sculpher (2012), Fenwick et al. (2020), Gratia (2014), Macauley and Laxminarayan (2010), and Myers et al. (2012) provides a comprehensive starting point. The approach used by the Washington State Institute for Public Policy (2019) to model the potential return on investment of different programs based on existing evidence demonstrates a way to identify potentially high-value interventions, which could then be used to inform future research. IDinsight is also piloting a new value-of-information methodology to assess how much they expect social impact to improve based on additional information (Allier-Gagneur et al. 2022).
Box 5. Examples of Evidence Use and its Impact on Policy and Lives at Scale
These examples describe ways in which rigorous evaluation has been used to enhance policy and program impact, and reflect the range of nonlinear pathways to real-world applications. This list is not intended to be exhaustive but illustrative; policy impacts include changes to program design, program scale-up, program drawdown or closure, program adaption and design in other contexts, and influence on policy decisions via wider policy dialogue.
Many evaluations also help improve institutional cultures of evidence use by demonstrating benefits, increasing policymaker interest and strengthening capacity (see recommendation #4), but the focus in examples highlighted here is on more direct pathways to real-world impact. While examples are numerous and tracking of use is increasingly recognized as a main component of evidence systems, the social, health and economic benefits of information generated from impact evaluation (in relation to the cost of the study) are only scratching the surface of what is possible.
- Allocating more resources to beneficiaries of a youth livelihood program in Uganda
- Boosting learning outcomes through Teaching at the Right Level in India and Africa
- Combatting COVID-19-induced poverty through community-based cash transfer targeting in Indonesia
- Combining HIV prevention campaigns to address the needs of different populations in Senegal
- Discontinuing rainfall risk insurance to support farmers in India in more effective ways
- Enhancing farmer productivity and livelihoods in India
- Expanding and broadening eligibility of a cash transfer program in Colombia
- Expanding cash transfers across sub-Saharan Africa to reduce poverty
- Expanding the child support grant and youth employment tax incentive in South Africa
- Improving children’s health through behavioral nudges in the Philippines
- Improving environmental and human well-being through protected areas and payments for environmental services in Cambodia
- Improving marginalized groups’ financial practices and outcomes through new technology in Latin America
- Improving Pakistan’s income support program
- Increasing cash in hand among poor households in Ghana
- Increasing take-up of public benefits programs in the United States
- Influencing major labor law reforms in Mexico
- Maintaining funding for road rehabilitation in Peru
- National adoption of food fortification policy in Uganda
- Preventing COVID transmission through increased mask use in India
- Promoting HIV self-testing among sex workers in Zambia
- Reallocating investments for better early child care in Colombia
- Recommendation of self-testing for HIV by the World Health Organization
- Reducing school dropout rates in Peru
- Safeguarding education progress during COVID-19 through a “low-tech” education intervention
- Saving resources in Peru, Chile, and Mexico based on shortcomings of “One Laptop per Child” program
- Scaling down a financial literacy movie in Nigeria to allocate financial literacy program resources to more effective uses
- Shifting global thinking and practice on free bed nets to fight malaria
- Shifting the focus of a youth program in the Philippines to enhance employability
Sources: Examples collated from Fishman and Christiano 2020, IEG 2012, Manning et al. 2020, OVE 2017, Legovini et al. 2019, Davis et al. 2016, 3ie Evidence Impact Summaries, IPA’s impact case studies, J-PAL’s case studies on pathways to policy change, and IDinsight’s featured projects.