The Way Forward:

A Renewed Agenda for Investments in Impact Evaluation and the Broader Evidence Ecosystem

Images of Empowerment

Images of Empowerment

This section provides an overview of the working group’s understanding of good practice in the design and use of impact evaluation for better public policy and programs. It offers recommendations on “what and how” to fund to deliver on the promise of impact evaluations and bolster the broader evidence ecosystem. Throughout, we highlight examples of good practice and innovations in the field. The working group directed its recommendations to the development community—government policymakers, funders, researchers, and NGOs.

Box 4. Understanding the Return on Investment

Throughout the report, we consider impact evaluation evidence, when used to inform a decision on the design and/or implementation of a public policy or program, as a development intervention in and of itself. Generating evidence entails a cost and produces a benefit in the form of increased or faster impact on outcomes or cost savings that could be deployed to other policy uses. This value-of-information approach serves as a guiding principle for the working group and the recommendations presented in this report.

An impact evaluation’s tangible value materializes when it informs a resource allocation or implementation decision, such as preventing expenditure on ineffective interventions or informing the scale-up of interventions shown to improve and save lives. The key ratio is the benefit of the evaluation’s information to the cost of the evidence.

There is a growing set of examples where the use of evidence from impact evaluations led to policy changes with tangible impacts on people’s lives. Impact evaluations have shifted global thinking and practice on reducing user fees for preventive health products like bed nets and deworming medicine; expanded a cash transfer program to reach more poor households in Ghana; and informed the timely design of a “low-tech” remote education intervention during COVID-19 in Botswana and subsequently other countries, among many more examples J-PAL 2018; 3ie 2020; Youth Impact; see Box 5 for additional details.

Policymakers, researchers, and funders should begin to use a value-of-information approach to proactively identify potential high-value cases of impact evaluation, including in areas such as climate. Programs that have the potential to have the greatest impact on lives and/or receive significant public resources and could easily be evaluated but have not yet been, such as COVID-19 vaccine delivery, should also be prioritized. Prioritizing evaluations of programs and interventions that are most likely to improve lives is sorely needed (Altshuler 2022). By embracing a value-of-information approach to prioritize studies and policymaker engagement efforts, funders can collectively harness greater benefits from impact evaluation.

Going forward, the cost-effectiveness of impact evaluation evidence as an intervention for better outcomes and/or savings should be modelled to quantify the benefits in terms that may resonate with a broad base of funders. The value-of-information literature by Carlson et al. (2013), Claxton and Sculpher (2012), Fenwick et al. (2020), Gratia (2014), Macauley and Laxminarayan (2010), and Myers et al. (2012) provides a comprehensive starting point. The approach used by the Washington State Institute for Public Policy (2019) to model the potential return on investment of different programs based on existing evidence demonstrates a way to identify potentially high-value interventions, which could then be used to inform future research. IDinsight is also piloting a new value-of-information methodology to assess how much they expect social impact to improve based on additional information (Allier-Gagneur et al. 2022).

Box 5. Examples of Evidence Use and its Impact on Policy and Lives at Scale

These examples describe ways in which rigorous evaluation has been used to enhance policy and program impact, and reflect the range of nonlinear pathways to real-world applications. This list is not intended to be exhaustive but illustrative; policy impacts include changes to program design, program scale-up, program drawdown or closure, program adaption and design in other contexts, and influence on policy decisions via wider policy dialogue.

Many evaluations also help improve institutional cultures of evidence use by demonstrating benefits, increasing policymaker interest and strengthening capacity (see recommendation #4), but the focus in examples highlighted here is on more direct pathways to real-world impact. While examples are numerous and tracking of use is increasingly recognized as a main component of evidence systems, the social, health and economic benefits of information generated from impact evaluation (in relation to the cost of the study) are only scratching the surface of what is possible.


 
Read less

Navigation

TopBuilt with Shorthand