Impact Blogs: The Power of Combining Ex-Ante and Ex-Post Impact Analysis

The combined application of ex-ante and ex-post impact analyses for the same investment mitigates the friction typically associated with solely applying ex-post impact analysis. This multi-pronged approach will also yield the most comprehensive process for generating a pipeline of research questions to shape right-fit ex-post studies.

Many international development institutions are trying to expand the role of evidence and analysis in informing their decisions. Examples include the United States Agency for International Development (USAID) and World Vision Canada (WVC), two organizations with which Limestone regularly works. World Vision’s global partnership boasts an aggregate budget of just above $3 billion, while USAID’s 2024 annual budget is approximately $20 billion, and both organizations have considerable autonomy in identifying, selecting, or designing interventions for a substantial portion of their portfolio. Therefore, institutional parallels exist regarding the endeavor to enhance evidence generation and utilization.

Upon his appointment as the Chief Economist of USAID in 2022, Dr. Dean Karlan swiftly prioritized the generation and use of evidence within his office. This emphasis was not unexpected, given Dr. Karlan’s reputation for advancing experimental methods and behavioral economics to identify effective strategies for poverty alleviation. His roles as academic researcher, the founder and president of Innovation for Poverty Action (IPA), and member of the Executive Committee of the Board of Directors of MIT’s Jameel Poverty Action Lab (J-PAL) underscore his pivotal contributions to advancing the use of scientific evidence in combating poverty.

In the months before this announcement, WVC enlisted me and my colleagues at Limestone Analytics to help bring evidence into decision-making across WVC’s broad project portfolio. This collaboration has proven invaluable, as both WVC and Limestone learn about and work to address challenges for increasing effective and efficient generation and use of evidence within multifaceted institutions. These collective endeavors have compelled us to innovate and venture beyond our accustomed boundaries, resulting in a mutually enriching partnership.

What is Impact Analysis? Impact analysis is a general term that captures a myriad of quantitative and qualitative analyses leveraging existing or new data to produce evidence and insights regarding the outcomes and impacts of the project, program, or policy. The impact analysis toolbox encompasses performance, impact evaluations, and value-for-money and cost analyses. While most institutions regularly undertake some form of performance evaluation to ensure accountability, the use of impact evaluations, cost analyses, and value-for-money assessments tend to be more sporadic, with institutional support for such studies varying across organizations and over time. Both USAID and World Vision fit this general profile. While the execution of impact analysis may vary across different branches of these organizations, neither mandates anything beyond performance evaluation as a standard operating process that applies to all activities.

Impact analysis can occur before, during, or after implementing a project or program. Typically, ex-ante analysis aids in design and feasibility assessments, while ex-post analysis, often carried out by a distinct unit or contractor, serves purposes of accountability and learning. Some institutions like the Millennium Challenge Corporation (MCC) have made significant strides in seamlessly integrating ex-ante and ex-post impact analysis into their operations, completing analyses at planning stages, and then updating and expanding the analyses as implementation data and evidence of impact become available. Many other organizations plan for and conduct these analyses separately. Our past work for USAID, including the 2023 assessment and the development of other guidelines for integrating ecosystem service values and gender and social inclusion into economic analysis, has consistently centered on the selective application of “right-fit” studies while systematically identifying knowledge gaps to develop a continuous pipeline of research questions. This approach must align with the current strategy of the USAID OCE, reflecting the “Right-Fit Evidence” approach coined by Dean Karlan.

Over the past two years, we have analyzed data and evaluations and conducted cost and value-for-money analyses of over 300 investments funded by World Vision Canada. Impact analysis at this scale is possible due to WV’s operational model (relying on a handful of customizable intervention recipes called core program models or CPMs) and their robust data collection with standardized performance data. One of the key lessons from this broad application of impact analysis is discovering new advantages in conducting both ex-ante and ex-post studies, such as cost-effectiveness and cost-benefit analysis, on the same investment. The ex-ante application relies primarily on secondary data, while the ex-post analysis can rely on primary data from performance and impact evaluations.

What have we learned? Unsurprisingly, subjecting an investment to ex-ante AND ex-post impact analysis (as opposed to ex-post OR ex-ante) maximizes the learning outcomes and completes the accountability cycle. However, in our work with WVC, we have discovered hidden benefits in the process. Applying ex-ante and ex-post impact analysis on the same investment will:

• Mitigate the friction typically associated with solely applying ex-post impact analysis. Ex-post studies, especially impact evaluations, can report unfavorable results (no or negative impact) and upset those involved in advocating, designing, funding, and implementing the interventions. Given the complexities associated with experimental studies in the real world, it can be challenging to pinpoint the underlying cause of such results. The cause could be the original theory of change, issues with implementation, the design of the evaluation, or factors unknown to the evaluators that have limited the evaluation’s ability to measure results. Impact evaluations answer specific research questions about impact, but most cannot explain why the expected impact did not materialize.
  
  The publication or exposure of unfavorable results is threatening and can damage an organization’s reputation, raising questions about the organization’s ability to design and implement cost-effective interventions. Implementing both an ex-ante and ex-post impact analysis can help mitigate these concerns.
  
  An ex-ante impact analysis, such as a cost-effectiveness study using secondary evidence, helps validate the socio-economic justification at the design stage. In other words, the ex-ante study can help establish that, given the available evidence at the time, the organization made the correct investment decision. Under such circumstances, if the ex-post study shows unfavorable results, there is less worry about the institution’s ability to design cost-effective interventions, and the attention is refocused on the contextual parameters, implementation issues, or evaluation shortcomings that might have resulted in observed impacts that conflict with the expectations.

• Result in the most comprehensive and systematic approach to creating a pipeline of research questions for right-fit ex-post studies. A detailed ex-ante study typically includes assessing existing evidence on effectiveness and costs. It can pinpoint and rank the critical knowledge gaps for the ex-post studies to consider. These knowledge gaps can include evidence on the external validity of the impact, impact on specific subgroups, or questions about the cost of a similar investment in new areas. Other knowledge gaps can include questions about the longevity of the impact or sustainability of an asset or behavior change in the future when donors and the project are no longer present.
  
  This approach will be more comprehensive and systematic than desk studies or evaluability assessments focusing on specific effectiveness or costing aspects. In other words, when we question cost-effectiveness ex-ante, we are forced to scan all existing evidence on costs and effectiveness. This process can pinpoint the most pressing knowledge gaps for ex-post studies and even include recommendations about the relevant methods to consider (based on past studies), building an actionable pipeline of knowledge gaps to shape right-fit ex-post studies to follow.

Together with colleagues from World Vision Canada and Limestone Analytics, we will present more about the learning outcomes and the experience of working with WVC at an innovative session at the Annual Meeting of the Society for Benefit-Cost Analysis in Washington DC, to be held on March 19, 2024, at 8:30 am EDT. Stay tuned for more insights published by World Vision Canada or at Limestone’s Impact Blog.