What data do we need for an economic evaluation of a social investment? Plan ahead to avoid disappointment!
A timely and helpful new guide by Aaron Schiff, PhD
Policymakers are increasingly cognisant of the need to reconcile resource use with evidence of societal impacts, as seen in social investment (NZ), evidence-based policymaking (Australia) and a proposed Office for Value for Money (UK).
Accordingly, social cost-benefit analysis (CBA) has an important role in the mix of evaluation methods. CBA is a method for evaluating gains and losses to society that are attributable to an intervention, by systematically counting up the value it creates (benefits) and the value it consumes (costs), valued in monetary units. I encourage greater use of this method in evaluation and policy analysis, usually in combination with other methods.
A recurring challenge in evaluating social investments is a lack of suitable data for ex-post economic evaluation. We sometimes find that a CBA is infeasible, though it could have been possible if the right planning and data collection had been put in place from the outset. This disappointment is avoidable, and now there’s a resource to help you avoid it.
This new guide by my colleague Aaron Schiff is for anyone planning to conduct a credible CBA. It provides essential advice on building in appropriate data collection during the design and implementation of programs, to enable CBAs to be conducted when the time comes to quantify, attribute, value, and evaluate their impacts and costs.
Who is this guide for?
This guide is for policymakers, program evaluators, social service managers, and researchers who need to assess the economic benefits and costs of social interventions. It sets out a series of considerations to help plan ahead and gather the necessary data.
In my evaluation work I’m seeing increased interest for economic or “return on investment” analysis of social intervention programmes. The ideal is a headline result like: “Each $1 spent on the intervention returned $X in benefits to society”.
It sounds simple, but in practice this is quite difficult to do well. A robust and credible economic or return on investment analysis for a social intervention is data-intensive. It needs suitable data to estimate changes in outcomes that were caused by the intervention, and dollar values need to be put on those changes.
If you want to do this type of analysis, I highly recommend planning it in detail before the intervention is implemented. This will help make sure that the necessary data on activity and outcomes is collected.
For example, outcomes for individual people may need to be tracked, which can involve issues of ethics and consent, as well as needing secure systems for capturing and handling data. You may need to find a comparison group, and you may need to capture baseline measures of outcomes before the intervention. It’s not always easy to reconstruct these things later from administrative data - at least you should check that you’ll have the administrative data needed and any limitations.
To help with planning, I’ve put together this brief guide to the data and information needed to do a robust economic or return on investment analysis for a social intervention. I hope it's useful!
Bottom line
This guide is an excellent resource for anyone looking ahead to an economic evaluation of a social intervention. I think it’s going to be very helpful and I'm pleased to be able to share it here. It's open-access so do grab a copy and share it with your networks.
Download a free copy
Schiff, A. (2024). Data and information for economic evaluation of social interventions. Schiff Consulting: https://www.schiff.co.nz/en/resources/data-for-economic-evaluation/
Could not agree more, it's so important anyone involved in designing, delivering or evaluating programs with social impacts has this on their desk. We'll be sharing this with our network, thanks for sharing!