How to choose methods for VfM assessment
What method or methods should you use to answer your value-for-money questions? On what basis should you decide?
So you have a value for money (VfM) question. And you have to choose how to answer it. What to do? How should you go about it?
This article follows on from last week’s, where I shared a few diagrams aimed at contrasting salient features of cost-benefit analysis (CBA), social return on investment (SROI) and value for investment (VfI).
The following article, like last week’s, aims to be practical and to-the-point. Feel free to follow the links provided for more detailed analysis and rationale behind the points below.
How to choose your methods
I recommend the following steps, working with stakeholders (including those affected by the investment and those who affect it) to:
Clarify VfM questions, how answers will be used, and by whom.
Describe the value of the investment in words. To whom is the policy, program, intervention, or organisation valuable? In what ways is it valuable? How is the value created? In other words, what is its value proposition?
Decide which aspects of the investment’s value are important to focus on (criteria).
For each of those aspects, describe how you would tell the difference between great value, good value, just enough value, or not enough value for the resources used (standards).
Consider what forms of credible evidence will be needed so that you can judge the value of the investment against the features described above.
Determine what method(s) should be used to gather and analyse the necessary evidence. These could include any bricolage or mix from the rich and diverse toolbox used in evaluation and economic analysis.
When we follow these steps, it often turns out that we agree we should consider more than one aspect of value, and that we need to collect and analyse more than one form of evidence. And that means we’ll need a framework to bring the different considerations together to make judgements that take all of the value criteria, levels of value, and evidence of value into account (i.e., evaluative reasoning). Rubrics and mixed methods belong together like peas and carrots.
For example
Many countries have Drug Courts, a therapeutically-oriented alternative to imprisonment for people with addiction issues who are convicted of drug-related offences. Drug Courts provide community sentences and intensive support services, with rewards and sanctions to encourage rehabilitation.
VfM questions and information needs: Decision-makers may ask whether a Drug Court represents good value for the resources used, and whether there are opportunities to create more value from the resources. The answers to these questions may help them to make decisions about future investments in the Drug Court, such as whether to continue, scale, adapt, or stop the program.
Value proposition: Drug Courts can be valuable to several groups in different ways. For example:
For offenders with substance use disorders, Drug Courts provide an alternative to traditional criminal justice processing, including access to treatment, opportunity to address root causes of their criminal behaviour, potential for reduced or avoided prison sentences, and support for long-term recovery, wellbeing and quality of life.
For the criminal justice system, Drug Courts can reduce recidivism rates compared to traditional court processing, alleviate overcrowding in prisons, provide judges and prosecutors with more options for addressing drug-related crime, and integrate treatment into the court process, allowing for closer monitoring.
For society and communities, Drug Courts offer the possibility of reduced costs compared to incarceration, reduced drug-related crime rates, improved public health outcomes (by treating addiction), keeping families together (by avoiding incarceration), and supporting offenders to become productive members of society.
For treatment providers, there’s value in a more coordinated, team-based approach to addressing addiction, increased accountability and engagement of participants in treatment programs, and opportunities to intervene earlier in the cycle of addiction and crime.
There’s more, but that’s enough to illustrate. You get the picture: the value of Drug Courts is multi-faceted. Some of the value might be relatively straightforward to express in monetary units (e.g., reduced costs compared to incarceration). Some might be moderately hard to monetise, but still possible (e.g., wellbeing). Some might be “too hard” (e.g., keeping families together). And that’s OK. Who said we have to monetise all the value anyway? That’s just one option.
Criteria: What aspects of value should we focus on? One aspect that’s usually quite important for VfM is impacts. Examples of impacts that past evaluations have attributed to Drug Courts include:
Reduced drug and alcohol use
Reduced criminal offending
Decreased incarceration
Cost savings to the criminal justice system
Increased social connectedness
Increased connections with cultural identity
Enhanced public safety
Improved inter-agency collaboration
Increased judicial expertise in addiction and recovery
Strengthened family relationships
Personal growth and long-term recovery
Improved quality of life
Increased personal responsibility.
In addition to impacts, other aspects of performance and value that could be relevant to the VfM of a Drug Court could include:
What resources are invested in it (monetary and non-monetary) and how well those resources are managed and marshaled to fuel organisational actions
The productivity of those organisational actions - for example: the quality of coordination, communication and working relationships between the agencies and teams involved in delivering the Drug Court program; the allocation of resources to the required mix of inputs and activities; delivery at an efficient cost per unit of output; working adaptively to increase productivity over time (and so on)
The equity of resource allocation, activities and outcomes (and supporting factors such as the cultural fit between participants and program delivery staff)
Whether the totality of impacts (monetary and non-monetary, measured quantitatively and/or qualitatively) represents enough value to justify the resources invested, bearing in mind alternative ways of achieving those impacts
And more. For a big-budget, high stakes evaluation we might be able to justify looking at all of these dimensions. For a quicker and lower-cost assessment we might prioritise a few key aspects.
Standards: What critical factors differentiate excellent, good, adequate and poor value on these criteria? Here we can start with generic definitions of different levels of value, and then translate the generic definitions into specific descriptions of what we would expect to see in the Drug Court. For example, our generic definitions could be something like:
Excellent: Exceeding expectations.
Good: Generally meeting reasonable expectations bearing in mind the context.
Adequate: Not meeting expectations, but fulfilling minimum requirements and showing acceptable progress overall.
Poor: Not fulfilling minimum requirements or not showing acceptable progress.
We can tailor these generic definitions to make them specific to each criterion. I have in mind to detail this step in a future article. For now, here’s an example: For the participant outcome of personal growth, perhaps stakeholders agree that a reasonable expectation is for more than half of participants to graduate with significant examples of personal growth and at least some people to sustain these elements of personal growth when followed up 12 months after graduating. And perhaps they agree that the minimum requirement to be considered just good enough is for around a quarter of participants to show examples of personal growth at the time of graduating. Therefore, stakeholders might agree to define:
Excellent as more than half of participants graduating with significant examples of personal growth that are sustained at 12-month follow up post graduation
Good as more than half of participants graduating with significant examples of personal growth, some of which are sustained at 12-month follow up
Adequate as around a quarter or more of participants graduating with examples of personal growth
Poor as substantially less than a quarter of participants graduating with examples of personal growth.
Evidence and methods: Now, look at all the criteria (e.g., impacts, stewardship of resources, productivity, equity, benefits relative to costs) and consider what evidence is needed and what methods should be used to collect and analyse the evidence. In this instance, we might decide that we need a mix such as:
An impact evaluation including quantitative and qualitative evidence to span the kinds of impacts listed above.
Analysis of administrative data, together with key informant interviews to address aspects of productivity and stewardship of resources.
Breakdowns of cost, activity, and impact data, and engagement with participants to understand how equitably the Drug Court is performing.
A social cost-benefit analysis, to examine the relative value of resources and impacts that are amenable to monetary valuation.
So what?
When we follow this approach, it helps to clarify what matters (criteria), what good looks like (standards) and what mix of methods is appropriate and feasible to address the criteria and standards. If you only need a CBA or SROI, great! This approach helped to provide a reasoned justification for that choice. If you need more than a CBA or SROI, also great! This approach clarified the information needs and helped guard against leaving important aspects of value on the table.
That’s what the Value for Investment (VfI) approach does. It isn’t a method. It’s a system for selecting and combining existing methods and tools from evaluation and economics in contextually appropriate ways. And it’s a system for combining criteria, standards and mixed methods evidence to make a holistic judgement. VfI isn’t a rival to CBA or SROI, but it can help you decide when and how to use them.
And those steps I outlined above? Here they are, embedded in the VfI approach.
I argue that the starting point for any evaluation of VfM should be to understand the context, questions, stakeholders, value proposition, and develop criteria and standards before turning to the question of what methods are needed. For these reasons, I argue VfI is a good fit for any situation where you need to answer a VfM question.
Of course, you don’t have to do it just ‘cos I said so. It’s a free world - and out there in the free world, I’m aware that CBA is often seen as the go-to method for answering VfM questions. I agree CBA is a valid and important method. I advocate for evaluators to use it more. However, I also argue that it’s not always enough on its own, and that methods should be selected according to context. This argument is consistent with Program Evaluation Standards, which caution against preordaining or imposing any solution as a “gold standard”.
But just in case your starting point is responding to a request for a CBA (for reasons that may not be within your control), here’s an alternative decision-making process that starts with CBA.
Another way to choose
Here are two flowcharts to help decide whether CBA is an appropriate method to use and, if so, whether it is sufficient on its own or better combined with other methods.
1. Is CBA an appropriate method to use in our situation?
2. Will CBA be enough on its own, or should we consider using it in combination with other methods?
If you conclude that you would like to combine CBA with other methods, then you can use the steps of the VfI process to determine the mix:
Bottom line
I am not a fan of hard-and-fast rules when selecting methods. Contextual decisions require human judgement. However, these guiding considerations are offered to help you decide. They’re based on doctoral research and practical experience. You can read more about them (and additional considerations) here, here, here, here, and here.
Your perspective, as always, is welcome.
Where’s this?
If you’re the first person to correctly identify the location of this photo in the comments below, I’ll shout you a free 12-month subscription to the full Evaluation and Value for Investment archive.
Thanks for a very helpful post. And that looks like Brisbane's South Bank bouganvilleas.