Value for Investment: Application and Insights
New open access guide from NZ
I’m excited to share this new document, which provides guidance on how to design and implement evaluations that provide better answers to value for money questions.
This open-access guide describes the principles and processes underpinning the Value for Investment (VfI) approach and a worked example of the approach in action.
I think this guide has the potential to be a significant resource for people assessing value for money (VfM) in health, social and other sectors. Similar to the way OPM’s Approach to Assessing Value for Money has influenced VfM assessment in the development sector, this new document offers guidance to implement VfI principles and processes in domestic settings.
Many thanks to Drs Alicia Crocket, Adrian Field and the Dovetail Consulting team for the great work on the Youth Primary Mental Health and Addictions evaluation and Te Whatu Ora for making this guide possible.
What’s in the guide?
This new document provides:
An overview of the VfI approach, including the conceptual principles and processes underpinning it
A worked example of the approach in action: evaluating the Youth Primary Mental Health and Addictions initiative in New Zealand
Transferrable learning for others considering the use of the VfI approach.
What’s Value for Investment?
Value for Investment is an evaluation approach. It re-frames value for money as good resource use. This framing shifts our focus from the money to a broader perspective that views policies and programs as investments in value propositions with potential to create significant social, cultural, environmental or economic value.
The VfI approach positions resource use and value creation as questions that evaluators already have the methods and tools to answer. It offers evaluators a set of guiding principles together with a stepped approach, and encourages evaluators to bring their full evaluation toolkits and match methods to context.
How did this guide come about?
During 2021-2023, an evaluation team led by Dovetail Consulting evaluated New Zealand’s Youth Primary Mental Health and Addictions (Youth PMHA) initiative, using the VfI system.
The Youth PMHA initiative was established as part of a program called Access and Choice. It aims to meet the needs of rangatahi (young people) aged 12-24 years who are experiencing mild to moderate levels of mental health and/or addiction needs. You can read more about Youth PMHA here.
The evaluation addressed three questions:
How does the Youth PMHA create value?
To what extent does the Youth PMHA provide good value for the resources invested?
How could the Youth PMHA provide more value for the resources invested?
We produced this guide to illustrate the use of the VfI system in the Youth PMHA context. You can also read a summary of the Youth PMHA evaluation here, and the full evaluation report here.
Reflection
The following paragraphs, drawn from the guide, reflect on insights gained from the use of the VfI system in the Youth PMHA evaluation:
Value for investment (good resource use) is a shared domain of evaluation and economics. The Value for Investment system builds evaluative and economic thinking into each step of the evaluation process, informing theory-building, criteria and standards, collecting and analysing credible evidence, evaluative judgements, and reporting. Although economic methods of evaluation (such as cost-benefit analysis) are often used in VfI, they are not always necessary, desirable or feasible. Methods are determined according to context.
Rubrics (contextually-defined criteria and standards) are the backbone of the VfI system, supporting clear reasoning at each step of the process. Developing rubrics brings stakeholders to the table to co-define what ‘good resource use’ looks like in context. Rubrics delineate the scope of the evaluation and help to clarify what evidence needs to be collected. They provide a framework for organising the evidence so that it is efficient to analyse. They provide a shared, agreed set of lenses for collaboratively making sense of the evidence and for evaluating resource use and value creation. Rubrics provide a structure for reporting findings, based on the aspects of resource use and value creation that are agreed to be important.
The process of co-developing and using rubrics supports utilisation-focused evaluation by helping stakeholders to gain a deeper understanding and experience of evaluation. This increases the likelihood that findings will be understood, endorsed, and used. Developing rubrics is an art and a science, requiring skills in facilitation, cultural competence and knowledge, robust conceptual thinking, wordsmithing, and graphic design. Having the right mix of skills on the team and the right mix of stakeholder perspectives in the room are paramount.
A theory of change helps to clarify relationships between a programme’s activities, outputs and outcomes, but it isn’t usually explicit about what is seen as intrinsically important in a programme, nor how it creates value. Adding value creation concepts to a theory of change helps to clarify what kinds of value are created, to whom and how, and helps to identify factors affecting the ability of a programme to use resources efficiently, effectively, equitably and to create sufficient value to justify the resources used. This brings extra clarity to rubric development through focusing on the elements that are seen as critically important to delivering value.
The VfI system encourages the use of mixed methods, whereas traditionally VfM questions have often been addressed using predominantly quantitative data and economic methods of evaluation. By blending economic concepts with broader approaches to evidencing and valuing performance, the VfI system allows the evaluation to integrate different forms of evidence in an intentional way, and to justify the selection of methods by explicitly identifying multiple forms of evidence needed to address each criterion. This helps the evaluation to unpack ‘the story behind the numbers’ and present a more nuanced assessment. It also permits robust, transparent assessment of VfI in complex and hard-to-measure contexts.
Ultimately, clients and stakeholders appreciate the findings of a VfI evaluation because explicit evaluative reasoning, embedded in the principles and process underpinning VfI, provides a pathway for evaluators to provide clear answers to value questions. The VfI process facilitates an evaluation that commences with an understanding of the programme and the system in which it is situated. It helps stakeholders to think clearly about what they are trying to achieve, the value proposition of the investment, and what success and good VfI look like. Because clients and stakeholders are engaged throughout the process, the evaluation is more than just a report; the conversations that take place throughout the evaluation add value and help to inform decision-making long before reports are written. These processes, in turn, guide the evaluation team to deliver reports that are cogent, accurate and useful.
Download the Guide
The new open-access guide is available here.
Know thy impact (Hattie, 2019)
Through this series of Substack articles, my ambition is to contribute to developing the field of value for money assessment, empowering you to use evaluation’s rich and diverse bag of tools in rigorous ways to provide better answers to value for money questions.
Substack gives me basic metrics like how many times my posts are read (5-6,000 per month) and how readers access them (43% email, 29% browser, 24% LinkedIn, 4% other).
What the stats don’t show is the one thing I really want to know: my impact. If one post helps one person approach an evaluation from a fresh perspective, it makes my day. If this is you, please leave a comment or drop me a line. And if something feels confusing, or warrants a deeper dive in a future article, I want to hear that too. All feedback, questions and suggestions are welcome!
Conference presentations
If you’re attending the Australian Evaluation Society (Brisbane) or The UK Evaluation Society (online) conferences, Adrian Field and I will be sharing the story of the Youth PMHA evaluation and the collaborative co-design process that underpinned it. Hope to see you there!
Need wellbeing support?
If you or someone you know needs wellbeing support in New Zealand, you can find out about the services offered through the Access and Choice programme here.