Value for money criteria
The 5Es aren’t the last word, but they’re a pretty good place to start
Quick recap of things I’ve said before
Value for money (VfM) assessments are often distracted by the money and unclear about the value. We can address this by reframing policies and programs as investments in value propositions. That’s why I prefer the term Value for Investment (VfI). Value propositions are useful constructs because we can define them and evaluate how well they’re met. Evaluating involves making value judgements. Those judgements must flow logically from facts (independent observations of performance) and explicit values (what matters to people). Those values can be expressed in rubrics, comprising criteria (aspects of value) and standards (levels of value). Interpreting evidence through the prism of a rubric is called evaluative reasoning.
OK, we’re up to date. This post dives into the criteria part. Let’s go.
Value for Money criteria
VfM criteria, one component of the evaluative reasoning process, define aspects of VfM to focus on. These aspects are often decided by organisations that fund and review policies, programs and services. For example, there’s a cluster of five criteria that are commonly used to define VfM: Economy, efficiency, effectiveness, cost-effectiveness, and equity – sometimes called the “5Es”. These criteria are used by the UK Government’s Foreign, Commonwealth & Development Office (FCDO) and National Audit Office (NAO) and they’re often used around the world.
However, there are no definitive criteria of VfM. Different organisations have different ways of defining VfM (check out some examples here). They’re all concerned with good resource use but they have different criteria reflecting their various needs and priorities.
Generic criteria won’t do, because each investment has a unique set of circumstances. I argue for a flexible, inclusive approach to developing criteria.
Value for Investment criteria
VfI criteria (i.e., VfM criteria ‘my way’) should be contextually defined, not prescribed. They represent negotiated and agreed qualities that we should look at to determine whether a specific investment is a good use of resources and whether it creates enough value. These criteria are co-defined with people affected by the program, aligned with the program’s value proposition, and described in terms that are meaningful and specific in the context of the program.
Note that criteria aren’t indicators, and they should be developed before we turn our attention to indicators, or more generally to what evidence is needed and what methods should be used to gather and analyse it. This sequence is important because criteria define ‘what matters’. If we start by defining what matters, we’re likely to design an evaluation that collects evidence on what matters. Conversely, if we were to start by defining what we can measure, we would run the risk of mistaking whatever we can measure or count with what is important.
The 5Es can be a useful starting point for considering potential criteria, but we don’t have to use all (or any) of the 5Es and there may be additional criteria to consider.
The 5Es are a good starting point…
Value is the merit, worth or significance that people and groups place on something. That ‘something’ is often an impact – real differences in people, places and things, caused by organisational actions. Those organisational actions can include policies, programs, products, services, etc, and are fuelled by resources. Those resources include money, people, relationships, know-how, and all sorts of other inputs.
Resources, organisational actions, impacts, and value can be represented in a chain, like this.
The 5Es address each of these elements in turn.1 Economy is concerned with stewardship of resources. Efficiency looks at the productivity of organisational actions. Effectiveness focuses on the impacts of those actions.2 Cost-effectiveness considers whether the organisational actions and impacts create enough value to justify the resource use. Equity is relevant at every level. The 5Es are a good starting point because they span the entire value chain and its components.
…but there’s more to VfI criteria than the 5Es
The 5Es reflect aspects of good resource use that FCDO and NAO have deemed to be relevant and important for their organisational purposes. But there’s more to good resource use than just the 5Es. For example, the OECD DAC Evaluation Criteria cover additional aspects, any of which could be applicable to a VfM assessment.
As I noted in an earlier post, the definitions of the DAC Criteria overlap conceptually with the 5Es, illustrating that VfM and monitoring & evaluation (M&E) aren’t just related – they’re entangled.
In OPM’s new VfM assessment guide (pages 35-37), we outline how the DAC criteria of Sustainability, Relevance and Coherence can potentially connect with VfM, along with two additional criteria – scalability, and acceptability.
When we look at wider criteria embedded in various organisations’ definitions of VfM, we can see that we could also consider ethics, accountability, transparency, affordability, probity, minimising wastage, proportionality, risk management, innovation, competition, adaptability, and more.
That’s OK - we can set boundaries. The point isn’t to throw the kitchen sink at every VfM assessment but rather to consider and select appropriate criteria for the circumstances.
Let’s keep it simple: 5Es +/- a few
The good news is that we can often incorporate the most pertinent of these various criteria within our program-specific definitions of the 5Es. This has the advantage of creating a bridge between the five familiar (and often expected) dimensions of VfM, and the aspects of good resource use that are most applicable to a specific program. For example, sustainability could be positioned as a sub-criterion of cost-effectiveness, recognising that long-term value creation often depends on sustained actions and impacts.
In some cases, there may be an extra consideration that warrants its own heading, giving it equal status to the 5Es rather than subsuming it within one of them. For example, the Australian Department of Foreign Affairs and Trade (DFaT) specifies a criterion of Ethics that includes accountability and transparency (and could include more). Some VfI evaluations in New Zealand have specified being a good Treaty of Waitangi partner as a distinct criterion of good resource use for government agencies.
In some cases, we might not need to use all 5Es. For example, in the early days of a program, it may be too soon to look at effectiveness and cost-effectiveness. It may be appropriate to focus only on economy, efficiency and equity.
The following summary, building on OPM’s new guide, offers examples of considerations that often arise when developing bespoke definitions of the 5Es. Not an exhaustive checklist, but a useful set of prompts.
Equity / addressing inequities
I’m being quite intentional about putting equity first because it too often appears in VfM assessments as an afterthought, yet equity is central to many of the interventions we evaluate.
Some policies and programs only exist because inequities or injustices exist. The purpose of these programs, in 5Es language, is to address these inequities or injustices as effectively, efficiently and economically as possible. Improving equity efficiently may be quite a different proposition with different resource requirements from just being efficient.
Equity considerations may be applicable at every level of a VfM framework, because there are aspects of equity that relate to stewardship of resources, ways of working, impacts, and how different people value these things. Therefore, a starting point for identifying equity criteria is to consider questions such as:
Why and for whom is the investment needed? What is an appropriate and fair distribution of resources, actions, impacts and value?
In some evaluations, we have hybridised the 5Es to reflect the centrality of equity to program objectives. For example: this evaluation and also this one, used three criteria along the lines of:
Looking after resources equitably and economically
Delivering services equitably and efficiently
Generating social value equitably and effectively.
Examples of sub-criteria of equity (aspects of equity that could be considered for inclusion in a VfM framework) include:
Equity in power dynamics – e.g., program design, delivery, monitoring, and evaluation are conducted with, by or as the communities affected by the program, e.g., those intended to benefit and those with a right to a voice (Wehipeihana, 2013; 2019).
Equity of design – e.g., program design and needs assessments explicitly identify groups intended to benefit and appropriate ways of including and working with them.
Equity of resourcing – e.g., project appraisal and investment decisions explicitly allocate resources to priority groups; resourcing is appropriate to work with the intended groups (…and perhaps our evaluations can provide insights into the costs of programs that are effective and efficient at addressing inequities?).
Equity of access – e.g., interventions are accessible and acceptable to people from key target groups (and accessed by those groups) and frictional losses (such as interventions benefiting people outside the target groups) are minimised.
Equity of delivery – e.g., implementing explicit strategies to reach target groups, and monitoring results to understand who benefits from the program, including the perspectives of recipients and their representatives.
Equity of outcomes and impacts – e.g., real improvements in the lives of people who are intended to benefit, including fairness between groups and incorporating the perspectives of recipients and their representatives.
Cost-effectiveness / value creation
The cost-effectiveness criterion focuses our attention on whether enough value is created to justify the investment of resources. An intuitive doorway into defining cost-effectiveness criteria is to ask:
To whom is the investment valuable? How is it valuable to them? What would enough value look like?
The way the value proposition is defined may affect the approaches we can use to evaluate it. Economic methods aren’t mandatory but can often contribute important insights. Choosing whether to use an economic method and which one(s), is contextual. Options include cost-effectiveness analysis (CEA), cost-utility analysis (CUA), cost-benefit analysis (CBA), break-even analysis (BEA), Social Return on Investment (SROI)… or the option of not including an economic evaluation.
The label ‘cost-effectiveness’ is potentially confusing because it is also used in health economics, where CEA is a specific method that doesn’t reflect the use of the term in the 5Es. What matters first and foremost within the context of the 5Es is not what method we apply but that we consider the relationship between value created and value consumed. Sometimes it may not be feasible or appropriate to use economic methods. Moreover, in equity-focused programming, an investment might not pass the cost-benefit test but rather might address inequities to a worthwhile extent. Accordingly, cost-effectiveness, just like every other criterion, is only sometimes applicable, requires a well-reasoned judgement, and shouldn’t be tied to any one method.
Potential ways of framing up program-specific sub-criteria, and their associated strategies for addressing cost-effectiveness, include:
Impacts and their associated costs are comparable to those seen in other relevant interventions in similar contexts (this could be informed by specific indicators such as cost per quality-adjusted life-year or a more qualitative assessment of other programs).
Impacts and their associated costs are justifiable relative to the costs of ‘doing nothing’ (e.g., the cost-effectiveness of a smoking cessation program may be judged in terms of its success or potential in reducing the burden of disease associated with tobacco smoking).
The value of impacts achieved to date exceeds the value of resources invested (it is this type of question that CBA may be able to address if benefits and costs are amenable to being valued in units of money).
The estimated value of future impacts (projected on the basis of early results and judgements of their sustainability, over a reasonable time horizon and discount rate) is likely to exceed the value of resources invested (e.g., break-even analysis).
Program expectations are met for the level of funding allocated (i.e., in a fixed-budget context, cost-effectiveness may justifiably be deemed equivalent to achieving ‘enough’ impact to meet expectations at the time of funding allocation).
The value of impacts to date, relative to the value of resources invested, is within an acceptable or anticipated range given the impacts achieved on equity.
Overall, we assess cost-effectiveness by judging whether a program meets its value proposition in a manner and to an extent that makes it worthwhile from the perspectives of relevant people and groups. Any of the strategies above can help us to make this determination. However, the foundational strategy in each case is a clear value proposition and a rubric, co-developed with stakeholders, to help us make explicit judgements from a situationally appropriate mix of evidence (qualitative, quantitative, and/or economic).
Effectiveness / making a difference
Assessing effectiveness is essentially outcome evaluation, and indeed where a VfM assessment and outcome evaluation are being conducted in parallel, the former should draw on the latter. However, within a VfM assessment, the focus of effectiveness may be subtly different from an outcome evaluation: When defining effectiveness and developing sub-criteria, it can be helpful to address the question:
What changes should we pay attention to in the short- to medium-term to understand whether the investment is on track to create longer-term value?
Outcomes (and impacts) can be understood as changes in people, places and things that are caused by the program’s actions, or to which the actions contribute. So when we’re assessing outcomes we need to investigate both what changed and what caused or contributed to the change. Causal questions and evaluative questions are separate and distinct, though sometimes conflated. Both involve making warranted judgements based on evidence and logical reasoning, though the former focuses on questions about why something happened (and how we think we know), whereas the latter focuses on how good something is (and how we think we know). Evaluators and economists have multiple options at their disposal for tackling causal questions - quantitative and qualitative, experimental and non-experimental. As far as I’m concerned all options are on the table and should be selected, mixed and used with due care and caution.
VfM assessments often focus on whether a program is achieving its intended outcomes. From this perspective, the assessment of outcomes should align with the program logic or theory of change. Often, intended outcomes are identified by program architects. However, we should also seek to understand and evaluate outcomes through the lens of recipients’ needs and expectations. Moreover, some effects may be unintended and could be positive or negative, with implications for the overall value of the investment. Different people can experience different outcomes, so it can be important to consider for whom a program is effective and in what circumstances.
Efficiency / ways of working
Within the 5Es framework, efficiency typically refers to the idea of maximising outputs for a given level of inputs. This is relevant to getting VfM from organisational actions, but it’s only one aspect. More broadly, we can ask:
What ways of working will ensure we get the most value from the resources invested?
An important aspect of maximising VfM through organisational actions is productivity. This term encapsulates multiple efficiency concepts, such as:
Allocative efficiency, or doing the right mix of things. “In economic theory, allocative efficiency occurs where the distribution of goods and services in the economy matches consumer preferences and price is equal to the marginal cost of production. In public services we can use the concept analogously to refer to allocating resources to the right mix of interventions to meet needs or objectives”. (King et al., 2023, p. 30). For example, does the program have a balanced portfolio of actions to deliver its outputs? Are there opportunities to do less of something in order to do more of something more valuable?
Dynamic efficiency, or adapting and improving. “In economics, dynamic efficiency refers to the idea of optimising resource allocation over time so that no generation can be made better off without another being made worse off. In more practical terms, we can look at how well a program adapted and improved its productivity over time, through evaluating, reflecting, learning, adapting, adopting new technologies, responding to emergent opportunities, etc.” (Ibid).
Relational efficiency, including relationships, communication and trust: “a foundation that enables programs to operate efficiently. Without them, resources may be wasted. A new program may become more efficient (and effective) over time as relationships develop and it becomes a trusted part of the landscape. Building relationships requires investment and should be made explicit in order to understand VfM” (Ibid). As a specific example of relational efficiency, consider cultural fit, in which “the contextual stance or positioning of a practitioner…as an insider, of the same culture(s) as the service user” and “having a congruency with [their] core cultural values…creates grounds for greater effectiveness” (Goodwin, Sauni & Were, 2015).
Technical efficiency, or maximising productivity by maximising delivery within the available resources – e.g., delivering the overall volume of work expected within budget, on time and to the agreed quality standard.
Ways of working that represent good use of resources go beyond productivity. An example is working in alignment with organisational strategy and values. In kaupapa Māori evaluation contexts, examples include tika and kawa (doing the right things, in the right ways, recognising that how something is done matters as much as what it achieves). Any aspect of organisational actions is up for consideration if it affects the extent to which the resource use creates value.
Economy / stewardship of resources
Typically, economy is concerned with the conversion of resources (principally money) into the inputs (e.g., staff, consultants, office space, equipment, etc.) needed to deliver the organisational actions. Usually, definitions of economy in VfM assessments focus on frugal management of funding and minimising the cost of inputs – with one organisation’s framework going as far as to define economy as “spending less”.
I am not a fan of this definition. As we well know from our own purchasing decisions, value for money does not equate with cheapness, and sometimes it is worth spending a little more to get a disproportionate gain in quality, fitness-for-purpose, and better meet the value proposition of our investments.
Nonetheless, appropriately reducing the costs of an intervention can improve VfM if the intervention can maintain its cost-effectiveness, equity, effectiveness and efficiency.
I suggest we take a broader view of economy as good stewardship of resources.
What resources are invested, and by whom? What does good stewardship of those resources look like?
When we approach economy from this perspective, it leads to the insight that money is just one part of the investment. There may be other important resources to be nurtured and marshalled to create value. Examples of wider considerations include:
Attending to intangible resources such as political capital, knowledge, data, relationships and reputations.
Understanding the footprint of the program on the extraction and use of natural resources and minimising harmful environmental side-effects.
Good human resource management practices, such as equitable pay and sourcing staff from diverse backgrounds, which make a broader contribution to program effectiveness and wider societal value.
Considering trade-offs between cost minimisation and broader dimensions of stewarding resources – e.g., balancing competition-based procurement strategies with opportunities to build the capacity of small local suppliers to the longer-term betterment of a more competitive, diverse and equitable marketplace.
Of course, good stewardship of financial resources remains important. Potential sub-criteria of economy often include considerations such as:
Paying reasonable prices for inputs (e.g., average costs of significant items such as consultant fees, airfares, etc).
Using sound procurement practices (e.g., competitive tendering where appropriate and feasible, whole-of-life costing of significant items).
Managing risks of cost increases (price and/or volume-related).
Proactively finding economies of scale and/or scope, savings and ‘best deals’.
Leveraging support from partner organisations to ‘grow the funding pie’ and/or access resources in-kind (e.g., pro bono technical assistance, low or no cost use of meeting facilities).
Sound financial management and probity arrangements.
Risk management to minimise frictional losses (e.g., potential leakage of funds due to informal or corrupt transactions after funding has been disbursed from the program to subcontractors or the community).
Overall, identifying what resources are invested and defining good stewardship of those resources is the starting point for developing an appropriate set of sub-criteria that focus on value maximisation rather than cost-cutting.
Don’t drink the ocean
It’s too big (and unpleasantly salty). The level of effort invested in a VfM assessment should be proportionate to the value we expect to get from it. In other words, we want our VfM assessments to provide good VfM.
If the value proposition is too detailed, apply the 80/20 rule. In other words, a few key considerations (say, 20% of the things that matter) may get you most of the way (say, 80% of the way) towards understanding the value of the investment. To help whittle the framework down, consider:
How does the investment create value? What critical factors affect whether the investment creates a lot of value or a little?
The question of how the investment creates value prompts consideration of the mechanisms through which resources (funding, expertise, relationships, etc) are converted into significant social value. Understanding these mechanisms helps to focus the evaluation. For example, in a school leadership program discussed here, this question led to the insight that the investment was seeking to build cultural capital in schools and their communities. A capital asset lasts across multiple accounting periods and provides an ongoing payback. In this case, the payback (the return on investment) would eventually come in the form of improved learning outcomes for students. However, before any payback should be expected, the capital asset has to be built up and nurtured. In this instance, the program was still building the cultural capital and it was therefore too soon to expect major improvements in educational outcomes. This insight helped to limit the scope of the evaluation and to focus it on the processes and progress of cultural capital building. This paper on ‘theories of value creation’ (King, 2021) canvasses value creation mechanisms.
The question of what critical factors affect potential value creation can help to prioritise the ‘big things’ to pay attention to in a VfI evaluation. For example, in this evaluation of the African Risk Capacity (ARC, a specialised agency of the African Union that provides capacity-building services and insurance to help member countries plan for and respond to extreme weather events and natural disasters), we identified ARC’s value proposition for countries and households affected by drought and then worked backwards along the value chain to identify critical, observable factors that affect the scheme’s potential VfM. These factors led to us prioritising nine sub-criteria spanning the most important aspects of economy, efficiency, effectiveness, and equity.
In summary
If we can define a program’s value proposition, we can evaluate how well it’s being met. The value proposition may be presented in a paragraph or a diagram, which could be integral to or separate from a theory of change.
The following diagram summarises the value proposition questions from above and their alignment with the 5Es framework.
Next week I have an example to share.
Feedback, please!
This one’s a book chapter for sure. All feedback is welcome. Thanks in advance!
The 5Es are often presented as bridging concepts, e.g., from resources to inputs (economy), from inputs to outputs (efficiency), and outputs to outcomes (effectiveness). This is fine, but I don’t think it’s necessary, and it risks overemphasising input/output/outcome efficiency at the expense of other aspects of good resource use.
Impact is a word that’s used in different ways. Here I’m defining it broadly to include any difference in people, places or things that occurs as a consequence of the organisational actions. It includes short/medium/long-term outcomes, intended or unintended, relative to a counterfactual and can be measured using any method, qualitative or quantitative, experimental or non-experimental (Gargani & King, 2023).