Scaling return on investment
The value of a pilot and the value of an intervention at scale are fundamentally different things
I often get asked to assess return on investment (ROI) from a pilot. My first response is to clarify why we’re asking and what we want to learn from the analysis, because ROI of a pilot is not a valid indicator of potential ROI at scale. For example, just because a pilot created $2 of value for each $1 invested it doesn’t mean we should expect the same result after scaling up.
ROI of a pilot is not a valid indicator of potential ROI at scale.
Confusing ROI of a pilot with ROI at scale can come at a high cost. Some pilots don’t show a positive ROI but turn out to be worthwhile investments when taken to scale. If we only looked at ROI of the pilot we might decide not to scale it and consequently we’d never realise the value.
Nonetheless it’s worth examining pilot costs and impacts. With a bit of care, we can learn important things from pilots that can help us estimate costs, benefits and ROI at scale.
What does return on investment mean?
ROI, technically speaking, is a specific indicator that comes out of a cost-benefit analysis (CBA), defined in simple terms1 as:
ROI = (benefits - costs) / costs
However, people often use the term “ROI” colloquially as a catch-all for any of the kinds of indicators we can get from a CBA, such as net present value (NPV = benefits - costs), benefit:cost ratio (BCR = benefits / costs), and internal rate of return (IRR: the discount rate that returns a NPV of zero). The following discussion applies to all of these indicators.
The unifying idea behind all of these metrics is that CBA helps us assess whether an intervention (or policy or program) creates more value than it consumes. When we do a CBA, we’re comparing the value of the intervention’s impacts (benefits) with the value of resources it uses (costs) to see if it makes society better or worse off overall.
A scenario
Imagine we’re piloting an approach to help people whose jobs have been displaced. In a changing world, some people find the skills they have used to earn a living are no longer required. Some are at risk of experiencing long-term unemployment. Our pilots aim to help them re-skill quickly and enter new job markets where there’s growing demand and a shortage of workers. We will run four pilots, each developing and testing slightly different approaches.
Our pilots involve brokering relationships with members of the displaced workforces, prospective employers, and industry training organisations, with a view to connecting the right employers with the right trainees. Costs include running the brokerage service, and a funding pool to subsidise entry to training. Benefits will (we hope) include faster return to work, and improved labour market efficiency (reduced structural unemployment).
Funding comes from a philanthropic grant, and the donor wants to know: What’s the ROI from the pilots? This seems a reasonable question to ask, and we can understand the reasons for asking.
However, there are probably better questions to ask, like:
What’s been learnt from the pilots? and
What’s the potential ROI at scale?
It’s important to evaluate pilots and scaled-up interventions in different ways, because a pilot and an intervention at scale create value in fundamentally different ways. This table summarises key differences, and the text below unpacks them.
A pilot and an intervention at scale create value in fundamentally different ways
An intervention at scale may aim to make the world a better place through impacts that create social, cultural, environmental and/or economic value. For example, the impacts of our workforce interventions include helping people retrain and re-enter employment, which will contribute to individual and family wellbeing as well as a stronger economy. Our motivation for designing and testing the pilots is ultimately to achieve these impacts. But the value that the pilots contribute to this ultimate aim comes from a different set of activities and impacts.
The value of a pilot comes from being a (relatively) modest investment in a testing ground to innovate, adapt, learn, and provide proof of concept. Sure, our pilots may improve outcomes for a small, one-time cohort of participants. However, the principal value of the pilots is the learning that might contribute to a sustainable, scalable approach. The key focus of the pilot should be “what did we learn?” and ensuring we get maximum value from this learning. This value can be enhanced through the use of other evaluation approaches, such as Developmental Evaluation. The value of the pilot will ultimately depend on how the learning is shared and used.
The value of learning doesn’t translate easily to ROI. For example:
Pilot A might be highly effective and represent a viable model for scale-up.
Pilots B and C might show mediocre performance overall, but pioneer distinct elements of good practice that can be combined at scale to make pilot A even more effective.
Meanwhile, pilot D might turn out to be ineffective (“a failure”) - and this, too, is valuable learning because we gain insights into why it didn’t have impact and how not to invest resources in scale-up.
Pilots B, C and D might all fail the ROI test. Even Pilot A might not have a positive ROI for reasons I’ll explain in a minute – yet all four pilots have been valuable in different ways.
ROI of a pilot does not predict ROI of an intervention at scale because their circumstances, objectives, benefits and costs are not the same. Let’s unpack that…
Pilots differ from an intervention at scale in important ways that can affect their ROI
I have seen many pilots that didn’t produce enough benefits to justify their costs (so could have been written off as ‘poor investments’). But when we carefully examined the results of the pilot we found benefits were likely to exceed costs at scale. This might seem paradoxical, but it makes perfect sense when we consider some of the differences between a pilot and a full-scale approach…
The pilot and the program are doing different things: During a pilot, we’re often ‘flying the plane while we’re building it’, learning by doing, developing new practices and clarifying the design. In contrast, if and when we scale up, our focus will shift to embedding successful ways of working by translating what was learned in the pilots. The pilots don’t reflect a single way of doing things that remained constant for a period of time, but rather a situation that was constantly in flux. The outcomes of the pilot might not accurately represent potential outcomes at scale.
Different implementers: The innovators behind the pilot are not necessarily the final deliverers - often they are different people and organisations. In the transition from pilot to scale-up, ownership of the model may be transferred to people who were not directly involved in the pilot, and who may work in a different organisational context and culture. Even with a good handover process, this shift in context can affect delivery and impact in unanticipated ways.
Different operating structures: For example, suppose our pilots are set up as short-term projects, with staff and office space hired on a temporary basis. In contrast, when we scale up we will be looking to partner with large-scale organisations with existing offices and staff nationwide. These organisations may be able to adopt the new practices at lower cost, using their existing resources to do things differently. The costs of the pilot might not represent potential costs at scale.
Economies of scale: Even if the operating environments of the pilot and the full-scale approach were similar, the costs of delivery often reduce with scale. The pilots are small – working with (say) 25-30 participants each – so that we can move quickly, try things, learn, adapt, and try again, all at relatively low risk and cost. However, this means that our fixed costs, i.e. costs that are scale-independent such as some overheads, are spread across a small number of participants – so the average overhead per participant is relatively high. In contrast, when we scale up we might work with thousands of people each year – so the average fixed cost per participant could be much lower.
Ramping up: Compounding the effect on average cost per participant, our pilots won’t be working at full capacity from the start. Initially they may start recruiting participants from a zero base, and could take several months to ramp up to their full capacity. Therefore participant volumes will be lower, and cost per participant will be higher than they would be in a steady-state system.
Characteristics of participants: Circumstances and needs of the employers and trainees who opt in to the pilots may differ from the larger pool of people and organisations that we hope will participate under a full-scale program. For example, early adopters may represent ‘low-hanging fruit’ who are more open to change and more motivated than the wider target group. If we’re only working with one industry group in the pilot, characteristics of participants may differ when we expand into other industries.
Hawthorne effect: Pilots are famously susceptible to the Hawthorne (observer) effect; project staff and participants know they are being watched, so they try extra-hard and get amazing results that prove more difficult to achieve in an ongoing, full-scale program.
Length of time to outcomes: We only have pilot funding for (say) 12 months. We think this will be long enough to see some positive effects for some people. However, it will probably take longer before we can observe the full impact of the intervention and see whether effects are sustained longer-term.
For all of these reasons, ROI of the pilots is meaningless as a predictor of ROI at scale. Ultimately, pilots will meet their value proposition if they provide learning that informs future endeavours, and the ROI that matters is the ROI of successful approaches at scale. And, when the time comes to evaluate ROI at scale, the pilots are a sunk cost, having already discharged their function, so are excluded from the analysis.
Pilot costs and benefits hold important clues about potential costs and benefits at scale
Instead of focusing on pilot ROI (its costs and benefits in the aggregate), we should instead analyse pilot costs and benefits separately, consider how each might change from pilot to scale-up, and then bring them back together to assess potential ROI at scale.
For example, pilot costs can be disaggregated into components such as staff, offices, equipment, and operating costs. We should also distinguish the pilot’s establishment (one-off) costs from its recurrent (ongoing) costs, and distinguish fixed (scale-independent) from variable (scale-dependent) costs. We should unpack the cost drivers (units of input or activity that cause costs to vary – e.g. number of staff, number of workshops, etc) and unit costs (e.g. cost per staff member, cost per workshop, etc). Then, we can define what the service could look like at scale, and reassemble the cost data to estimate the costs at scale.
Similarly for benefits, we need to examine the results of the pilots and consider whether we can expect the same sort of effect at scale, or whether some adjustments are needed (e.g. for demographics, timing, Hawthorne effect, etc).
We won’t know the precise values of some variables. But the pilots may help us estimate a plausible range of possibilities. One of the strengths of CBA is that we can model different scenarios and play with a range of values for input variables to see how they affect the results. This can help us think about different scaling options (for example: what models should we scale; in what configuration; where should we scale; when should we scale; how big should we go?) You can read more about sensitivity analysis, scenario analysis and breakeven analysis here.
Finally, in between piloting and operating at scale is the transitional process of taking things to scale, and this process could have its own ROI based on how economically, efficiently and effectively it is managed. Monitoring and evaluating the process of scaling could help to maximise the ROI of the scaling process.
Bottom line
Pilot ROI doesn’t represent ROI at scale.
Pilots and scaled-up interventions create different kinds of value in different ways. The value of a pilot comes from learning. The value of scaled-up interventions comes from impacts.
Pilots can provide important clues about the nature and structure of potential costs and benefits at scale, and can inform careful and thoughtful modelling of potential ROI at scale.
Instead of asking “what’s the ROI of our pilot?”, alternative questions that can get more useful answers are:
What’s been learnt from the pilots?
Based on this learning, what approaches may have a positive ROI at scale?
ROI is just a number. In order to think in terms of ROI, we’re reducing value to two aspects of performance: resources consumed and value created. This is a valid perspective to take and cuts through a lot of complexity to help answer an important question. But the complexity doesn’t just go away. A social intervention is a complex system introduced into a complex system. Pilots don’t scale like machines; the process of scaling isn’t a copy-and-paste exercise and may have unexpected consequences. Non-linearity, chaos, self-organisation and emergence eat ROI for breakfast.
ROI can tell us something important, but the number isn’t an evaluative conclusion. We still have to make a judgement. That judgement should be based on a wider range of considerations (e.g., viability, efficiency, effectiveness, equity and sustainability of scaling), not ROI alone. There’s a solution to that.
Many thanks to Matt Healey for peer review of this article. Errors and omissions are mine all mine. This article updates a post I first shared on my website in December 2020.
Also see previous posts on cost-benefit analysis
Cost-benefit analysis and the logic of evaluation: CBA can be understood as a form of evaluative reasoning, demonstrating it belongs to the field of evaluation as much as it does to economics.
Social Return on Investment vs cost-benefit analysis: Similarities, differences, and opportunities to improve practice of both.
Modelling costs and benefits in uncertainty: Three easy uses of CBA to clarify implications of uncertainty and support better decision-making.
Cost-benefit analysis through an evaluative lens: Harness the unique strengths of this important evaluation method and mitigate its limitations.
Contextually-responsive CBA: What would it look like to conduct CBAs that are inclusive, responsive, contextually viable and meaningful for people whose lives are affected?
More accurately, it’s the present value of benefits and the present value of costs that are used. The present value is the aggregate value of a time series of benefits or costs that has been adjusted for the opportunity cost of capital or social rate of time preference using a discount rate.