Not all Social Return on Investment (SROI) studies are created equal. Some offer a useful, defensible picture of value; others give you a shiny ratio on top of a black box, with little transparency about who was involved, how impacts were estimated, where the monetary values came from, or what was left out. One kind is valuation, the other is marketing.1 This post is about telling the difference, using a simple quality checklist you can apply to your own or others’ SROIs.
I have previously written about the importance of transparency in SROI and social cost-benefit analysis (CBA). A good SROI or CBA is more than a benefit-cost ratio: the study ought to be transparent enough about its design choices, assumptions and uncertainties for readers and stakeholders to judge its quality and fitness for their decisions. It also needs to attend to the matters that any good evaluation takes care of, like ethics, responsiveness to context and stakeholders, etc.
I recently read an SROI report that fell short. I don’t think it would be helpful to name and shame, but I will describe in anonymised terms what I observed, and more importantly what I didn’t see. Even without sharing the specific report, it has value as an illustrative case.
Please note: variability in quality is about how SROI is practised, not an inherent flaw in the method. I’m not claiming that all SROIs are like this one (some are excellent). What I am claiming is that this single case illustrates important issues we can all avoid in our next SROI.
Quick primer
Social Return on Investment (SROI) is a method for describing and quantifying the social, environmental and economic value an organisation or initiative creates compared to the value it consumes. SROI aims to equip decision-makers to make more informed choices about resource allocation and helps organisations demonstrate their impact.
SROI analysis involves working with key stakeholders to map outcomes, assigning financial proxies to those outcomes and calculating a ratio of benefits to costs. The approach is underpinned by eight Principles of Social Value that guide its application: 1) Involve stakeholders; 2) Understand what changes; 3) Value the things that matter; 4) Only include what is material; 5) Do not overclaim; 6) Be transparent; 7) Verify the result; 8) Be responsive. These principles are aimed at ensuring SROI analyses are robust, credible and accurately reflect the value of the organisation or initiative.
For deep dives into SROI from my perspective as an evaluator, you can check out my previous articles here, here and here.
SROI is capable of contributing important insights
Systematic analysis of costs and benefits in commensurable units can be very useful. I argue that evaluations addressing value-for-money (VfM) often should include cost-benefit analysis (CBA) or SROI (along with other methods).
In my opinion, SROI is strengthened when analysts follow accepted good practices developed over decades by theorists and practitioners of CBA, e.g., for causal inference, discounting, monetisation of benefits, and dealing with uncertainty. The SROI guidance mentions this too. It’s just not always followed.2 Maybe it’s no coincidence that many of the best SROIs I’ve seen were done by teams with strong economic expertise.
I also think that CBA, in turn, could pay greater attention to engaging directly with stakeholders, in keeping with SROI principles - and (as I argued in a recent paper) with program evaluation standards.
Ultimately, whether you call it SROI or CBA, what matters is how well you do it. To that end, I developed a 2-page PDF checklist for assessing the transparency and quality of a social CBA or SROI study. (The downloadable version is for paid subscribers, but I’ll summarise it below). I used this checklist to provide a rapid assessment of the following SROI report.
A black-box SROI case
This SROI report on a social service was presented as a professionally designed glossy PDF report of 20 pages. I do like reports that are visually appealing, brief and to the point. However, there are other aspects that I value just as much, and this report failed on most of them. Here’s an overview of what the report contained.
In this 20‑page glossy PDF, only six pages are actually about the social service; the rest is generic text about the company that provided the analysis and a high‑level boilerplate description of its in‑house method that reads more like promotional copy than methodology, plus front matter, references and artwork. The six substantive pages offer a quick fly‑past of participant demographics, the service team, a bullet‑point description of the service, an outcomes map, a summary of findings with no visible methods, and a list of good practice features this service shares with similar interventions studied elsewhere (with vague hand‑waving about “international literature” but no citations linking the bullet points to actual research). There is a reference list that includes relevant external evaluations and previous CBA work but does not show how, or even whether, those studies informed the current analysis.
What was missing?
Meaningful detail on the service and its context
Study design and methods (e.g., scope of costs and benefits, data sources, data collection and analysis methods, approach to determining causality or contribution, approaches used to value impacts in monetary terms, time horizon covered by the study, how costs and benefits were adjusted for timing, assumptions and their rationale, limitations)
Transparency on calculations (they might be spot on, but we have to take their word for it)
Range estimates, sensitivity analysis, scenario analysis, or break-even analysis - ways of exploring how much the ratio depends on key assumptions
Stakeholder voices
A coherent, evidence-based performance story
Evaluative conclusions.
One issue that I find particularly concerning is the use of “proprietary methods” as a reason not to show how impacts and values were derived. Commercial intellectual property is a legitimate concern in some fields, but in SROI and social CBA you are asking people to trust a headline ratio that may influence public decisions. If the core logic of the analysis can’t be scrutinised because it’s a trade secret, it does not meet basic standards of transparency or evaluability, no matter how glossy the report looks. Put bluntly, if your causal logic and valuation methods are too commercially sensitive to explain, then in my view they are too opaque to be used for real-world value-for-money decisions.
Taken together, you can see how this falls short on core Social Value principles like “Involve stakeholders”, “Do not over-claim”, and “Be transparent”.
This study scored zero (missing/not stated) on 13 of my 16 checklist items, and only reached a total score of 2 out of a possible 48 (or 4%). It scored similarly poorly on the eight Social Value Principles.
This criticism isn’t driven by perfectionism. It’s about the minimum information needed to judge whether the results are good enough to trust and use. Expectations may vary with decision stakes, but I don’t think this report is good enough under any circumstances.
I would grade this F for Fail. To show my working, I’ll detail the criteria and standards, and how the SROI study performed, in the next section below.
How I assessed the study
In keeping with sound evaluation practice, I’m aiming for a reasonably transparent assessment using explicit criteria (aspects of quality) and standards (levels of quality). In principle, that means you don’t have to agree with my criteria, my standards, my interpretation of the evidence, or the conclusions I reached. One advantage of this approach is that if we do disagree, we can pinpoint which part of the assessment is in contention.
In this instance, though, I’m intentionally falling short on the ‘transparent evidence’ side by electing not to share the specific report. It’s publicly available and I thought hard about linking to it, but ultimately decided to anonymise the case so as not to throw anybody under a bus. The irony of preaching transparency while withholding the report is not lost on me. I hope you’ll accept the trade-off and the illustrative value of the case description for what it is.
Levels of SROI quality
For the purposes of this assessment, I used a four-point rating scale. In practice, I would want to see a well-documented SROI score highly (at least 2 and preferably 3) on most criteria before using it to inform high-stakes decisions:
0 - Missing / not stated (i.e., the report doesn’t say)
1 - Stated but not justified
2 - Stated with minimal justification and transparency
3 - Stated with sufficient justification and transparency to make the study replicable.
Aspects of SROI quality
I could have assessed this study against the SROI Guide, CBA guidance such as the Green Book, or Program Evaluation Standards like these. They each contribute useful and complementary pointers. I used a checklist that I developed which spans prioritised aspects of all three, drawing on my PhD and a subsequent peer reviewed journal article. You can use this checklist to self-check your own or others’ SROIs.
Here’s how the report performed on each of the 16 items in my checklist:
Identifying and engaging stakeholders to understand their context and what they value: 0 (The report included a one-page summary of participant demographics and a description of the service delivery team, but no details on whether or how their input was sought).
Not mandating or imposing any method of evaluation as a gold standard: 0 (The approach and methods were preordained, not negotiated).
Defining the perspective of the study (value to whom) and possibly carrying out the analysis from more than one perspective, to understand how costs and benefits affect different groups: 1 (The perspective was “social value”, defined as total benefits to individuals and government. This was stated without explicit rationale, though it is a commonly used perspective in SROI studies. The study explored different levels of social value based on participant engagement, showing a positive dose-response gradient in which stronger engagement leads to better outcomes and more social value. However, there was no distributional analysis).
Defining the scope of the study (which costs or benefits are included or excluded): 0.5 (The outcomes map identified which outcomes were in and out of scope. No information was given on the scope of costs included).
Selecting an appropriate time horizon (how long costs and benefits are expected to last or matter) or a range of time horizons for scenario analysis: 0 (The summary of participant demographics stated that it was based on one year’s data. Implicitly, this might also be the time period covered by the analysis of costs and benefits - but not necessarily. For example, it would be possible to model multi-year benefits on the basis of one year’s data plus transparent assumptions. As the benefits (and possibly some costs) of this intervention should last longer than one year there are sound arguments for a multi-year analysis, which could involve ex-post measurement and/or ex-ante modelling. A time horizon for the analysis should be stated and justified).
Selecting an appropriate discount rate (to adjust the value of costs and benefits according to their timing) or a range of discount rates: 0 (Not stated. Unclear whether discounting was applied. If the study only used one year’s data, discounting could be deemed unnecessary - though it would also underestimate benefits from this intervention. If the study was purely ex-post, an argument could have been made for using undiscounted figures. However, the decision should be stated and justified).
Selecting defensible methods for estimating monetary values for benefits and costs: 0 (Not stated; no details provided; some reference to proprietary methods with no transparency about how they work).
Taking a robust approach to causal inference, analysing impacts as the difference between outcomes with and without the intervention, and not simply assuming that the intervention caused or contributed to a set of outcomes: 0 (Not stated).
Avoiding false precision (where a precise-looking return-on-investment figure obscures inherently imprecise estimates) - for example, by including sensitivity analysis (to understand the implications of key assumptions and how they affect the conclusions reached by the study), and scenario analysis (to understand the range of potential results under different sets of conditions (from optimistic to pessimistic combinations of assumptions): 0 (Missing).
Transparently reporting methods, data sources (including sources of costs, impacts and their monetary valuations), assumptions and procedures: 0 (Missing).
Transparently reporting limitations of the study: 0 (Missing).
Conducting the study with integrity, in line with Program Evaluation Standards and professional evaluation codes of ethics: 0 (Not stated).
Specifying ethical considerations, such as informed consent, confidentiality, and potential impacts on participants and stakeholders: 0 (Not stated).
Taking a mixed methods approach where necessary to address social, cultural, environmental or ethical considerations that aren’t fully reflected in the monetary analysis of benefits and costs: 0.5 (Three methods were mentioned: outcome mapping, literature review and quantitative analysis. No attempt was evident to evaluate outcomes that were excluded from the monetary analysis).
Providing judgements that go beyond the benefit-cost ratio to answer any evaluative questions, supported by evidence and logical argument (i.e., explicit evaluative reasoning): 0 (Missing. The benefit-cost ratio isn’t the conclusion - it’s a key piece of evidence. In order to reach an evaluative conclusion, a judgement has to be made taking into account all of the relevant evidence together with an explicit performance framework. Some examples of additional factors that may be taken into account alongside the number include the degree of uncertainty and risk involved, distributional implications, and any significant intangible benefits or costs that weren’t included in the analysis. This is not just my opinion: the Green Book concurs).
Outlining how findings will be used and how stakeholders can engage with the results to ensure that evaluations provide actionable insights and contribute to positive impacts: 0 (No discussion of this).
Overall, this study scored 0 (missing/not stated) on 13 of my 16 checklist items, and a total of just 2 points out of a possible 48 (4%).
For thoroughness, I also assessed the study against the eight Principles of Social Value, using the same four-point rating scale:
Involve stakeholders: 0 (Unclear whether they did or not).
Understand what changes: 1 (The report explicitly identified changes that it labelled as outcomes but didn’t explain if or how the analysis connected them causally to the service).
Value the things that matter: 1 (The outcomes map was explicit about what’s in and out of scope but didn’t explain why; based on the nature of the outcomes listed, it appears to me that the scope was primarily driven by what could readily be converted into monetary values rather than by what matters).
Only include what is material: 0 (The report wasn’t explicit about this and didn’t show its working so I can’t say for sure).
Do not overclaim: 0 (The report wasn’t explicit about this and didn’t show its working so I can’t say for sure).
Be transparent: 0 (See all comments above).
Verify the result: 0 (The report wasn’t explicit about this so I’m unable to say).
Be responsive: 0 (The report wasn’t explicit about this so I’m unable to say).
So it’s an F on the SROI principles too (scoring 2 out of a possible 24).
When a report looks like this, the problem isn’t just one consultant or one client. It’s a pattern, and the more we tolerate it, the more we turn SROI from an evaluation tool into a branding exercise. The SROI gets attached to a mysterious process with a shiny number at the end. The number may look impressive, but without clear methods, causal inference, uncertainty analysis, and stakeholder voice, it is not safe to use for real-world decisions.
I know from conversations with SROI practitioners I respect that tight budgets, short timelines, and pre-set expectations can make it hard to do this work as rigorously as we’d like. Those constraints are real - but they don’t make black-box ratios OK. If a study isn’t good enough to support decisions, we should be honest about that.
Let’s do better SROIs
There are compelling reasons to accept SROI into the field of program evaluation. In order to do that, SROIs should be conducted in accordance with program evaluation standards (as well as their own guidelines) and be subjected to meta-evaluation.
Studies like this one do nothing to support the credibility and potential of SROI to provide important information for sound resource allocation decisions.
If you’re commissioning SROI:
Ask for a clear study perspective, time horizon, discount rate, and approach to causality and uncertainty in the terms of reference.
Ask for an example of a previous SROI (or social CBA) where methods, assumptions, and limitations were reported transparently, and check that it aligns with your expectations.
Require transparency on data sources, assumptions, and limitations, and insist that stakeholder involvement is described, not just asserted.
Ask how the analyst will report uncertainty (for example, range estimates or sensitivity analysis).
If you’re conducting SROI:
Treat the benefit-cost ratio as one piece of evidence, not the conclusion, and make your evaluative reasoning explicit.
Use the 8 principles and a quality checklist as a design tool up front, not just a compliance check at the end.
Report transparently - strive for full marks on the quality checklist!
This is what a good SROI looks like
Here’s an example of what we should be aiming for. It tells a credible story about how change happens, for whom, and with what degree of confidence, instead of hiding behind a single impressive ratio. In this study, the headline figure sits on top of an explicit framework of outcomes, a documented approach to establishing impact, and sensitivity analysis that shows how results shift under different assumptions. You can see what is in and out of scope, which evidence sources were used, how benefits were valued, and what the limitations are, so a critical reader can follow the chain of reasoning rather than taking it on faith. The report explicitly uses and documents the 8 Principles of Social Value.
The analysis is clear about what actually matters, not just what can be measured and monetised. It clearly connects recreational physical activity to multiple wellbeing outcomes, drawing transparently on evidence. Equally important, it does justice to value that can’t be translated into dollars: the report includes a dedicated chapter that explores outcomes for Māori through a Te Ao Māori lens, and makes a principled decision not to monetise these while still treating them as central to the value story. Stakeholders are involved in identifying what changes and what matters, materiality and overclaiming are discussed openly, and the result is framed as a conservative, revisable estimate to inform decisions, not a magic number that ends the conversation.
Hats off to Sport NZ and the team who conducted this study.
How I use SROI and CBA
For context, I argue that even at their best, SROI and social CBA give primacy to outcome efficiency and less explicit weight to other important criteria such as equity, cultural sustainability, and procedural justice. Some outcomes are easier to value in monetary terms than others, and this can skew perceptions about what is important or material. In public policy, I argue SROI and CBA should usually be treated as contributors to mixed‑methods evaluations that bring multiple criteria and strands of evidence together in a wider evaluative frame, rather than as stand-alone value for money assessments. See here for more.
Thanks for reading!
If this was useful, a quick tap on the ❤️ helps me know it landed - and it also nudges Substack to show the piece to more people who might benefit from it. Thank you.
Comments are welcome, but please keep them civil. If you think you recognise the black-box report, keep it to yourself. There are plenty of SROI reports at both ends of the spectrum; the point here is introspection and learning across the field, not naming and shaming.
Also see…
If that sounds blunt, it’s intentional. When SROI is used to justify decisions with weak methods and opaque assumptions, we should feel uncomfortable.
Reviews of SROI practice have highlighted variable reliability in causal inference, considerable latitude in how impacts were valued, and weaknesses in the handling of uncertainty - such as the use of questionable monetary proxies, risks of overstated and unstable SROI ratios, double counting of overlapping benefits, and a tendency for precise-looking SROI ratios to obscure wide uncertainty around underlying assumptions (Siegal, 2022; Fujiwara, 2015; Maier et al., 2014; NSW DCJ, 2020).
Great analysis, Julian. I have probably been guilty of being a little dismissive of SROI more generally because of the problematic approach you've so eloquently skewered. Clearly there are good ways to apply this methodology. I haven't seen the 'black box' SROI approach be particularly effective with decision makers (despite the myriad reports that seem to be commissioned); the lack of transparency just isn't compelling.