I didn't realize that SROI was so squishy about causation! I do like the "four filters" since they are good vocabulary to help stakeholders understand why we can't just use static group comparisons.
Yes, the filters themselves aren't bad, they just don't replace causal inference. I have used similar filters in the past, too. The 4 considerations in SROI overlap with a broader set of considerations under the general heading of "additionality" - maybe a topic for one of us to write about in the future.
Great article and I agree with all the yells at the cloud. In my experience, it's the stakeholder consultation that really adds value to these judgement calls. The right kind of consultation and enough of it can add great robustness.
One time, we did extensive consultation as well as a regression analysis to estimate the causal impact of a program. One could sort of validate the other, but I suspect I have more to learn about mixed methods evaluation to understand how to truly leverage the mixing of the consultation insights with the regression results.
Causal inference requires defining the counterfactual scenario (what would have happened without the intervention) and systematically ruling out alternative explanations.
I think extreme caution is required with the options presented under:
causal relationships can be inferred to an acceptable degree of robustness through a range or combination of strategies, such as:
I think extreme caution is generally advisable. There are plenty of poorly designed and inconclusive RCTs out in the world too and even the good ones have limitations like external validity. Multiple options are necessary because contexts vary. Careful detective work and triangulation are needed no matter how we approach causality.
I didn't realize that SROI was so squishy about causation! I do like the "four filters" since they are good vocabulary to help stakeholders understand why we can't just use static group comparisons.
Yes, the filters themselves aren't bad, they just don't replace causal inference. I have used similar filters in the past, too. The 4 considerations in SROI overlap with a broader set of considerations under the general heading of "additionality" - maybe a topic for one of us to write about in the future.
But thanks for the insights.
Great article and I agree with all the yells at the cloud. In my experience, it's the stakeholder consultation that really adds value to these judgement calls. The right kind of consultation and enough of it can add great robustness.
One time, we did extensive consultation as well as a regression analysis to estimate the causal impact of a program. One could sort of validate the other, but I suspect I have more to learn about mixed methods evaluation to understand how to truly leverage the mixing of the consultation insights with the regression results.
I agree with:
Causal inference requires defining the counterfactual scenario (what would have happened without the intervention) and systematically ruling out alternative explanations.
I think extreme caution is required with the options presented under:
causal relationships can be inferred to an acceptable degree of robustness through a range or combination of strategies, such as:
I think extreme caution is generally advisable. There are plenty of poorly designed and inconclusive RCTs out in the world too and even the good ones have limitations like external validity. Multiple options are necessary because contexts vary. Careful detective work and triangulation are needed no matter how we approach causality.
Completely agree.