Blog

Exploring the Magenta Book and the key evaluation concept of the counterfactual through Maradona’s Hand of God goal in 1986.

The Hand of God

Forty years after Diego Maradona's "Hand of God" goal helped Argentina beat England in the 1986 World Cup quarter-final, it remains one of football's most controversial moments. Indeed, the goal even has its own page on the Encyclopaedia Britannica website and recently generated a flurry of 40 year on pieces which helped to inspire this blog. While fans still argue about whether the goal was impudent brilliance or straight cheating, as evaluators we are more interested in a different set of questions: What was the impact of the goal? What would have happened if it hadn't been given?

These type of questions sit at the heart of our impact evaluation work … albeit the topic areas we consider are rather different!

For readers more interested in evaluation than football, Maradona famously described his first goal that day as being scored “a little with the head of Maradona and a little with the hand of God.” In fact, TV replays showed Maradona’s left hand punching the ball into the net. So, handball and no goal you might think, except that the referee didn’t see it. In the days before VAR, the goal stood.

A few minutes later Maradona slalomed his way through the English defence to score ‘the goal of the century’, putting Argentina on course for the semi-finals and eventually winning the World Cup itself.

But, how would things have turned out if the referee had spotted the handball? We can use this question to illustrate a key element of evaluation.

The counterfactual

For the Hand of God moment, we know what happened: the goal stood and Argentina beat England. But we don’t know what would have happened if the (divine) intervention hadn’t occurred.

Our evaluations face a similar challenge. We can usually observe the results of a policy, programme or project (‘an intervention’) after it has happened. But to properly understand its impact, we need to compare those results with what would have happened in the absence of the intervention: the counterfactual.

For example, did a business support programme lead to higher than expected business growth, a skills intervention raise skills amongst its participants to enable them to find new employment, or a regeneration scheme in a town centre enhance the attractiveness of the area to investors and local residents? The key issues for us are not only whether outcomes were observed (which can sometimes be tricky to assess), but whether the intervention caused those outcomes. For example, a business might grow following the advice provided during the support programme. But if it hadn’t received this advice, could it have pursued another route to growth, and what else might have happened that can explain this growth (e.g. changing market or economic conditions)?

The Magenta Book and approaches to the counterfactual

Establishing a credible counterfactual is a central challenge of impact evaluation because the counterfactual is, by definition, unobservable. Instead we need to develop a reasoned hypothesis about an alternate future (or futures). This is not straightforward and is where the skill of the evaluator comes in.

The recently updated Magenta Book sets out UK Government guidance on evaluation, including how to develop a counterfactual. There are experimental approaches such as randomised controlled trials and quasi-experimental approaches like difference-in-difference. SQW has considerable experience of using these techniques, including comparing outcomes for programme beneficiaries against carefully matched comparison groups, e.g. businesses participating in a support programme compared to firms that did not participate but which have similar observable characteristics of size, sector and location. If this comparison group is created carefully so that ‘beneficiaries’ and ‘comparators’ are sufficiently similar, we can reasonably conclude that any difference in outcomes between the two groups can be attributed (at least in part) to the intervention.

However, these experimental and quasi-experimental methods are not always possible. There are practical and ethical considerations regarding randomised controlled trials, and beneficiary numbers may be too small or the data unavailable to implement quasi-experimental approaches. The revised Magenta Book therefore places considerable emphasis on choosing methods that are proportionate and appropriate to the intervention, rather than relying solely on experimental/quasi-experimental methods. For many of the complex interventions that SQW evaluates, theory-based approaches such as Contribution Analysis can provide a robust and proportionate way of understanding how interventions contribute to observed outcomes. This involves developing a Theory of Change to explain the steps through which an intervention was intended to cause an outcome, gathering evidence to test whether the outcome was achieved, and assessing the relative role of the intervention compared to external factors in generating the outcome.

Why evaluation (and so the counterfactual) matters

It is essential that public money is well spent, that government intervention is well-targeted, and that any regulation is an appropriate balance between burden and protection. The government, public and all other stakeholders should be able to learn from and build on what has gone before. They should also be able to scrutinise whether the intervention was effective, the outcomes were achieved and the money was well spent. The Magenta Book

Robust evaluation is essential for learning and accountability. Without a credible assessment of impact, policymakers risk overstating success or underestimating the value of an intervention, and making future spending decisions based on weak evidence. Simply observing that intended outcomes occurred after an intervention does not prove that it was the intervention that caused the change.  A counterfactual or well-evidenced theory based approach is required.

Forty years on, the Hand of God remains one of the World Cup’s most debated moments because it invites a question without a clear answer: what would have happened otherwise? Evaluation asks the same question in a different (and arguably more important) setting. It helps policymakers move beyond whether outcomes happened, towards understanding whether a specific intervention made the difference, how and why. This evidence is essential for improving future policy and making better use of public resources.