Monitoring for delivery, evaluation for everything else

Overly complex monitoring is everywhere. 🫠

Part of the problem is that, when we develop our monitoring plans, we try to optimise for everything.

Common advice suggests, for example: indicators for every step in our theories of change; indicators that both prove and improve our impact; a mix of quantitative and qualitative measures and, of course, the question of ‘contribution’ is never far away. We squeeze all this into our plans, and monitoring soon feels overwhelming.

There are several remedies, but one of the most useful is to revisit the basic distinction between monitoring and evaluation. The distinction I outline below is roughly conventional. My argument is that leaning into it – really applying it – can make a big difference. It can give teams the confidence to prioritise what they monitor and avoid the overload that comes from trying to do everything at once.

Two questions, two processes

To understand impact, two questions matter most:

1. What's changed – why or why not?
2. How did our actions contribute to that change?

Answering these questions requires two distinct processes:

1. Monitoring: A first pass at "what's changed - why or why not?"
2. Evaluation: A deeper look at change and our contribution to it.

Monitoring: A first pass at “what‘s changed - why or why not?”

Conducted by implementing teams, monitoring typically centres on a theory of change and a set of indicators. The theory sets out changes we think will occur as a result of our actions. Changes in behaviour, performance and perceptions. The indicators help us track whether, and to what extent, these changes actually occur.

If we're doing monitoring well, we're holding our theories lightly. We expect change to stall, and to move in unexpected ways. So when data deviates from expectation, we're not surprised – rather, we're eager to leave the office to explore what's happening and why. We put our indicators down, and go have some conversations.

We don't need complex monitoring plans. We're not trying to track everything that moves. We're tracking the most important whats – a few key hypotheses - and digging into the why and nots when we need. For this, a simple ToC and a few good indicators will do. We know that an evaluation can follow, so we have the confidence to prioritise.

Evaluation: A deeper look at change and our contribution to it

Whether conducted internally or outsourced, evaluations give us a chance to return to key projects with more resource and rigour. They can go deeper into the question of “what changed - why or why not?” building on monitoring data. A good evaluation will go further to ask: what factors caused these changes? And what role, if any, did the project play?

Evaluations can, of course, do more. We can ask them to judge value or draw lessons. These are super important questions but they can’t be addressed until we first understand change and its causes – these are the core.

Why this distinction matters

Internalising this distinction can be freeing. It can give teams the confidence to be more selective, more discerning, in what they monitor. Simpler theories, fewer indicators – testing key hypotheses.

When monitoring plans contain more than, say, five or six good indicators, we should remind each other: "Monitoring is just the first pass - if this project goes anywhere, we can choose to return to it later with evaluation”.

Thirsty for an example?

A real and current example. Imagine a team working with several regional water companies. The team identified a problem: companies haemorrhaging water via leaks, undermining their commercial performance and household access. They traced this to out-dated infrastructure and a lack of know-how.

The theory was simple: If they could subsidise strategic infrastructure and close the knowledge gap, water companies would: Invest to upgrade, reducing leaks; improve commercial returns; and improve household access to water.

Monitoring: what’s changing?

The team developed a simple theory of change and a light monitoring plan. They focused on a few priority indicators – designed to confirm or challenge their hypotheses – and set up systems to check these regularly. To what extent were companies investing in upgrades? Were leaks reducing? Was revenue increasing? Was household supply improving?

This plan formed the formal backbone of their monitoring. They were in regular dialogue with water companies but when data didn’t align with expectations (as was the case for a few companies) this focussed that dialogue: they spoke with staff and customers to understand what was happening and why – to adjust their approach accordingly.

Evaluation: did it really change and what caused this?

Jump forward a few years. The team felt their theory had broadly held. Water companies had reducing leaks and improved commercials and access. But before talking too extensively about 'their impact', an evaluation was commissioned.

The evaluation is currently underway. It has clarified the team’s impact claims and is assembling evidence around change and causality. It's looking at the project's role and weighing other explanations – regulatory incentives had shifted, and other donors were active. Perhaps these factors explain more than the team's efforts, or perhaps they complement them – I'm hopeful that the evaluation will show the way.

As my boy goes off to football camp, an analogy comes to mind. Monitoring is like watching the game as it happens – adjusting tactics. Evaluation is the mid- or post-match analysis, when you can take the time to go deeper and understand causes. Don’t try to do the match analysis while the game's being played – you’ll only overwhelm the players.