Getting to the root

Root Capital evaluated a digital extension service for macadamia farmers in Kenya; SMS-based agronomic advice. It was a replication of a model that had showed promise with coffee farmers in Rwanda.

The evaluation concluded that whilst most farmers said they'd recommend the service to a friend, a rigorous randomised controlled trial found no evidence that they had gained knowledge.

The evaluation and an accompanying article are now doing the rounds. Both make an important point: in a field increasingly reliant on "lean data" (simple perception surveys that ask beneficiaries how they feel about a product or service) it's easy to confuse satisfaction for impact. A positive "Net Promoter Score" - a classic lean data tool - tells you something, but it doesn't tell you whether the intervention or investment actually worked.

I agree with this argument.

This argument is right, but it overlooks something important

There is another lesson to draw from this evaluation.

For conducting a rigorous assessment and publishing a failed result, Root Capital deserve tremendous credit. They've set an example for the industry.

The problem is that the evaluation confirmed the intervention failed but couldn't tell us why.

To be clear: they knew the Kenya context differed from Rwanda. They suspected the weaker relationship between processors and farmers might matter. They found that 22% of farmers had actively unsubscribed from mass SMS campaigns; a signal that something about the channel wasn't working. These insights informed the "Alternative Mechanisms" section of their report. They had thought about the why.

What was missing?

What was missing? Not the thinking about why but a structured attempt to test it.

The intended behaviour change here was straightforward: macadamia farmers adopting better agronomic practices. In designing the intervention, several assumptions were made: that adoption was genuinely in farmers' interests; that a lack of agronomic information was what was stopping them; and that SMS was a plausible way to fix that.

These assumptions needed testing, not just articulating:

• Rationale: did the macadamia farmers have a sufficient reason to adopt these practices, given the real-world costs and risks? How strong was the business case in this specific context?
• Blockers: if the rationale held, what was actually preventing farmers from acting? Was it really a lack of agronomic information, or something else: trust, liquidity, social proof? Could be more than one thing.
• Enablers: was SMS a plausible way to address those blockers, fully or in part? Or were farmers being offered a solution to a problem they didn't have? And was the channel strong enough to stand alone, without the in-person support that made it work in Rwanda?

The RCT did its job; it told us that farmers didn't gain knowledge. But it was never going to tell us which part of the approach failed.

Coupling the RCT with even modest research into these assumptions would have given Root Capital both: confirmation that the programme wasn't working, and a clearer understanding of why. Thinking through mechanisms and formally testing them are two different things.

The wider lesson

Root Capital are not an outlier. The gap between measuring outcomes and understanding mechanisms is a persistent problem in how development and social impact programmes are evaluated. I've made this mistake myself more than once. But the result is that we learn whether things worked, but not why.

We've established that simple satisfaction metrics aren't enough. But this case suggests that measuring what changed isn't enough either; even when the methods are rigorous. It needs to be coupled with research pointed at the underling assumptions/mechanism.

Root Capital came closer than most. As a sector we need to close the gap further.