Skip to main content
Feed the Future
This project is part of the U.S. Government's global hunger and food security initiative.
  • About
subscribe

Agrilinks

Search Log In

Main Menu

  • Topics
  • Events
  • Activities
  • Tools & Training
  • Members
  • About
subscribe

Disentangling Natural Resource Management Impacts — One Evaluation at a Time

International Food Policy Research Institute

May 16, 2018
A woman works in a field on a farm in Ghana.
Alison Slack/IFPRI

Last week was a tough one for our small group of IFPRI researchers in Accra. On Tuesday, we delivered a message to one of our partner organizations there that its project to promote integrated soil fertility management (ISFM) to Ghanaian smallholders did not yield any measurable impacts during the course of our two-year evaluation. Farmers were no more likely to know about, understand or apply ISFM as a result of the project, plain and simple.

But it’s never so simple, is it? Maybe the project did have sizable impacts. Our implementing partners had to make a few course corrections during the project to reach the number of farmers they had originally targeted. Unfortunately, that correction disrupted our evaluation design. In effect, we were measuring outcomes in a sample of farmers that was no longer representative of the population served by the project. So we just couldn’t detect any impacts even if they did occur.

That’s life in the impact evaluation world. Many of our colleagues already know this, but it’s worth repeating—again and again—to be better aware of the potential for such situations and to learn how to avoid them in the future.

That is what we did after delivering our disappointing message. Then we discussed our experience further with colleagues from ISSER, IPA, UDS, CABI, IFDC, AGRA and IITA at a learning event in Accra on April 25. We used this occasion to review our approaches to project implementation and impact evaluation, focusing our attention on how agricultural extension services can be leveraged to promote natural resource management (NRM) practices and principles to Ghanaian smallholders.

NRM is complicated: It encompasses a range of knowledge-intensive and site-specific management principles combined with a broad array of modern inputs and technologies. When applied successfully, good NRM principles and practices can improve and sustain the productive use of land, soil, water and biodiversity for present and future generations.

Programming and evaluating in the NRM space can be extremely complex. Here are a few thoughts, gleaned from our discussions, on how to deal with that complexity.

  • Choose your unit of analysis carefully. Many of our colleagues working on NRM think intently about designing projects to achieve landscape-level impacts. But evaluators tend to focus on individuals and households as the unit of analysis because that allows for simplification from high, often abstract, levels of aggregation to something more tractable to infer causal relationships. But does that mean we are missing some of the big picture, landscape-level issues?
  • Be ready with flexible designs. Rigorous evaluation designs are good and fine, but we always need to prepare for the unexpected. Changes in the content and timing of project rollout, treatment groups not being treated, control groups being treated, mass attrition between baseline and endline — these are but a few of the problems evaluators faced. Maybe we need to think small: Small field experiments at the outset of the project, such as survey-based and lab-in-field experiments, to understand farmers’ behaviors and preferences, augmented by a lot of well-thought out qualitative inquiries. These small experiments can produce useful insights at the outset of a project and are particularly useful in case the evaluation itself goes awry.
  • Recognize the dictatorship of near-termism. Project implementers often want proof that their NRM project had an impact on household welfare, especially income, consumption or nutrition. But those outcomes often take time to show up in the data. And it’s surprisingly difficult to measure productivity changes in a single year or season, not only because of measurement error (e.g., this recent paper from my colleagues) but also because some technologies take a long time to generate benefits.
  • Explore choices and behaviors, not technologies, per se. Efficacy trials — research station trials and farmer-managed trials — are useful in testing whether a technology works. In these trials, we hold farmer behavior constant and vary the inputs, practices and technologies. When we move into effectiveness trials, we are ultimately more interested in whether efforts to promote technologies are successful and how the input/practice/technology affects farmers’ choices and behaviors. Do they invest more in complementary inputs, do they expand area under cultivation, do they work harder? In a sense, we want to elicit reactions to a treatment and don’t really care about disentangling those reactions from treatment itself. (see this paper and a related blog for interesting perspectives on the topic.)

A negative evaluation is often an implementer’s worst-case scenario. It may give a government or donor a reason to pull project funding. But a good evaluator can help cushion the blow by highlighting lessons learned, encouraging innovation and opening opportunities for further evaluation. The reality is that many implementers are too afraid to experiment or too afraid to subject their experiments to rigorous evaluations. But learning from failures is as important as learning from success, as the growing popularity of fail festivals suggests.

By David J. Spielman, Kwaw Andam and Simrin Makhija

David J. Spielman is a Senior Research Fellow with IFPRI's Environment and Production Technology Division (EPTD). Kwaw Andam is a Research Fellow with IFPRI's Ghana Strategy Support Program. Simrin Makhija is an EPTD Research Analyst. The post first appeared on IFPRI’s website here.

The learning event in Accra was organized by IFPRI and its partners in Ghana. The event was generously supported by the International Initiative for Impact Evaluation (3ie); the U.S. Agency for International Development (USAID) Feed the Future Developing Local Extension Capacity (DLEC) project, led by Digital Green; the CGIAR Research Program on Policies, Institutions and Markets (PIM); and the Alliance for a Green Revolution in Africa (AGRA).

Filed Under: Climate and Natural Resources Education and Extension Monitoring, Evaluation, and Learning

Comments

You May Also Like

post

Evaluating Video-Mediated Extension in Ethiopia

For many researchers at IFPRI, the ultimate aim of our work is to convey findings to an audience that has a stake in the research and is in a position...

profile image of David Spielman

International Food Policy Research Institute

David Spielman

Sep 04, 2018
post

Policy, Regulation, and Digital Agriculture in Africa

The entrepreneurial spirit in Africa’s emerging digital agriculture ecosystem is cause for excitement and optimism. But when asked what governments can do to accelerate digital agriculture, our initial excitement often...

David Spielman

Dec 04, 2019
post

The research-practitioner divide: Why we are still optimistic

The Feed the future Developing Local Extension Capacity (DLEC) project is a rare and successful example of an extension researcher and a development practitioner teaming up as co-directors.

Julie MacCartee

Apr 19, 2019
post

Designing Gender-Sensitive Agricultural Extension Information Campaigns

This study tests if household-level outcomes — awareness, understanding, experimentation or adoption of a technology or practice — can be influenced by small changes in the extension approach through several...

Bjorn Van Campenhout

Oct 05, 2017
Follow Agrilinks:
  • Youtube
  • Facebook
  • Twitter
  • RSS
Agrilinks
Achieving agriculture-led food security through knowledge sharing
MarketLinks
Market-based solutions for development
LearningLab
A collaborative learning community of development professionals
Landlinks
USAID's knowledge sharing platform focused on land tenure and property rights
Climatelinks
A global knowledge portal for climate change & development practitioners
Urbanlinks
USAID’s sharing platform for resources on sustainable urban development
Resiliencelinks
A global knowledge platform for resilience practitioners
Globalwaters
Sharing knowledge and ideas to solve global water and sanitation challenges
U.S. Agency for International Development
  • Accessibility
  • Contact
  • Help
  • Privacy Policy
  • Terms of Use
The information provided on this website is not official U.S. government information and does not represent the views or positions of the U.S. Agency for International Development or the U.S. Government.

Search Agrilinks

My Account

  • My drafts and published content
  • My profile
Sign out

Contribute To Agrilinks

  • Add a post
  • Add an event
Need help?

Subscribe

Subscribe to receive our weekly newsletter featuring the latest and greatest Agrilinks content.

You'll also receive invitations to upcoming Agrilinks webinars and other special announcements. We won't share your email address and you can unsubscribe at any time.