Draft Study on Evaluation of NRM Interventions Linked to Climate Change - Now Available for Comments

Natural Resource Management (NRM) is at the center of the Climate-Eval Community of Practice’s expanded focus for the next phase of engagement. As such the community is looking to explore the issues relevant to evaluation practitioners when climate change aspects are incorporated into NRM interventions. Although the NRM sector is not new, looking at it from the environmental evaluation perspective is somewhat new and unchartered territory. That is why Climate-Eval commissioned a ‘scoping study’ of evaluations of NRM interventions linked to climate change a few months ago (see approach paper). The study was conducted by Frederick Swartzendruber – Senior Consultant at the GEF IEO, with myself as a Research Assistant. We are proud to say that the draft study is now ready and available for comments and feedback here: Scoping Study.  

The scoping study had some interesting findings which give insight into the gaps in research and practice, and ways forward for this nascent field. Below is a brief summary of the key findings, presented here with the intention of proviking comments and reactions from our audience who may be able to relate to some of it with their own experiences. We would love to hear your feedback and any comments received before January 15th, 2015 will be taken into account in the final iteration of the study. 

Key Findings:

NRM Terminology: There is a lack of clarity on the definition of NRM as a term, and this creates confusion. Donor-financed NRM interventions often exclude extractive usage from the definition. For this study, NRM was limited to non-extractive, renewable usages. 

Accountability over Learning: At the project level, most evaluations are terminal studies emphasizing accountability for use of donor resources rather than learning from the lessons of previous evaluations on aspects such as design and implementation.

Mismatch in Time-Horizons and Scale in Project Evaluations: Time horizon and spatial scale are often the most common and difficult challenges in NRM evaluations, and there is little sign of this scenario changing. The project modality may be largely to blame for this, as NRM projects are usually not longer than 4-6 years, a time too short to realize outcomes such as carbon sequestration or restoration of ecosystem functions. Furthermore, scale-related mismatches can be seen in new approaches such as payments for ecosystem services or landscape approaches to NRM.

Logframe approach won’t work: The linear cause-and-effect logic of a typical project logframe is ill-suited to NRM interventions. Longer term models incorporating a Theory of Change, experimental and quasi-experimental methods, and adaptive management hold more promise, as the complex interactions of biophysical and human systems need to be identified and new responses put into effect on a rolling basis. The trajectory of outcomes or responses to interventions may also vary, and some processes may also be reversed after a project has ended.

Some learning-focused program and portfolio evaluations exist, but not enough: For example the GEF IEO’s South China Sea Impact Evaluation; NORAD’s Real-time evaluation of the NICFI program; World Bank/ IFPRI’s impact evaluation in Niger; and USAID’s retrospective stock-taking evaluation of NRM technologies in agriculture and livelihoods in Senegal, Malawi and Namibia. The study did not find many other examples.

Poor Data Quality: Data quality and credibility remains a challenge, with regards to whether the data being collected are appropriate for supporting conclusions about attribution of project outcomes and impacts. Moreover, adequate baseline data is often not available or unreliable, indicators are poorly selected, there is incomplete monitoring of site-level data capturing biophysical conditions over time, and there are no standards in measurement (“core” indicators).

Valuation of natural resources is a grey area: Economic valuation of natural resources as well as the ecosystem services that they provide is not only a challenge for measurement but also for policy with respect to global regulatory frameworks (e.g. for REDD+ and PES).

Ways forward:

There is already a significant body of literature discussing the complex ways in which natural resource systems interact with human systems; examples include the two-system evaluand approach; sustainability science; land change or land system architecture; ecohealth; panarchy; and others.  Complexity science can be useful in framing the multiple interactions between systems which are unavoidable aspects of NRM. 

NRM interventions often take place in nested or interacting systems which function at different scales of time and space, and consideration of these factors in future evaluations is something that needs to be tried and tested over longer time periods.  There is some support for theory of change models in accounting for these longer-term processes. Mixed methods as well as carefully selected case studies can also help solve some of the methodological issues involved in evaluating NRM interventions.

Evaluations in this analysis failed to underline the lack of scientific understanding as a weakness in project or program design. Long-term monitoring of closed projects is possible with the in-country monitoring systems such as remote sensing that are now available in many developing countries. One way forward would be to fund such initiatives to better inform the design of future evaluations through improved baseline data and monitoring.

Finally, the lessons learned from evaluations in NRM need to be incorporated into more robust project and program designs in future. USAID, ADB, AfDB, GEF and the World Bank are some organizations that have expressed commitments to strengthen the knowledge management functions, and this bodes well for the future of evaluation overall. The increased interest recently in the paradigms of resilience and adaptation provide a clear entry point for integrating climate change into NRM interventions and evaluations.

Thanks!

On behalf of the author and myself, we would like to thank those of you who supported the study by providing various documents and information, comments and feedback during its formulation. We hope you enjoy reading the study and it provokes some thought on this increasingly pertinent topic in climate change and development evaluation.  Have a great holiday season, and happy new year!

Comments

Thanks, that was so thought

Thanks, that was so thought-provoking, especially both the short- timeframe expectations and accountability. Let me add to your great "Accountability over Learning: ... : At the project level, most evaluations are terminal studies emphasizing accountability for use of donor resources rather than learning from the lessons of previous evaluations on aspects such as design and implementation." A huge issue is that we limit ourselves to only what we accomplish during projects, rather than look at the abilities of communities to self- sustain activities after project close. Rarely if ever do we design for that, evaluate and learn post-project nor learn from (potentially positive) unexpected outcomes. Imagine how much better our planet's NRM could be if we supported communities own abilities to do so... More on ValuingVoices.com. Cheers!

Dear Jindra,

Dear Jindra,

Thank you for reading. Yes absolutely, evaluation in the NRM field will have to look at the learning aspect retrospectively, as there may be many reasons for unexpected outcomes and uncovering them can help inform future interventions. Cheers!

Dear Najeeb,

Dear Najeeb,

Thank you very much for sharing this interesting report.

I have read it carefully and enthusiastically as evaluations of NRM projects and programmes present several challenges to be addressed. Let me share some ideas.

When complexity and uncertainty of the socio-ecological systems and NRM processes are understood as inherent factors of reality and management processes, a wider conceptualization of evaluation is required. I mean that there are methodological choices to be done. Selected approaches should fit into the nature of environmental challenges: high and multiple values at stake in the decision making processes, resources property & tenure, short-term projects outcomes and multiple time and spatial scales (natural resources renewal rates depending if they are renewal/ non-renewal resources), wide socio-ecological processes time scales), and uncertainty of phenomena among others.
It becomes necessary to move from scientific research and consultancy advice to an integrated assessment field, enhancing the quality of the evaluation process & results reliability.
In the evaluation object side, despite general formulas to undertake evaluations rarely fit into the specific characteristics of projects and processes, adopting a “weak comparability” of values approach may be useful to design the evaluation process and choose an appropriate combination of non- compensatory methodologies and tools. The “Weak comparability of values approach” emphasizes a “strong sustainability” scenario of NRM through the analysis of impacts on a multiple set of relevant dimensions of sustainable development (monetary, bio-physical, human wellbeing (health, incomes, empowerment, happiness, equality), cultural (symbolic, aesthetic, spiritual), others) and the inclusion of externalities and distributional aspects of NRM.
In the process side, it is required to manage explicitly analytical constraints and discus how they can be affecting the evaluation (process & results reliability, highlighting and prioritizing evaluation goals, recognizing systems complexity: from the attribution to the contribution approach and clear identification of the evaluation subject and object between others) and take caution when comparing results between evaluations and projects.
So, when evaluating the impacts of NRM projects into sustainable development, the nature of environmental challenges requires enhancing the evaluation process quality to get reliable results. The evaluation results would become then a compromise solution between involved parties and evaluation criteria rather than an optimal solution. What do you think about it?

Thank you very much for your contribution to the debate!

Dear Neus Marti,

Dear Neus Marti,

Thanks so much for your comment. I agree with you that looking to improve the quality of evaluations in terms of object and process criteria is clearly needed. This is an area where reconciliation between donors and academia is needed. This study aimed to map out the landscape of issues in evaluations of NRM activities, and moving toward an integrated assessment model for evaluating NRM may well be the way forward to deal with these issues.

Add comment

Plain text

  • Allowed HTML tags: <p> <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Image CAPTCHA
Enter the characters shown in the image.

Latest Blogs

What is Safe Monitoring and Evaluation Practice in the 21st Century?

Monitoring and evaluation practice can do harm. It can harm:

the environment by prioritizing economic gain over species that have no voice
people who are invisible to us when we are in a position of power
by asking for...

The GEF and Cleantech: Evaluating Efforts to Build an Innovation Ecosystem

Global factors such as technological advances, lowered costs, available capital, consumer demand and climate change have been encouraging the development and deployment of clean technologies as part of low...

Recent FAO evaluation highlights the role of GEF in programmes addressing integrated natural resource management for sustainable food and agriculture systems

In November 2018, the Food and Agriculture Organization of the United Nations (FAO) has released an evaluation of its contributions to integrated natural resource management for sustainable agriculture.

CIF Evaluation & Learning: What Are We Learning About the Big Issues in Climate Finance?

The scale and urgency of the climate crisis demands rapid action to prevent a range of consequences – from increased droughts and floods to huge economic losses and more extreme natural disasters.  

All major climate funds and...

Rethinking resilience, adaptation and transformation in a time of change

“How the world deals with large and serious problems is almost certainly going to shape our collective future” is the second line – and a promising start – in this edited Springer volume titled...