What Do Evaluations Tell Us about Climate Change Adaptation? A Realist Review

This is a co-authored piece by the following authors: Takaaki Miyaguchi(Associate Professor, Ritsumeikan University (Japan), takaaki@fc.ritsumei.ac.jp) and Juha I. Uitto(Director, GEF Independent Evaluation Office, juitto@thegef.org)

Hi, I'm Taka, and I’d like to give you a preview of what my co-author Juha and I are going to discuss at the 2nd International Conference on Evaluating Climate Change and Development in November. Our paper analyses Climate Change Adaptation (CCA) programmes and projects implemented by United Nations Development Programme (UNDP) and its partners that have:

  1. finished initial implementation cycles; and
  2. been already evaluated.

We have conducted a meta-analysis of the final evaluation reports of the CCA projects from nine countries, i.e. Armenia, Egypt, Malawi, Mozambique, Namibia, the Philippines, Tanzania, Turkey and Zimbabwe.

Takaaki Miyaguchi and Juha I. UittoIn this analysis, we have adopted a realist approach. The realist approach belongs to the school of theory-based evaluation. It emphasizes the underlying assumptions about the way certain interventions are expected to yield certain outcomes. Theory-based evaluation is guided by a programme theory that intends to explain how an intervention is to achieve its outcomes (Pawson and Tilley, 1997). If a more judgmental approach focusing on quantitative measures and correlations were adopted, this review would not have been able to answer such questions as "for whom a programme/project may work", "under what circumstances", and "how". These considerations are indeed of high relevance for policy makers, programme implementers and donors.

In a realist approach, identification of CMO (context, mechanism, outcome) configurations is vital. We have reviewed all the CCA evaluation reports, and itemized every evaluative remark/mention in those reports by assigning either a positive (+) or negative (-) attribute. This step has at the end generated a total of 577 items (of remarks). By fully utilising these itemized remarks, we have sought to identify the key CMO configurations for the four evaluation criteria, i.e. relevance, efficiency, effectiveness, and sustainability. Here are some of CMO configurations identified through the analysis:

Context   Mechanism   Outcome

For Relevance Criterion

Host government and line ministries have already identified national and sectoral CCA priorities, and understand programme's objective
Local CCA priorities are identified by the local government and local partners
+ Close coordination and working relationship with the national and local government enables both partners (government and UN implementing agency) to develop appropriate CCA programme = (+) High relevance of programme strategy and intervention components with the national, and global priorities

For Efficiency Criterion

Sector specific and focused programme design
Relevant stakeholders are supportive of UN and well aware of CCA issues and risks
+ Partnerships with stakeholders are built at an early stage, where they feel more motivated to participate in the programme = (+) High stakeholder involvement

For Effectiveness Criterion

Specifically identified types of participants are well aware of the climate risks
Specific types of skills that they need to acquire are clear to them
+ Training and transfer of needed techniques and practices for the relevant people facilitate these skills, techniques and knowledge to be applied and used = (+) High level of adaptive capacity
(+) High level of utilization of adaptive measures

For Sustainability Criterion

Relevant stakeholders have strong sense of ownership and have adequate resources and capabilities + Introduction of adaptive measures to the stakeholders/institutions with relevant function/mandate enables "rooting" of these measures by the respective stakeholders/institutions = (+) High likelihood for initiatives & replications

During the conference, we will be sharing with you all the identified CMO configurations for further discussion. However, one major limitation we have faced is the fact that such CMO configurations were constructed for each of the four evaluation criteria (i.e. relevance, efficiency, effectiveness, and sustainability) that were pre-set by the CCA evaluation reports. What this means is that the CMO's "O" (outcome) is not consistently treated as the programme’s overall outcome, but as a sub-component that comprises these evaluation criteria. For example, the degree of financial execution as CMO's outcome (as one sub-component that can lead to efficiency) is not a final programme outcome leading to higher level of adaptive capacity or replication effects.

Although we are of the view that applying a realist approach even to these pre-set evaluation criteria has yielded some useful results to the CCA evaluation field and future CCA programming, the questions we want to cast to the reader is what types of methodological challenges you have faced in adopting a realist approach and/or identifying CMO configurations for CCA project(s). We look forward to receiving your comments.

Reference

Pawson, R. & Tilley, N. (1997) Realistic Evaluation. London, Sage.

Add comment

Plain text

  • Allowed HTML tags: <p> <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Image CAPTCHA
Enter the characters shown in the image.

Latest Blogs

What is Safe Monitoring and Evaluation Practice in the 21st Century?

Monitoring and evaluation practice can do harm. It can harm:

the environment by prioritizing economic gain over species that have no voice
people who are invisible to us when we are in a position of power
by asking for...

The GEF and Cleantech: Evaluating Efforts to Build an Innovation Ecosystem

Global factors such as technological advances, lowered costs, available capital, consumer demand and climate change have been encouraging the development and deployment of clean technologies as part of low...

Recent FAO evaluation highlights the role of GEF in programmes addressing integrated natural resource management for sustainable food and agriculture systems

In November 2018, the Food and Agriculture Organization of the United Nations (FAO) has released an evaluation of its contributions to integrated natural resource management for sustainable agriculture.

CIF Evaluation & Learning: What Are We Learning About the Big Issues in Climate Finance?

The scale and urgency of the climate crisis demands rapid action to prevent a range of consequences – from increased droughts and floods to huge economic losses and more extreme natural disasters.  

All major climate funds and...

Rethinking resilience, adaptation and transformation in a time of change

“How the world deals with large and serious problems is almost certainly going to shape our collective future” is the second line – and a promising start – in this edited Springer volume titled...