Using SNA to evaluate impact in complex systems

jgarcia2's picture

My name is Jeneen R. Garcia. I’ve been a full-time evaluator at the Independent Evaluation Office of the Global Environment Facility (GEF IEO) for the last seven years. The GEF is the largest multilateral funder of environmental programs worldwide. Because the programs we evaluate almost all take place in complex social-ecological systems, we constantly need to seek out new methods for dealing with complexity.

One of the methods we’ve used is Social Network Analysis (SNA). In one evaluation, we wanted to assess the role of the GEF in increasing interactions among environmental actors at the regional level. Two things made this system complex:

1) the work of these many actors intersected, but they had no hierarchical structure, and

2) interventions took place at multiple scales, which ultimately shaped interactions at the regional scale.

It’s hard to keep track of what everyone says they’re doing and who they’re doing it with. By mapping the relationships among actors, SNA allowed us to see how well-connected the actors in the region are, and which ones are key to keeping the network well-connected.

Because it was an impact evaluation, we also needed some sort of counterfactual to compare our observations with. The big problem was, it is practically impossible to “randomly select” a region that is comparable to any other, much less find a high enough number of them to ensure statistical robustness. In this case, we were looking at the South China Sea, a region with several territorial conflicts, and which GEF has funded for > 20 years. How could we find a region to compare with that?

Hot Tips

  • Instead of looking outward, we created a scenario of the same region without GEF’s presence. We did this by redoing the SNA with the same set of actors except the GEF. The result was, without GEF support, some actors that were important at the country level became disconnected from the regional discussions.


  • We did not rely on this analysis alone to assess the impact of GEF funding in the region. We triangulated it with field visits, interviews at multiple scales, document reviews, environmental monitoring data, global databases, and satellite images, among others. A wide range of evidence sources and methods for analysis is your best defense against data gaps in complex systems!

Rad Resources:

To find out more about which SNA measures were used to come up with our findings, you can check out this paper that I wrote up on the analysis.

You can also see how this analysis fits in with the larger impact evaluation by reading the full report.

For more on the basics on the basics of SNA and how it can be used in evaluation, you can explore this Prezi I made. It includes links to evaluations, software, and other resources related to SNA. (CAVEAT: I delivered that presentation to a Spanish-speaking audience and haven’t translated it yet. My apologies to the non-Spanish speakers!).

This blog post was originally featured on AEA365 | A Tip-a-Day by and for Evaluators. The original posting can be found here.

Add comment

Plain text

  • Allowed HTML tags: <p> <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Enter the characters shown in the image.

Latest Blogs

What is Safe Monitoring and Evaluation Practice in the 21st Century?

Monitoring and evaluation practice can do harm. It can harm:

the environment by prioritizing economic gain over species that have no voice
people who are invisible to us when we are in a position of power
by asking for...

The GEF and Cleantech: Evaluating Efforts to Build an Innovation Ecosystem

Global factors such as technological advances, lowered costs, available capital, consumer demand and climate change have been encouraging the development and deployment of clean technologies as part of low...

Recent FAO evaluation highlights the role of GEF in programmes addressing integrated natural resource management for sustainable food and agriculture systems

In November 2018, the Food and Agriculture Organization of the United Nations (FAO) has released an evaluation of its contributions to integrated natural resource management for sustainable agriculture.

CIF Evaluation & Learning: What Are We Learning About the Big Issues in Climate Finance?

The scale and urgency of the climate crisis demands rapid action to prevent a range of consequences – from increased droughts and floods to huge economic losses and more extreme natural disasters.  

All major climate funds and...

Rethinking resilience, adaptation and transformation in a time of change

“How the world deals with large and serious problems is almost certainly going to shape our collective future” is the second line – and a promising start – in this edited Springer volume titled...