Using SNA to evaluate impact in complex systems

jgarcia2's picture

My name is Jeneen R. Garcia. I’ve been a full-time evaluator at the Independent Evaluation Office of the Global Environment Facility (GEF IEO) for the last seven years. The GEF is the largest multilateral funder of environmental programs worldwide. Because the programs we evaluate almost all take place in complex social-ecological systems, we constantly need to seek out new methods for dealing with complexity.

One of the methods we’ve used is Social Network Analysis (SNA). In one evaluation, we wanted to assess the role of the GEF in increasing interactions among environmental actors at the regional level. Two things made this system complex:

1) the work of these many actors intersected, but they had no hierarchical structure, and

2) interventions took place at multiple scales, which ultimately shaped interactions at the regional scale.

It’s hard to keep track of what everyone says they’re doing and who they’re doing it with. By mapping the relationships among actors, SNA allowed us to see how well-connected the actors in the region are, and which ones are key to keeping the network well-connected.

Because it was an impact evaluation, we also needed some sort of counterfactual to compare our observations with. The big problem was, it is practically impossible to “randomly select” a region that is comparable to any other, much less find a high enough number of them to ensure statistical robustness. In this case, we were looking at the South China Sea, a region with several territorial conflicts, and which GEF has funded for > 20 years. How could we find a region to compare with that?

Hot Tips

  • Instead of looking outward, we created a scenario of the same region without GEF’s presence. We did this by redoing the SNA with the same set of actors except the GEF. The result was, without GEF support, some actors that were important at the country level became disconnected from the regional discussions.

 

  • We did not rely on this analysis alone to assess the impact of GEF funding in the region. We triangulated it with field visits, interviews at multiple scales, document reviews, environmental monitoring data, global databases, and satellite images, among others. A wide range of evidence sources and methods for analysis is your best defense against data gaps in complex systems!

Rad Resources:

To find out more about which SNA measures were used to come up with our findings, you can check out this paper that I wrote up on the analysis.

You can also see how this analysis fits in with the larger impact evaluation by reading the full report.

For more on the basics on the basics of SNA and how it can be used in evaluation, you can explore this Prezi I made. It includes links to evaluations, software, and other resources related to SNA. (CAVEAT: I delivered that presentation to a Spanish-speaking audience and haven’t translated it yet. My apologies to the non-Spanish speakers!).

This blog post was originally featured on AEA365 | A Tip-a-Day by and for Evaluators. The original posting can be found here.

Add comment

Plain text

  • Allowed HTML tags: <p> <a> <em> <strong> <cite> <blockquote> <code> <ul> <ol> <li> <dl> <dt> <dd>
  • No HTML tags allowed.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Lines and paragraphs break automatically.
CAPTCHA
This question is for testing whether you are a human visitor and to prevent automated spam submissions.
Image CAPTCHA
Enter the characters shown in the image.

Latest Blogs

CIF Evaluation & Learning: What Are We Learning About the Big Issues in Climate Finance?

The scale and urgency of the climate crisis demands rapid action to prevent a range of consequences – from increased droughts and floods to huge economic losses and more extreme natural disasters.  

All major climate funds and...

Rethinking resilience, adaptation and transformation in a time of change

“How the world deals with large and serious problems is almost certainly going to shape our collective future” is the second line – and a promising start – in this edited Springer volume titled...

You Win Some, You Lose Some – Synergies and Trade-offs of GEF Support (Part 2)

You can’t have your cake and eat it, too. At least that’s what we’ve always been told. Having finished the...

You Win Some, You Lose Some – Synergies and Trade-offs of GEF Support (Part 1)

They say multitasking is counterproductive. But what about trying to achieve multiple environmental and socioeconomic objectives in one project?

Using SNA to evaluate impact in complex systems

My name is Jeneen R. Garcia. I’ve been a full-time evaluator at the Independent Evaluation Office of the Global Environment Facility (GEF IEO) for the last seven years. The...