CCSL M&E Workshop on Evidence Gathering

From ccsl ilriwikis

CCSL M&E workshop : Evidence gathering on social learning Date: 16-17 June 2014

Body & Soul, 99 Rosebery Avenue, London EC1R 4RE


Introduction

This workshop is part of the CCSL Evidence Gathering project

Documentation of this workshop

Blog posts:

The photos: [[1]] IF you have photos, please do send them to Cecilia so she can upload them to the CCAFS Flickr. Live-tweeting has been ongoing from the workshop using the decided #CCSL tag. Some fun social media stats: - About 40 tweets with 53 re-tweets = 100 tweets in 2 days linked to photos, materials, quotes, to the Wikispace, background document etc- We "reached" -> 75.000 people (should be taken with a pinch of salt) - A nice team effort: as we had Blane, Ewen, Pete, Cecilia, CCAFS official account, IIED, CARIAA, Marissa and others tweeting away, both quotes, updates, photos and materials.

Workshop objectives/outcomes

  1. To get buy-in and clarity on the workstream we are undertaking with participating projects and to introduce them to each other
  2. To build an agreed framework together for agreeing key criteria for which we should gather the evidence
  3. Testing our framework against some reflections or principles from a body of experts (this may have to happen more iteratively as we go forward)
  4. Timelines and guiding principles for how we will collate and share the growing body of evidence as we go forward

Participation

We have three types of participant - the CCSL family members, our project participants and our "specialists or critical friends" We have been aiming for a broadly even mix but it is looking likely that we will have less specialists at the meeting but may be able to share findings with them going forward. We will also be inviting their opinions on our thinking before the workshop.

Self-assessment and background reading

Agenda

June 16th 9.00- 10.00 - Coffee and introductions 10.00 – 10.15 - Purpose and aims of the meeting [Presentation] 10.15 – 11.00 - Social learning – what it is and what it isn’t, difference between participatory methodology and social learning – really understanding triple looped learning [Presentation] Presentation, short plenary discussion for clarification 11.00 – 11.30 - Coffee + Tea 11.30 – 1.00 - Flash presentations from projects (2-3 minutes - guidance sent in advance) who they are, what they are about, reflections on the social learning self assessment, challenges, opportunities - look for synergies 1.00 – 2.30 - Lunch 2.30 – 3.30 - Flash presentations from projects (2-3 minutes - guidance sent in advance) who they are, what they are about, reflections on the social learning self assessment, challenges, opportunities - look for synergies 15.30 – 16.00 - Coffee + Tea 16.00- 17.30 - Presentation on M&E for social learning – what we know so far, how do we track triple loop learning, what should the KPIs be. Something on good conditions for social learning. Where are the synergies between project self assessments and criteria we know about - start to cluster and agree top ten factors/criteria [Presentation] 17.30 - Drinks 18.45 - Dinner

17th June 8.45 – 9.00 - Coffee + Tea

9.00 – 11.00 - Working groups looking at key criteria and building KPIs or success factors - what would tell us whether our criteria were in place or behaviours changing to ensure good performance? 11.00 – 11.30 - Coffee + Tea

11.30 – 13.00 - Report back from each group - agreement from group around suggested KPIs - look for duplications, refine into framework 13.00 – 14.30 - Lunch 14.30 – 15.30 - Final confirmation of M&E framework/programme and round of commitments to adoption by projects, challenges ahead, what is possible 15.30 – 16.00 - Coffee + Tea

16.00 – 16.30 - Round up, evaluation and next steps and some guiding principles for ways of working (responding to straw man)

Participants

Project participants

  1. Farid Ahmad, ICIMOD
  2. Osana Bonilla, South-South Learning
  3. David Campbell, Shamba Shape-up
  4. Mary Cole, ASSAR - Cape Town Team
  5. Peter Dorward, Supporting Farmer Decision Making Through Weather/Climate Data, Univ. of Reading
  6. Gerd Foerch, Makerere University
  7. Blane Harvey IDRC/CARIAA
  8. Fanny Howland, South-South Learning
  9. Peter Laederach, CIAT Social Learning Under FP4
  10. Kate Lloyd-Morgan, Shamba Shape-up
  11. Rebecca Nabatanzi, IITA Learning Alliance – CCAFS Flagship 4 Results Based Management Trial Project
  12. Pamela Pali, IITA Learning Alliance – CCAFS Flagship 4 Results Based Management Trial Project
  13. Leandro Pinheiro, FAS
  14. Anjal Prakash, ICIMOD
  15. Anja Rasmussen, ICIMOD
  16. Sarah Schweizer, ASSAR
  17. Tonya Schuetz, CGIAR
  18. Yiching Song, CCAP (Center for Chinese Agricultural Policy)
  19. Dian Spear, ASSAR - Cape Town Team
  20. Roger Stern, Univ. of Reading (awaiting confirmation)
  21. Krystyna Swiderska, IIED
  22. Joost Vervoort, CGIAR CCAFS Regional Socio-Economic Scenarios
  23. Emma Visman, KCL
  24. Anne Waters-Bayer, Prolinnova, Farmers Innovation Fair Model
  25. Alex Zizinga, Makerere University

M&E Specialists

Georgina Cundill Claire Hutchings, ASSAR Richard Taylor, SEI Barbara Van Mierlo, Wageningen University

CCSL Team Liz Carlile Pete Cranston Wiebke Foerch Ben Garside Christine Jost Ewen Le Borgne Cecilia Schubert Philip Thornton Marissa van Epp Teresa Corcoran Natalie Brighty

Description of the projects introduced

Adaptation at Scale in Semi-arid regions (ASSAR): aims to improve understanding of climate change in semi-arid areas across Africa and Asia and is one of the consortia within the CARIAA program (see below). ASSAR is trialling a number of innovative approaches, including a Transformative Scenario Planning (TSP) social learning process that enables players to understand holistically the biophysical and political-economic system they are part of, and then to develop strategies that can change the vulnerability of the system to climate change as well as research on barriers and successes for effective communication of climate information and knowledge, via social learning and game playing. Learn more: [[2]]

Collaborative Adaptation Research Initiative in Africa and Asia (CARIAA): is a program supporting four consortia (ASSAR, DECCMA, PRISE, HI-AWARE) to conduct a common research program on climate change adaptation. At the program level CARIAA is taking a social learning approach to promote joint reflection on how CARIAA as a whole (consortia + IDRC) is working collectively to achieve an overall programmatic impact. In short the programmatic impact is better use of evidence in planning and practice to support vulnerable livelihoods in three climate change hotspots of Africa and Asia. CARIAA's aim is that social learning approaches (through face to face and virtual learning reviews) will provide a platform for ongoing review of the assumptions that shape its theory of change. As CARIAA updates this, it will be able to make strategic decisions about what to focus on and how, and to re-draw its impact pathways accordingly. It will also be learning about the contributions that the CARIAA model is bringing to the consortia and IDRC's collaborative effort: www.idrc.ca/EN/Programs/Agriculture_and_the_Environment/CARIAA/Pages/default.aspx


East Africa trial: social learning and case study documentation: research by the International Institute for Tropical Agriculture (IITA) on how to influence and link policies and institutions from national to local level for the development and adoption of climate resilient food systems in Uganda and Tanzania through the integration of the scientific community with policy actors. The project will build platforms and work with existing ones to facilitate knowledge-sharing.

Hoima watershed management: researchers, students, stakeholders assess and determine water resources and water uses within the watershed, document threats to the watershed and its availability of water for local uses; prioritising the problems and looking for solutions. This joint action, watershed assessment, is a good basis for knowledge sharing between actors and the necessary change of attitudes concerning adaptation measures.

Prolinnova: Promoting local innovation in ecologically oriented agriculture and natural resource management (NRM). Prolinnova is an NGO-initiated multi-stakeholder programme to promote local innovation. The focus is on recognising the dynamics of indigenous knowledge and enhancing capacities of farmers: [[3]]

Promoting forest stewardship in the Bolsa Floresta Programme: The program is the largest program of Payment for Environmental Services (PES). The Bolsa Floresta, “forest allowance”, program provides payments to families in the State of Amazonas, Brazil in exchange for keeping the forest standing. The investment decisions are made based on a participatory approach, involving all stakeholders. Stakeholders can decide to invest in social infrastructure, or income generating activities among many things. There is also capacity building among reserve leaders, who meets twice a year to define a common agenda, and propose improvements on the program criteria and process: [[4]]

Regional-economic scenarios: Scenarios are different “what-if” accounts of the future, told in words, numbers, images, maps and/or interactive learning tools.This is a participatory, stakeholder driven activity to co-develop future scenarios that can support decision-making in the CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS) five regions: www.ccafs.cgiar.org/scenarios

Relevant Climate Change Information meets Decision-Making to influence Policy and Institutions: The project, being led by the International Centre for Tropical Agriculture (CIAT), works together closely with ministries and research centers to make sure that the latest climate science is being used for National Mitigation Plans (NAMAs) and National Adaptation Plans (NAPs). Furthermore the project is supporting and training country negotiators to assure that an agreement on climate and forestry is being reached, gender is being considered in NAMA/NAP and that negotiators are well prepared to represent their countries.


Results-based management: The CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS) is transforming how it is doing its research, to focusing on results-based management, including creating Impact Pathways (IPs), Theories of Change (ToCs) and a monitoring and evaluation framework (M+E). CCAFS Flagship 4 (FP4) ”Policies and institutions for climate-resilient food systems” has been identified, as champions for this process and it will be communicating it openly to build internal capacity.

Shamba Shape Up: A farm makeover TV show broadcasted primarily in East Africa. Aims to improve farms across the region to become more productive and climate resilient. Incorporates text-message services, and information materials. By adhering to farmers’ needs, the activities want to change knowledge, attitudes and skills among farmers: www.shambashapeup.com

“South-South” learning: “Southern” countries learn from each other. In this particular case, a climate service project in Senegal, being carried out by CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS) and partners, is being considered for Colombia. By sending a Colombian delegation with relevant stakeholders to Senegal, to learn how they work, and vice versa, the project is seeing how joint learning can support the project and delegates. Learn more: www.ccafs.cgiar.org/blog/sowing-unions-harvest-hope-senegal-shares-approaches-colombia

Supporting Farmer Decision Making Through Weather and Climate Data: This project is an ongoing activity with Reading University, supported by the CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS), that includes developing participatory approaches and tools to support smallholder decision-making related to climate services and agriculture planning as well as linking with met services and household modelling. Learn more: [[5]]



Meeting notes

Feedback from the groups about how they visualise social learning

The

The participants were invited to reflect on how social learning relates to other fields and how these relations could be visualised. All kinds of different results emerged. Here are some.

  • Group Wiebke/Roger, Cecilia and ICIMOD person: talking about up-scaling…
  • Group Joost, Georgina, Blane, ??: 2 constituent parts: change in understanding + learning that moves. Complex ecological systems and complexity… Deep learning and moving to learning – SL as a form of learning or as a process…
  • Group Ann, Anjal, peter and Diana: flower (R4D, KM etc.) constituting the petals. Complexity and gender+equity as butterflies and taking the learning ahead
  • Group Ben, Fanny, Kryztyna, Claire: A/R doesn’t always change explicitly. Take the best bits out of PAR and combine them with change from e.g. OL, institutional change etc. and complexity theory/problems è It’s a mix and being more explicit about all of this. SL isn’t just about changing perspectives but increasing power to people.
  • 20140616 111525.jpg
    Group Marissa, Mary, Tonya, Farid: Who’s the audience for SL? Is it an academic exercise. But the added value of SL relative to other methodologies has much more concrete responses to that question. Outcomes = added value of SL is social differentiation + power dynamics that are explicitly dealt with… A lot of methods are inputs to SL but one thing that is missing is the strategic planning of bringing it together… All these things have the value of adding understanding + see learning moving up to the vision/visualization of an action plan etc. The management field of strategic planning is missing.
  • Group Leandro, >??, ??, ??: SL should work over wicked problems and resilience to eventually find solutions (don’t have to be permanent). Lots of other pathways to solutions: e.g. from wicked problems looked at with systems thinking, CAS, going straight to solutions (ie. It’s a way to address a wicked problem) – another way is via participatory approaches… Participatory approaches are closer. Social media / crowdsourcing etc. are tools for participatory approaches. From action learning to PAR it’s very close to SL. A final group about the how of knowledge being created etc. The difference is the feedback loop etc.

Insights about things that matter to get social learning right

  • Time (to reach results)
  • Room for failure
  • Ownership: bring social learning back into your organisation
  • Understanding of where different partners groups are on the 'journey'
  • Continual (daily) engagement
  • What matters? Developing resilient innovation capacity
  • Opportunities to jump on, to reach results
  • Participatory plant breeding
  • Recognition of the importance of strategic planning and management
  • Are there specific partners leading or advising on the facilitation of SL? Do you have someone champion the SL aspect of your work?
  • What is the aim of the SL dimension of the work? What specifically is it expected to contribute?
  • How is the process and outcomes being documented? How systematically?
  • Is there flexible time and financial resourcing available to act on emergent themes?
  • Stakeholders - more success if shared interest in and analysis of the problem to be solved
  • Recognition of the local innovation capacity (as an incentive for social learning)
  • ICTs to help scale social learning? Must be supported by trusted real people and networks

Bringing it back into the Samoan circle: What factors make or break social learning? What elements are essential to make it work?

  • Need for facilitation / support (to monitor it)
  • Who are we sharing with?
  • We are learning about SL too
  • Time... and people coming in the process
  • Who does it (with)? They have to keep an open mind
  • Listening, admitting failure
  • Do people agree with social learning being deep and moving? Not sure...
  • Fund-raising for social learning --> New funding structure --> Splash in the unknown
  • Room for mistakes (and supporting it)
  • SL right or wrong? It's just a process towards a goal
  • Flexible budget and planning
  • Difficult to describe? Wishy washy?
  • What's distinctive about SL?
  • Relation with participatory methods?
  • No envy and jealousy in social learning, everyone cooperating towards the same goal? That's just not true
  • Power and competition
  • Whose food security are we focusing on?
  • The third loop learning gets very sticky. Its is very difficult
  • Desire to see change "we want to change"
  • Not just doing it with your guts but doing it with planning etc., in an integrated way, not as an add-on
  • Potential of social learning to bring sectors and actors together

Top 5 factors

That matter:

  1. Looped learning (23)
  2. Engagement (21)
  3. Institutional opportunities and barriers (18)
  4. Power dynamics (17)
  5. Capacity (13)
  6. Intent (11)
  7. Facilitation + process support (10)
  8. Social differentiation (10)
  9. Endogenous processes (10)
  10. Timescales (7)

That we want to monitor:

  1. Engagement (22)
  2. Institutional opportunities and barriers (20)
  3. Capacity (19)
  4. Looped learning (18)
  5. Power dynamics (17)
  6. Social differentiation (14)
  7. Facilitation + process support (14)
  8. Endogenous processes (8)
  9. Timescales (4)
  10. Intent (2)

What’s missing from these KPIs?
20140618 145718.jpg

  • How do you evaluate SL? Or “To build an evidence base on when to use SL”? or “How to take a SL approach to ML&E?”
  • Teamwork learning
  • Gender’ – specifically
  • Adaptation – change itself
  • Acting upon decisions/solutions
  • Options by context: complex / CCRP program
  • Developmental evaluation as the/a facilitation process for social learning
  • Flexibility – Time – Financial
  • Resource use efficiency (of the process)
  • Trust building
  • Freedom to fail
  • Who target outside SLB BoP + how to engage
  • Autonomy (self sustaining continuous learning processes
    20140618 145521.jpg
  • Change in KAS practice I B outcomes
  • Local dynamics of innovations process
  • Flexibility. Adaptive capacity . Capacity to innovate (- resilience)
  • Learning to value local knowledge + solutions (amongst “experts” in gov’t + CGIARs)
  • Increasing collaboration between different actors, sectors and communities

Selection from the KPIs - how to decide which 5 to work on?

The project groups identified what are top KPIs that matter for their own project and we ended up with the following matrix to eventually decide to focus on:

  • Looped learning (see picture to the right here)
  • Institutional opportunities and constraints
  • Engagement
  • Change process (change in resource allocation) - see picture below
  • Capacity of farmers / stakeholders
Change.jpg


Feedback from the M&E specialists

B. Van Mierlo: You are brave. You are adding up to each other and finding common ground on this very challenging issue. I miss the substance of what you work on (climate change, food security etc.). If you are talking to outsiders about this they would criticise the fact that it's just about the process. Learning is never disconnected. The idea of taking stock of social learning is good but how long will you do that. How should you prove that SL has a value. Each case is unique and we're going to develop a common framework etc. or you can think about preparing cases etc. You can do both. You have examples I talked about. A final issue: you are the best example of working on a social learning approach. You should be the evidence of working on that. You could formulate a learning agenda and come up with a really challenging question for yourselves. You can keep on monitoring yourselves on this. You need to show the change in yourselves.

R. Taylor: The system my institute has developed (PMAC): SEI has come up with that system (in 2010) which is quite interesting to reflect. It's based on Outcome Mapping but it's information systems-based. We're not there yet with qualitative information. SEI is working on a lot of similar themes (complex problems, adaptive management)... Very grateful to be invited here and in the spirit of being an expert, I have made all the mistakes possible. My recommendation is to pay attention to how you support staff of their use of M&E. People here are very motivated to use M&E. We always see people who are disengaged with it. Allow other people to give feedback. PMAC has changed institutional management at SEI. Research leaders can see all projects related to a theme etc. On a personal level project managers can review new tasks. M&E takes time and benefits don't come straight away. Outcome Mapping takes time. Collecting information on a wiki - would that be consistent with the set of information you map against a database... Best of luck.

G. Cundill: I've spoken to everyone over the last 2 days but I want to reemphasise something - I hope you appreciate how much you break new grounds. We don't know how yet how to do this. The only take-away message: I wouldn't get too bogged down with indicators. You need agreement to a definition and whether you get to indicators or stories.

Feedback from the four working groups

Leandro: The SL checklist is quite long and we might want to review it via someone remote from the social learning area. We did an exercise on visualisation. We need to work further on... I feel the need to present this back to people in my foundation. There's some language and process to adjust.

Liz: We tried to explore what looped learning looks like. Some of that is emergent, is open, free etc. and some of it is slightly more organised and combines looped learning attributes into a more organised solution or organised learning for a change.

Marissa: KPI group. We took a step back to clarify questions. How do you measure good social learning in terms of your change areas? How do you do that in terms ... How does it lead to better sustainable outcomes in terms of that area? We've been doing two different things. The first question is what the core group was aiming at and others need to develop indicators for that area. We don't have a plan of action... It would be really useful to outline the theory of change for an initiative that encompasses that and we could reference that for indicators. There will be a process to review these 5 areas. There's a bit of frustration to answer the really high level questions (does SL contribute to improve sustainable outcomes) but the first question here is about whether we understand SL to baseline it and improve it. The second question will come further down the line.

Tonya: We've been trying to look at how we can start from a baseline for each of these projects with KPIs and looking at what inputs in the project. What results does this project bring about? It fits nicely...

Next steps

We're on a journey and we wanted to share some items. We need to take this away, we've got good work from you, good reflections from M&E colleagues. We have a partnership meeting. we heard your next steps & reflections about what you want to be working on. We don't have a fantastic funding pool but we have some opportunities to work with these projects and as we go we might find additional projects to work on. We need to synthesise this and road-test it and bring it back to you. We'll pull out that menu of options etc. on the sandbox or a discussion group etc. We hope to come up with a proposal in a couple of months about a piece of work responding to this...

Krystyna: we need to focus on the definition of social learning... We need to work further on that definition. Liz: We need to be much firmer about this. Good moment to review this definition.

People who came to this meeting were interested. Who in this group is interested to be on this journey in a way or another.

Phil's thank you's I take a lot of heart from the fact that we're all a bit mad and super ambitious. We're on a journey somewhere. Thank you for Pete and Ewen for the facilitation, for Liz and the assistance, and to our M&E team. Thanks to all of you and we hope you will stay involved and engaged.

See more pictures from this event: a

media type="custom" key="26277158"
















Agenda for organizers