Challenging evaluation: an introduction to Outcome Mapping- learning evaluation

Sarah Earl from the International Development Research Centre (IDRC), a Canadian organisation which supports a wide range of development research, introduced Outcome Mapping.

It's difficult to demonstrate that particular programmes result in significant and lasting changes in the well-being of intended beneficiaries when a range of agencies and external factors are involved: no single agency can realistically claim full credit.

Outcome Mapping helps people working in health and development programmes to be realistic about their sphere of influence and to evaluate specific changes in the behaviours, relationships, actions and activities of people they work with. Earl facilitated a short taster workshop for the discussion participants and several key questions were raised:

Identifying “boundary partners”, the people and organisations that programmes work with directly, helps programme teams understand their work as part of a complex, interconnected world. Earl said: “A programme at its best influences its partners. Nothing more.”

Earl emphasised that Outcome Mapping “asks you to measure yourself against your contributions rather than your dreams”. But the methodology nonetheless encourages dreams about what health and development work will ultimately contribute to, whether this is an end to violence against women, drinking water for everyone, or universal access to health care. Earl noted that many development workers have been conditioned to focus only on what they are confident they can control and deliver, rather than their contribution to wider goals.

The Outcome Mapping methodology includes a set of “progress markers” – simple statements that are developed through participatory workshops. Three categories provide a starting point to organise the progress markers: expect to see, like to see, and love to see. But these can be added to or changed, depending on the particular context and purpose of the work.

Swayamsiddha project, Pune, India IDRC’s boundary partner in this project is BAIF Development Research Foundation, an Indian NGO. BAIF’s boundary partners are women’s self-help groups in villages throughout the region. IDRC used Outcome Mapping with BAIF to plan and evaluate the project. BAIF then used the methodology with the women’s groups.

The women’s groups developed progress markers including: “Women can look each other in the eye when describing their experiences of violence.” This couldn’t have been a progress marker for IDRC or BAIF, but helped the women’s groups see a specific behaviour that clearly measured progress in their work. Progress markers in the women’s group love-to-see category included: “Can stop violence against women while it is happening.”

Outcome Mapping can be used to develop a framework to collect data, prioritise what you need to know, and organise thinking, but it “won’t make the decisions for you”, Earl said. Understanding the programme’s context and power relationships is essential.

See IDRC’s website for a full description of Outcome Mapping methodology.

Earl made it clear that Outcome Mapping has to be adapted to each specific context: it is not a product that IDRC is selling. Instead it offers a flexible, practical way of doing evaluation that can be used alongside other evaluation and planning tools.

In response to a question about measuring capacity development, Earl said that using Outcome Mapping would make sure that the programme team were clear about how they defined capacity development as well as developing progress makers to measure it with.

One discussion participant described how she had used Outcome Mapping in a UK-based international campaigning organisation. Many participants were nodding their heads as she described the difficulties of promoting evaluation at the start of a programme or project, as many people still see evaluation as something that’s done at the end rather than as part of planning and ongoing work.

Earl emphasised that promoting evaluation is an ongoing process and you have to be persistent, but that once people “get it” they suddenly appreciate it can strengthen their work and stop seeing it as a “weird abscess”. Earl echoed the issues that discussion participants had encountered:

Donors’ role in evaluation was an issue for several discussion participants who agreed that an open organisational culture was a prerequisite for influencing donors.

In regards to the Swayamsiddha project, Earl was up front about IDRC's relationship with its primary donor in that she didn't consider CIDA to be within their sphere of influence. But Earl suggested that Outcome Mapping can help here too: when she asked herself fundamental questions about how and why IDRC communicated with CIDA, she could see that creative dialogue did take place between individuals in the two organisations.

Andrew Chetley of Exchange said that Outcome Mapping highlighted the importance of the communication process and helped to track where communication was working. We have to keep training ourselves in communication processes. Then we will be able to see what a positive difference it makes when a programme or organisation really starts to communicate effectively.