Most Significant Change - monitoring, evaluation, communication

Most significant change is a participatory monitoring system that can deal with the unexpected. It draws meaning from actual events, rather than being based on indicators. The method involves systematically collecting stories which are are then analysed, discussed and verified. The stories capture changes in the lives of ‘beneficiaries’, their colleagues and in the character of their participation. The method also helps to identify why change happens.

Pat Norrish, an independent consultant, outlined the basic approach. Rick Davies, who developed the methodology in Bangladesh answered questions. Daniel Jones, from VSO, talked about VSO’s recent use of the methodology for capturing volunteer learning.

To frame the discussion there was a brief presentation on the basics of ‘Most significant change’ from Pat Norrish, some responses from Rick Davies, who developed the methodology in Bangladesh, and Daniel Jones, from VSO, talking about VSO’s recent experience using the methodology for capturing volunteer learning.

Pat outlined the basic components of the approach, based on the experience from facilitating a workshop in Bangladesh of the Support to Regional Aquatic Resources Management (STREAM) initiative, part of the Network of Aquaculture Centres in Asia-Pacific (NACA). This initiative sought to facilitate the understanding of poor people’s livelihoods, identify appropriate strategies and processes, promote communication about these issues, and facilitate the voices of the people in the development of policies and institutions set to support them.

Components of ‘Most Significant Change’ (MSC)

The most significant change approach is a participatory monitoring system not based on indicators, drawing meaning from actual events, and therefore able to deal with the unexpected. The method enables the selection of stories that capture changes in the lives of ‘beneficiaries’, their colleagues and in the character of their participation, as well as a range of other participants and ‘stakeholders’. The method also helps to identify how change happens and why.

A systematic approach, it involves asking the same questions of everyone and rigorously and regularly collecting stories. These stories are then subject to analysis, discussion and filtering, verification and documentation. Pat summarised the stages in the process as:

1. Selection of domains to be monitored

2. Deciding the reporting period (frequency)

3. Deciding the participants (different stakeholder groups)

4. Phrasing the question

5. Deciding on the structure of participation

9. Monitoring the monitoring system

Pat then raised a number of questions that she put to Rick Davies:

How do you make the change from anecdotes to credible significant change examples? Rick noted that the key thing here was the systematic collection of the contextual information framing the anecdote – who said it, where, when and to whom. In addition people were asked to explain their stories, and to give reasons why they considered this change had occurred.

Choice of examples - how is agreement reached at the different levels? The method attempted to deal with the variety of stories gathered at different levels and in different places by framing their significance in terms of the agreed and declared objectives of the project concerned. Common methods, such as group discussions, votes and ranking were used to select the stories on this basis. Ultimately these processes of selection remain dependent on the priorities of the organisation.

Donors like to see indicators in the log-frame (monitoring the expected) - so what do you present to donors if you want to use MSC? Rick suggested that it was best to say that the ‘most significant change’ methodology was being used and whether it was in place of, or a complement to, the usual evaluation approaches would be outlined. He also suggested that one way of guaging the succes of the method might be to demonstrate that the most recent changes were the most significant (though this would need a large enough sample).

The VSO experience

Daniel Jones from VSO gave us a refreshingly frank and honest account of some of the strengths and weaknesses of the MSC method as used by VSO, who recently adopted the method as a standard part of the reporting for volunteers finishing their placements. Stories were selected at country level, and the programme office level, with limits placed on the number to narrow the number from roughly 500 from volunteers to around 150 by May this year. A further selection by region then produced around 60 stories covering Africa and the rest of the world, and of these around 24 went to DFID.

The process acted as a valuable way to build dialogue and communication within the organisation, as well as provide evocative material for ‘marketing’ purposes too. Daniel noted that the full process of feeding the stories back and forth between the levels did not happen (in this way there were some ‘short-cuts’ taken), however, the final 24 stories were reported back to country programmes.

Daniel also noted the importance of recording the process of selection, something VSO did not adequately do on this occasion, largely due to the short time period in which it was completed. He also noted the general lack of negative stories that came through the process. A worry that the programme stories would get ‘fast-tracked’ in favour of those from the volunteers however, was not confirmed.

The method was powerful for highlighting the importance of the relational and interactional processes that go on at the ground level in projects, as well as the importance of context. In this way the process was useful for learning, but less so for upwards ‘accountability’. It was hard to convince donors of the value of the method due to the lack of emphasis on the quantitative.

In future the stories could usefully be collected more regularly, rather than only at the end of volunteer placements, and they could be ‘databased’ in some way to allow them to be systematised. The stories may subsequently inform annual country reviews, and at first glance, a number of them had some relevant evidence in this regard. It would be good however, to further integrate the process into VSO’s other systems and processes to maximise the learning.

Group Discussions: How can MSC be used to monitor health communication work?

One group noted the very evocative and ‘communicative’ output of the method, and the value of the process of communication, and the ongoing ‘conversations’ that the method seemed to entail. It was a valuable way of ‘dignifying the anecdote’; finding a place for storytelling, something that was an ‘experience near’ way of capturing people’s concerns, and giving them validity.

The following concerns were raised in the group discussions:

Evaluation of the meeting

Participants enjoyed hearing about what was seen as a ‘fascinating’ and ‘promising’ new approach. The initial presentations were thorough and complemented each other well, and the background information sent out beforehand to frame the meeting was appreciated. It would have been good to allow time for responses from the speakers to the questions raised in the discussions, and some way of building further on the reporting of the break-out groups. As ever time was seen as too short.

Most Significant Change: An evolutionary approach to monitoring that facilitates organisational learning