Health communication, HIV/AIDS, social mobilisation, learning evaluation, capacity building

Full report:Exchange lunchtime discussion 23 August 2001

Geoff Barnard, Head of Information at the Institute of Development Studies (IDS) and Rob Vincent, Learning Coordinator at Exchange give an overview of key issues in evaluation. These include getting stakeholders to participate in the evaluation process and communicating the results to different audiences.

Geoff Barnard noted that the results of many evaluation exercises carried out by the large bilateral donors are not making an impact. He explained the findings of a study on Evaluation for Effective Learning and Accountability (EFELA) carried out for the OECD Development Assistance Committee's working party on aid evaluation during 2000 and presented at a conference in Tokyo in September 2000 (copies of the full study and many of the background papers are available at http://www.ids.ac.uk/efela). Although staff in evaluation departments of bilateral agencies have valuable experience with evaluation, the learning generated is not having impact. One reason for this is that evaluation is often trying to deal both with learning and accountability, yet these are two different things.

There are many different audiences for evaluation feedback, and it is crucial to identify them clearly, early on in the process. There is also a need to prioritise them and decide whose needs can be met. At the same time audiences for learning are not the same as those for accountability. This is also the case within an organisation, where the audience for accountability is often at a different level from the audience for learning.

Key communication lessons

Dilemmas in evaluation

Evaluation is not an easy thing. Some real dilemmas include how to deal with:

These are very real tensions and it is not surprising that people find them difficult. They will not be changed overnight. This leads to two challenges:

Institutionalising learning

Institutionalising learning is essential if real progress is to be made. For learning to be institutionalised, it cannot be optional. However, there are obstacles and it is important to recognise and deal with those obstacles. They include:

Senior management has to champion learning and recognise that it may mean working in different ways. Evaluation is recognised as a key part of a learning organisation, but most know that they are a long way from having the learning process in place. Making time available for learning and reflection, and rewarding that process is critical to having it institutionalised.

Sharing learning

This is vital, but it is an equally big challenge to get stakeholder involvement in the process. This needs to start at the beginning of the project cycle and it is complicated, takes time, and needs new attitudes and skills. For many stakeholders and partners, involvement in monitoring may be more important to than the evaluation.

There are also risks involved in sharing learning - both personal and institutional. People and organisations need to feel confident and secure to acknowledge mistakes and demonstrate learning. However, this is essential if the rhetoric of stakeholder involvement and of participatory monitoring and evaluation (PM&E) is to become a reality.

Evaluation feedback in a health communications context

Compared to the evaluation feedback from major programmes that bilaterals are considering, most work in health communications is on a different scale, and that includes smaller organisations.

There are also different pressures and audiences. For example, most evaluation for health communication NGOs is triggered by the funding loop – the need to secure funds for the next phase of activity – and is often seen as a negative pressure from a learning perspective.

For health communication projects, lack of communication skills should not be a problem (in theory). However, there are many similar dilemmas and blind spots. The key challenge for everyone is how to facilitate 'real learning'.

Rob Vincent took up the discussion by focusing on the two concepts of learning and accountability. He led a brainstorm with the audience to identify what people thought the words meant.

Does evaluation practice help learning?

Although the objectives of a project might fit into a log frame and be captured in indicators to facilitate measuring the outcomes, changing factors through the project’s cycle are not often reflected, and it is difficult to deal with the unexpected. Maia Green (forthcoming) has commented that ‘project space’ is developed in participatory workshops, but asks whether this reflects what happens on the ground in actual real-life situations.

Rob referred to some findings from evaluations carried out by Healthlink Worldwide, where through its work with partners, effective feedback and learning mechanisms were ensured by good ongoing communication and dialogue.

Where formal frameworks were in place, these were based on an existing foundation of established communication channels and dialogue. In this way some formal evaluation tools may be useful, but they need to be used flexibly, if they are to contribute to learning.

Reflection is critical for learning, and this came up frequently in evaluations with Healthlink partners. There is a need for more time to reflect. It is exciting to see that donors are beginning to recognise the importance of having this time and of the need for reflection.

Looking at ‘accountability’, it is interesting to note the emphasis on donors and financial or institutional constraints. Sometimes, this gets extended to the public at large, in that government donors ultimately are responsible to their citizens, but what about accountability to the very people that development is supposed to help? What about the ‘beneficiaries’- people living in poor communities?

The former kind of accountability is a part of the growth of new managerialism, which has spread across a number of areas of public policy in the north, if not globabally, according to work in the anthropology of policy (Shore and Wright 1997). And at a time when the transparency of governance is in some doubt, with the growth of unelected ‘quangos’, regulatory bodies, and the spread of corporate power, it is important to look at who is using this notion of accountability? We may need to look at a different sense of this word, and look at ways of taking into account the needs of communities.

It is interesting to note that in the case of the Bolivian Miner’s ‘Radios Mineras’ popular Radio programming in the 1970s, effective monitoring and evaluation mechanisms grew directly out of the desire to consult and adequately express the priorities of the mining communities the radio programmes served (Dagron 2001). In the light of this we can ask whether we need ever better techniques of monitoring and evaluation?

Or do we need to look at ways in which people can set their own agendas and encourage self-determination? Making use of participatory approaches is a move towards including others, but does that also have some dangers too, in that it might be co-opting people into development agendas rather than allowing real self-determination?

Questions and answers

What about the tensions around the disclosure of information in evaluations? How much of the truth is actually revealed in evaluations? You got far more learning in an off-the-record talk.
Geoff Barnard replied that for most government agencies or institutions like the World Bank, there is a realpolitik involved in evaluation. Whilst there is a trend to the position that disclosure in the long run is a good thing, there is a practical limit to how much you can disclose. Certainly some of the

Scandinavian agencies believe that disclosing mistakes is beneficial in the long term. However, most of the agencies also acknowledge that they need to do more. Finding ways of capturing the learning during the course of the evaluation is probably more important, and it is often further back in the evaluation that the learning goes on. It was noted in discussion that corruption was not disclosed in evaluations. It was easier for a large organisations to divulge such information, but particularly difficult for small organisations, since staff could be in very real danger, including personal danger.

Is the tension between learning and accountability inevitable and simply something that has to be managed? It is hard to do both at the same time very well.
Geoff commented that it should be clear what the evaluation is for and where it is placed in the spectrum of learning and accountability. Of all the major agencies, IFAD has gone further along the learning approach. It tells its project partners that evaluation activities should be done primarily for learning purposes. Most NGOs are caught between donors and partners. Evaluations done for donors are for donors. However, if you want to do good formative learning together with partners, they need separate funding put in for this. It would be nice to think that the EFELA report provides a way to advocate this with donors.

Group discussions

Three small groups looked at three questions. Below is a summary of their discussion.

1. How do we deal with the tension between what many see as the dual roles of evaluation, namely 'learning' and 'accountability'?
The group concluded that although there were very different situations in which evaluation was carried out, the evaluation document was not the main objective to achieve. Developing the evaluation report is part of the overall process and what we should end up with is a set of different products to communicate with different audiences. It is important to think about evaluation at the initial planning stage, before the needs assessment and build in allowance for discussion and change of objectives during a project, including allowing funding for that. The process involves shared learning with donors, pushing their boundaries, and encouraging and supporting agencies to build this into funding proposals. There is a need at the beginning and in the middle of project activity to look at the different audiences and see what their needs are. Accountability to the donor is important, but it is also important to add accountability to the beneficiaries of the project. The group also recognised that there were situations where the environment is difficult and there is a need to have realistic expectations about what can be achieved in terms of balancing this tension within a project. It means having to look at different ways of producing learning and sharing it.

2. How do we make learning and reflection a reality?
The group first identified two important constraints: time and resources. In most cases people do not feel they have the time for learning and reflection and in some cases they may not have the resources. Here a very key issue was the degree to which the benefits of learning are seen as important by senior management and that the development of a culture of learning is seen as an organisational approach, rather than being only housed in a department or individual. The next question to consider is who is the 'we'? Whose learning are we talking about? Is it local partners, donors, NGOs in the North? What are the benefits of the learning? Once learning occurs, where is it taken? What happens to it? Once we have learned, how are processes changed and how do we move into advocacy? A key point here was also how to document the learning? Verbal, face-to-face, interpersonal communication was very important for learning. More was learned from that type of interaction than from what is written down. There is usually a more frank and useful outcome. It is also important to find ways of capturing learning for institutional memory, so that learning does not simply stay with individuals and the organisations loses that learning when an individual moves on. Two key concepts around making learning and reflection a reality are the need for honesty and bravery. As one of the members of the group noted: 'If you're not on the edge, you're taking up too much space'.

3. How do we involve developing country partners more in evaluation and lessons learning?
This group first asked the question why only developing country partners? Surely the approach would be similar no matter where the partner was? Then the groups asked whether the need was to involve partners or stakeholders. In the end, the group settled on looking at how to involve developing country stakeholders, which might include local NGOs as well as expected beneficiaries, and doing this essentially from the perspective of an international NGO. The next question to consider was who is the learning for? If it is for donors, the evaluation reports are likely to be different than if it is for the local NGO, or for other international NGOs. However it might be useful if donors got the reports aimed at other bodies! The group was convinced that the best approach was to ensure that all the various constituencies are involved together in the evaluation process, right from the time the process is started. They should then come back together at times to regularly review learning, even before evaluation reports are written up. With all stakeholders it is important to identify the 'what's in it for me?' factor, which increases the motivation to be involved. If they are involved in setting the parameters and indicators, then using the degree of local participation as an indicator will help to stimulate and enhance involvement.

Issues in evaluation for health and disability communication