METHODOLOGY
BBC Media Action is committed to contributing to a stronger evidence base on the role that media and communication can play in humanitarian emergencies.
This research synthesises findings from six different humanitarian emergencies using an evaluation framework that is available here for other agencies to replicate. Collecting more consistent data in this way can help to build a systematic, shared evidence base for the humanitarian sector.
For this research synthesis, BBC Media Action has evaluated the effectiveness of media outputs in the context of six different humanitarian emergencies:
- Syrian refugee crisis in Jordan and Lebanon (2013–2014)
- Gaza conflict 2014
- West Africa Ebola epidemic 2014–2015
- Nepal earthquake 2015
- Somalia drought 2017
- Bangladesh Rohingya refugee crisis 2017–2019
While methodologies differ, all BBC Media Action evaluations use the same evaluation framework and overarching research questions wherever possible to enable a level of consistency and synthesis between findings.
The framework incorporates media metrics developed from theories of change that inform project-level outcomes and criteria from the widely used evaluation criteria adapted for humanitarian interventions by the Organisation for Economic Co-operation and Development’s Development Assistance Committee (OECD-DAC).
These criteria draw on best practice and are commonly used in humanitarian evaluations. They are based on aid principles developed by DAC members in consultation with the World Bank, the International Monetary Fund and the UN Development Programme. They were adapted into evaluation criteria for humanitarian interventions, covering: relevance and appropriateness, connectedness, coherence, coverage, efficiency, effectiveness and impact.
BBC Media Action’s evaluations focus on measuring the effectiveness of media interventions for the benefit of people affected by humanitarian emergencies, assessing their relevance, appropriateness, effectiveness and impact. BBC Media Action’s framework incorporates specific media metrics:
- Engagement and trust under relevance
- Project-level outcomes cover audience members’ motivation, confidence, discussion, knowledge, attitudes and actions.
For this research synthesis, data from evaluations of media interventions was coded into a qualitative framework against each criterion in the framework. Coders were guided on how to code the data and 20% of data from each coder was double-coded to ensure inter-coder reliability.
The overall research question for the synthesis was: What are the achievements and limitations of mass humanitarian broadcasts during emergencies and how can they best serve people affected by crisis?
BBC MEDIA ACTION’S RESEARCH APPROACH
Along with many similar actors in the humanitarian sector, BBC Media Action’s research approach is evolving to meet ethical, methodological and other challenges, and build on existing evidence bases to develop and refine theories of change.
BBC Media Action’s research approach is determined by the specific project design, which draws on and integrates different models and communication theories from the communication for development and health communication sectors. However, while these and other components of communication theory help to test the relationship between variables and their effect on programme outcomes, they cannot always be applied directly to communication in the context of humanitarian emergencies.
BBC Media Action’s communication interventions in emergency contexts are not usually designed to influence specific behaviours or social norms (the usual focus of health communication models). Rather, much of the focus in emergency communication is on enabling people to understand what has happened and equip them with the information, motivation and confidence to work out for themselves how best to respond in their context.
In humanitarian settings, the drivers that media can influence, such as knowledge, attitudes and motivation, become outcomes – since they become elements of psychosocial support that are crucial in alleviating suffering, generating self-efficacy and aiding recovery. These outcomes form the foundational blocks of project designs, with specific subject areas selected according to identified local needs. Previous research (such as audience preferences, and cultural and social motivators and barriers) helps to inform appropriate media formats.
Ethical considerations require special attention in humanitarian responses. BBC Media Action has incorporated elements of the Research for Health in Humanitarian Crises framework and Médecins Sans Frontières’ ethical guidelines into its quality assurance and guidelines for research in this field.
Research design involves a combined approach of continuous assessments and evaluative research throughout different phases of the humanitarian emergency to inform immediate programme outputs and learning. More evaluative research is built in as the emergency moves out of the critical phase, when fewer needs assessments are required. Collaboration is a key part of the approach – BBC Media Action does not conduct primary research if other agencies are able to collect information on information needs and communication behaviours as part of assessments they would be undertaking anyway.
The specific approach and research methodology is adapted to each emergency context. The balance between quantitative and qualitative approaches, the timing, content and sample size of surveys, the number of focus group discussions and so on is all highly context-dependent, with inevitable consequences for research quality and rigour.
For example, in BBC Media Action’s evaluation of its radio response to the Ebola epidemic in 2014–2015, it was impossible to gain direct access to people affected by the epidemic without the research process exacerbating the risk of spreading the virus (for example by asking people to gather together for focus groups). For this reason, BBC Media Action used a short SMS survey administered by GeoPoll in combination with expert interviews, and focus group discussions took place later when access became possible. Similarly, during the organisation’s response to the 2014 Gaza conflict face-to face-research was deemed unsafe, so evaluative research was based on telephone interviews with audience members.
RESEARCH CHALLENGES IN HUMANITARIAN INTERVENTIONS
Humanitarian agencies face challenges in conducting research during emergencies. “Research” in this context takes many forms and includes continuous assessments to understand the needs of people affected and to inform outputs.
This section looks at the general challenges of any research in an emergency context but focuses particular attention on the specific challenges of researching whether media programmes are effective in achieving their intended outcomes.
Outside humanitarian emergencies, assessing whether a communication intervention has been effective normally involves a research design comprising a series of straightforward components. These are:
• Formative research to inform programme content, objectives and main indicators
• A baseline survey to measure these indicators in relation to the target audience (such as existing knowledge, attitudes and practices)
• An endline survey to measure the effect of the programme on those exposed to it.
These surveys are representative of the national or target population and provide reach figures and detailed information about audience members. Alternatively, or in addition, qualitative research (such as focus groups or in-depth interviews) are used to evaluate the programme. And there is ideally a counterfactual or control group to test the robustness of key findings.
For both practical and ethical reasons, few of these elements are possible in an emergency.
METHODOLOGY CHALLENGES
Many evaluation challenges are not specific to communication interventions. Conducting any research during a humanitarian emergency – particularly traditional, “robust” evaluations – poses multiple practical challenges for researchers, who have to manage several, often conflicting, pressures.
These include logistical constraints (such as limited access to beneficiaries), time and resource constraints and donor requirements. Researchers are working in a landscape that has shifted, so earlier baseline data may no longer apply and conducting fresh baseline surveys is often impossible.
In addition, the ethical considerations of working with people affected by a humanitarian emergency limit the types of evaluation that can be done. Deliberately setting up a counterfactual or control group that would not receive aid, for example, is clearly unethical.
One of the biggest ethical challenges is ensuring the research is of “direct benefit” to the people affected by an emergency. This principle is highlighted in the R2HC Ethics Framework, an ethical framework for health research during crises.
Researchers therefore need to create a design that directly benefits people affected by an emergency so that it informs the immediate programme output (not just in the longer term) and is also sufficiently robust to provide some kind of evidence on what is working. Added to this is the need to assess the benefits and the risk of harm to both the people affected by an emergency (who do not want to be burdened by questions and surveys) and the research team going into a potentially dangerous situation.
The reality is that most humanitarian agencies struggle to conduct any form of research during emergencies and are heavily constrained by the environment and pressure to deliver quickly.
CHALLENGES FOR THE MEDIA AND COMMUNICATION SECTOR
Compared with other international development sectors, the communication sector is relatively new. A clear reason for prioritising a communication intervention in an emergency context, and an obvious indicator of success, is the number of people it can reach. This very benefit of mass communication also makes it difficult to evaluate.
Media broadcasts can reach people who are isolated and who other NGOs cannot reach but it is difficult to record how many people broadcasts reach and their impact. Face-to-face services provided by NGOs often have in-built monitoring and feedback mechanisms that can gather data relatively easily, for example on how many people have benefited from access to clean water or how many immunisations have been administered.
In other sectors, such as health and sanitation, the intended outcome is well known. The direct benefits of information and communication cannot be measured to the same degree as, for example, access to clean water. Additionally, communication does not have the same quality measures as other sectors. For example, we know that water needs to be clean to be effective in improving health. While we know that information needs to be accurate, relevant and trustworthy, there are not universally agreed quality standards or specifications on what makes “good communication”.
Communication interventions during humanitarian emergencies have also been historically poor at determining clearly defined, measurable outcomes from the outset. This is partly due to the lack of information needs assessment data, and partly because of the urgent need to get programmes on air – for example, the BBC World Service/BBC Media Action Nepal response was on air within nine hours of the first 2015 earthquake. Communication responders have the additional challenge of having relatively underdeveloped theories of change to draw on.
