Purpose of this section
The purpose of this section is to make sure that the methods you use to evaluate or monitor your work will demonstrate whether or not the work has contributed to reducing socio-economic inequalities in health.
For instance, if you are proposing a capacity-building activity, you will need to evaluate whether it has increased awareness of socio-economic factors that influence health outcomes and behaviours and people’s confidence to act on health inequalities. If you are planning to evaluate an intervention or proposing an evidence review, you will need to test for differential outcomes between different socio-economic groups.
We realise that it may be difficult within the timeline of our funding to evaluate any long-term outcomes and major changes that could result from your work. However, you should establish effective monitoring systems to identify the effects (anticipated or not) your work could have on health inequality.
Imagine the following scenario. You are evaluating a new intervention to assess whether it has increased access to a particular service. The evaluation team also want to know whether increased access leads to improved health outcomes for different social groups, and if not, why. You know that it is not practical to get relevant data to answer this question within your evaluation timeframe. However, your team realise that this evaluation exercise is an opportunity to set up robust structures and methods to collect data that could be used to measure long-term outcomes. In this way, the team are in a strong position to get funding for a longer-term evaluation to assess whether the intervention helped to reduce health inequalities.
This section also focuses on how relevant members of the public are, or will be, involved in monitoring.
Questions to help assess whether evaluation or monitoring plans will provide evidence on how your activity will affect socio-economic inequalities in health
3.1 Which short-term and longer-term effects on health inequalities or the socio-economic causes of these inequalities (or both) will you look at?
3.2 Will your evaluation (or evidence review) provide evidence on:
- (a) unequal access to services to be developed or already provided (for example, whether some ethnic groups have poorer access than others)?
- (b) differential health outcomes (whether the interventions you have evaluated or included in your review are less effective for some groups than for others)?
3.3 If you are evaluating or monitoring the effects of a capacity-building initiative, will your evaluation provide evidence on increased awareness of socio-economic factors that influence health and people’s confidence to act on health inequalities?
3.4 In addition to socio-economic status what key social variables will you use to assess the differential effect of your work on health inequalities? (gender, age, disability, ethnicity, place of residence, occupation, etc.)
3.5 Will you be able to identify any possible unintended effects (positive and negative) of your activity, particularly on health inequalities and their socio-economic causes. If so, how? If not, why not?
3.6 How will you measure how the costs and benefits of your action are distributed across the different groups, including different socio-economic groups where appropriate?
3.7 Are there ways in which you could lose the focus of your activity over time? How will you make sure you maintain this focus?
Involving the public
3.8 Has your target group been involved in designing the evaluation and monitoring?
3.9 If so, how did this affect the design? And if it didn’t affect it, why not?
3.10 How will you involve the public in evaluation and monitoring?
An example of using the Toolkit: Increasing uptake of Health Checks
Section 3: Making sure the evaluation and monitoring can assess the different effects on health inequalities and their socio-economic causes
The team originally planned to set up their intervention in five GP practices which have high numbers of people from BME backgrounds. They chose a further five GP practices with similar BME numbers as control practices. The team proposed to compare the change in uptake of health checks among BME groups in the intervention and control groups. They also planned to carry out research into people’s beliefs about the health checks and what people saw as barriers (or aids) to this service.
In response to the questions in section 3, the team agreed that they needed to redesign the evaluation to look at the socio-economic causes of low uptake of health checks and whether people can act on advice following a health check. They considered the following changes in research methods:
- • Choosing intervention and control GP practices based on different levels of deprivation and numbers of people of South Asian heritage registered with the practices.
- • Matching people in both groups on ethnic background and socio-economic status but also other relevant social categories, for example, where they live, their occupation, gender, religion, education, disability, sexuality and so on, to determine how these factors influence access and health outcomes9-11.
- • Comparing uptake of health checks between matched groups.
- • Comparing differences in change in relevant categories (smoking, diet, high blood pressure, cholesterol and so on) between these groups.
- • Comparing health outcomes.
The team also decided they should collect data on:
- • People’s experiences of health checks and of barriers and aids to accessing them;
- • The type and quality of information and advice that people get during or following the health checks; and
- • Barriers and aids to people’s ability to act on this advice. The public advisers discussed the possible barriers to people taking recommended medicines, including side effects, lack of knowledge about how the medicines should be taken, or cost.
The team felt that they could use in-depth interviews and visual techniques to explore these issues, using a sample of people of South Asian heritage. They would base the sample on people’s socio-economic status and the deprivation scores of the areas where the GP practices were based.
The sample could also be chosen to reflect other relevant social differences as defined across the research framework PROGRESS-Plus, such as gender, disability and age11.
9 Bowleg, L. The problem with the phrase women and minorities: intersectionality—an important theoretical framework for public health. American journal of public health 102, 1267-1273 (2012).
10 Lorenc, T., Petticrew, M., Welch, V. & Tugwell, P. What types of interventions generate inequalities? Evidence from systematic reviews. Journal of epidemiology and community health 67, 190-193 (2013).
11 O'Neill, J., et al. Applying an equity lens to interventions: using PROGRESS ensures consideration of socially stratifying factors to illuminate inequities in health. Journal of clinical epidemiology 67, 56-64 (2014).
Hosseinpoor, A.R., et al. (2014) Equity-oriented monitoring in the context of universal health coverage. PLoS medicine 11(9): p. e1001727.
Cornwall, A. (2014) Using participatory process evaluation to understand the dynamics of change in a nutrition education programme. IDS Working Paper N. 437.
Websites and Visuals
The Health inequality monitoring eLearning module – developed by the World Health Organization (2015)
Good Health Counts: Measurement and Evaluation for Health Equity, resources by the Prevention Institute (USA).
Measurement Tools, resources by the Institute for Innovation and Improvement (UK).
Videos and illustrations of fundamental concepts on health inequalities - developed by the World Health Organization
Interactive graph "Monitoring health inequality: An essential step for achieving health equity". Geneva and the WHO Health Equity Monitor (2014)
Public Involvement resources
Israel, B.A., et al.(2014). Documentation and evaluation of CBPR partnerships: The use of in-depth interviews and closed-ended questionnaires. In Israel, B.A. et al.,(Eds). Methods for Community-Based Participatory Research for Health, Jossey-Bass: San Francisco: CA. p. 369-398.
Patients’ perspectives- resources by the NHS’ Institute for Innovation and Improvement Website.
Real-world HIAT Assessment Examples
STEP Service Evaluation – the service aims to reduce avoidable A&E attendances. Blackburn with Darwen Borough Council
Community Connectors. Sefton CVS
Evaluating the impact of Liverpool Community Care Teams. Liverpool CCG