Service improvement evaluation
We evaluate any service improvement activities/projects that we do, to find out if we have achieved what we set out to achieve.
This section contains information on what evaluation is, why it is important and how to evaluate service improvement projects.
What is evaluation?
Evaluation is a systematic process, designed to enable you to answer important questions about your service improvement project whilst it is underway, and once it has been completed.
Why is evaluation important?
The purpose of service improvement is to change something for the better. A good evaluation will not only tell you if you have achieved what you set out to achieve, but also how and why.
For this reason, evaluation can help to inform and improve a project over time, as information about its progress is regularly fed back in. Combined with project management and monitoring, evaluation can help to keep things on track by telling you if things are going according to plan or if you need to change your plan because something unexpected has happened. Evaluation is therefore key to continuous service improvement.
Evaluation is also important because it helps to spread best practice. If you know what you achieved and how you achieved it, you will be able to describe it to others, who will then be able to try out it out for themselves in their own areas.
How should I evaluate my improvement project?
The most important first step in the evaluation process is to develop a clear and detailed plan for your improvement project. Your project plan will need to describe the aims, actions and resources of your project, as well as who will be involved. Without this information, it will not be possible to carry out a reliable evaluation. Your evaluation should therefore be part of, and be informed by, your project plan. In many respects, evaluation is the ‘study’ phase of the ‘Plan, Do, Study, Act’ cycle, which is described in the methodologies section, and everyone should agree at the very beginning how it should be carried out.
The terms ‘evaluation design’ and ‘evaluation methods’ may be familiar. The first refers to the overall approach your evaluation will take and the second to the tools you might use. There is no ‘best way’ to evaluate your project and your approach will depend on a number of things. These include the aims of your project, the type of questions you want your evaluation to answer, who the evaluation is for and what you intend to do with the information.
Evaluation design
For the purposes of service improvement, there are two main types of evaluation: formative and summative.
Formative evaluation is carried out whilst the project is underway and, together with project monitoring, will tell you if things are going according to plan and, if not, what the unexpected barriers to implementation are. Information from a formative evaluation should be fed back into a project whilst it is still underway so it can continuously improve. A good formative evaluation will therefore provide regular progress reports. Whilst a formative evaluation will gather information about project inputs, actions and outputs, it will also include clear expected outcomes to direct the project plan and the evaluation design and process.
A summative evaluation is carried out at the end of a project. It will tell you if it achieved what it set out to achieve and so focuses on its impact and outcomes. Generally, a summative evaluation alone will not be enough for most service improvement projects, because it will not provide information about why the project did or did not work. For this reason, most service improvement projects should use a combined formative and summative approach.
Figure 1: Evaluation design and methods
Evaluation methods
Most service improvement projects should also aim to gather two main types of information.
Quantitative data are reported in numerical form, such as rates or percentages. Quantitative data enable the impact or outcomes of an improvement project to be assessed by comparing numerical measures before the project begins and after it has been completed. It is also used to measure outputs, such as the number of people who were treated by a particular service within a particular timeframe.
Qualitative data are reported in narrative form, including written and spoken descriptions. It is often used to enable participants to describe their own perceptions, experiences and feelings. Qualitative information is also often used to identify the type of quantitative information that might be useful. For example, the findings of a focus group discussion could inform the content of a structured questionnaire survey – or vice versa.
Health care professionals often think it is easier to analyse qualitative data than quantitative data. In fact, the opposite is often true. This is because it is very easy for people carrying out the evaluation to misinterpret or misunderstand people when they describe their perceptions and feelings using their own words and language.
Figure two sets out a number of tools commonly used in quantitative or qualitative evaluations. However, there is many more so further information and advice should be sought before deciding which to use.
Figure 2. Common types of evaluation tool
| Evaluation tools | Quantitative | Qualitative |
| Focus group discussions | * | |
| Unstructured interview | * | |
| Structured interview | * | |
| Observation | * | |
| Structured survey questionnaire | * | |
| Semi-structured survey questionnaire | * | * |
| Diaries | * |
Will I need ethical approval to carry out my evaluation?
The safest answer to this question is to assume you do, but the Research and Development Manager of your local NHS Trust will be able to advise you. Whilst sometimes time consuming, it is far easier to take the necessary steps to ensure your evaluation will protect the interest of the people that use the services and staff before you begin. Otherwise, at best you risk having your work discredited and at worst, doing harm to others. Further information on the ethics of research and evaluation and how to get ethical approval is available at www.corec.org.uk