Learn how to evaluate the process, impact, and outcomes of an intervention and make needed adjustments. Show
A Community Health Center conducted an evaluation of its program to promote physical activity among those with higher risk for heart disease. The evaluation showed mixed results. A small number of participants (15%) had very good outcomes. They had marked increases in physical activity and improved nutrition. Their fitness improved and they lost weight. As predicted, their blood pressure dropped, their pulse rates went down, and they reported feeling more energized. They reported high levels of satisfaction with the program and results. A large majority of the original group (70%) exercised, but not as regularly as hoped. The health benefits for this group varied, with several reducing blood pressure at least slightly, and the rest maintaining the levels they had entered with. A final group (15%) consisted of dropouts – several participants left the program, most within a short time – and other people who simply never managed to exercise on any schedule at all. There was virtually no change in their weight, blood pressure, or sense of well-being...except for a small number that had relatively positive results. What could the Community Health Center do with these results? It knew that, while the intervention apparently worked if people stuck with it, the program was only partially successful. How could it use the evaluation to improve the program, and so improve the health of those it served? This chapter so far has discussed the elements of conducting a research-based evaluation. But evaluation itself is only a means to an end: a tool to help you see what is happening so you can improve the effectiveness of your work. In this section, we’ll examine how you can use your research – the results of your evaluation – to do just that. What do we mean by refining the intervention?Data allow you and other group members to critically reflect on your work and look for opportunities to improve.Some key reflection questions that you and your group might consider:
The reflection questions you ask will depend on the nature of your intervention, but the above set of questions is a good starting point. Consider holding a meeting or brief retreat where the evaluation results can be presented through graphs and charts, and key questions can be discussed. Such a meeting might benefit from an experienced facilitator to keep the process moving toward consensus for specific recommendations on how to improve. Refining the intervention is the process of making your work more effective by using data collected from your evaluation. Depending on what you’ve learned from this data, you might want to:
It will be important for you to meet with other members of your group to review the data, identify key areas for improvement, and brainstorm and come to consensus on how to address issues that have been raised. Careful attention to your evaluation results can help inform which courses of action you should take to improve your efforts.
Why should you use your evaluation research to refine the intervention?Refining the intervention is the primary purpose of an evaluation. If you find out that your intervention wasn’t effective, you have three choices: you can quit; you can blindly try another approach; or you can use your evaluation research to guide you towards a more effective intervention. Using evaluation results is vital: it points you in the direction that your research tells you is apt to be most helpful. Using research to help you choose your course of action also establishes you as a credible and practical organization, one that’s concerned with what works. That kind of reputation is likely to increase your opportunities for getting funding and other resources, and to gain and sustain your community support. Most importantly, it helps the group succeed in addressing the important problems or goals of your community. When should you refine the intervention?The short answer to this question is “constantly.” Monitoring and evaluation should go on throughout the life of the program or project, and should be used to adapt and adjust what you do on an ongoing basis. In practical terms, it’s wise to reevaluate your work regularly – once a year is typical – and make any major changes at that time. Of course, you can and should make minor adjustments throughout the year, based on your monitoring and on feedback from participants, staff, and others who implement or experience the intervention. There are, in addition, some specific times when adjusting your work can be especially helpful:
Who should be involved in refining the intervention?The best plan here is to involve a number of stakeholders, depending to some extent on who has been involved in the planning and evaluation of the effort. Some people who definitely should take part:
How do you refine an intervention based on research?Changes in interventions should be focused on one or more of the three aspects of evaluation: Process (both your process – activities implemented, doing what you intended, etc. – and participants’ process – what did they actually do?), impact, and outcomes. You have to examine each of these separately, and ultimately integrate them to decide what adjustments you need to make in your intervention. Each aspect of the evaluation builds on what comes before. In order to have the impact you want, you have to put together and run your program well, and that’s a matter of process. If your process didn’t go properly, then you haven’t really conducted the program you planned for. If you didn’t get the impact you hoped for, it may be due to the fact that you simply didn’t do what you planned, and the first adjustments should be to the process, to ensure that the intervention is implemented as intended. Similarly, to get the outcomes you intend, the program has to have an impact on the appropriate risk and protective factors or other environmental conditions. If the program had the impact you envisioned, but not the outcomes, then adjustments need to take place at the impact level, perhaps in the risk and protective factors and/or conditions that influence outcomes. ProcessAn evaluation of the process of your effort compares what you planned to do with what you actually did. Process has a number of elements to which evaluation might be applied. They encompass both logistics (the handling of details, such as finding space and buying materials) and program implementation (methods, program structure, etc.). These elements can include:
If all or most things went as planned, and any that didn’t were trivial, you’ve essentially done what you set out to do. If they didn’t, there are a number of possible reasons for changes in the intended process:
The deviation from your plan may have had made very little difference at all, or it may have made all the difference. Some differences might be positive – a delay might make it possible to find a more stable funding source; a change in method might make for a more effective program – but they’re still differences. It’s worthwhile to understand what changed to make sense of the evaluation results and to make any needed adjustments.
ImpactYour program or initiative’s impact is the effect it had on the environmental conditions, events, or behaviors that it aimed to change (increase, decrease, sustain.) In most – but not all – cases, the immediate impact of the program is not the same as the eventual intended results. Generally, a program aims only to influence one or more particular behaviors or conditions – risk or protective factors. The assumption is that such influence will then lead to a longer-term change, which is the ultimate goal of the program.
Your process might have gone perfectly – you might have done exactly what you set out to do – and might still have had no impact on the risk and protective factors you targeted. By the same token, you may have ended up running a program markedly different from the one you planned, and still have had the impact you hoped for. The results of the process evaluation will tell you how closely you stuck to your plan in setting up and running your program. The results of your impact evaluation will tell you whether your program made the changes or intended results. Your program worked as you planned if the behaviors and risk and/or protective factors changed in the ways you intended. The big question that remains in this case is whether the changes your program influenced led to the ultimate outcomes you were working toward. We’ll consider that when we look at outcomes a little later in the section.
If your program actually had a negative impact on the targeted behaviors or risk and/or protective factors – the intervention aimed to increase childhood immunizations, and fewer children were immunized, for example – it is important to look more deeply into what is happening. Some possibilities:
Just as you might find that your process went well and your program still didn’t influence the risk and protective factors you meant to, it’s possible that you created exactly the changes you intended in risk and protective factors, and the program still didn’t achieve the outcomes intended. We’ll look at outcomes to consider that situation. OutcomesThe outcomes of an intervention are the changes that actually took place as a result of it. The goal of an intervention is usually not just a change in behavior or circumstances, but the changes in community health and development that occur as a result of that immediate change. A tobacco control program, for instance, aims to help participants avoid or quit smoking: that’s its impact. Its real goals – the hoped-for outcomes of the program – are reduced rates of heart disease, lung cancer, and other smoking-related diseases for participants and their family members. The ultimate outcomes may take years to assess, but others – like the blood pressure goals of the Health Center exercise program, or the results of a job training course – can be determined at or soon after the end of the intervention. Outcomes are the true measure of the success of the intervention, because they are the reason it was conducted in the first place. However, the impact made – such as changes in community programs of policies– can be an important intermediate outcome since it can take years to see changes in longer-term outcomes. The program produced the intended outcomes If the program produced the outcomes you intended, congratulations: you’ve achieved the goals of your effort. This isn’t the time to consider your work complete, however. How can you make the intervention even better and more effective?
Good interventions are dynamic: they keep changing and experimenting, always reaching for something better. Programs can always be improved. The program only produced some of the intended outcomes If the intervention produced only some, or some lower level, of the desired outcomes, you may be headed in the right direction. The program may also have greater effects in the long run, as participants incorporate the changes they’ve made into their everyday lives. Some possible reasons for the program’s effect not being as great as planned:
The program produced no outcomes If the program produced no outcomes at all, you may have to make big changes.
The program may have produced unintended outcomes, either positive or negative. If they’re positive, you might want to understand how they came about so that you can continue to produce them. If they’re negative, you’ll probably want to learn more so you can seek to eliminate them. Most of the reasons for unintended outcomes are similar to those for lack of outcomes.
Given all this, how do you approach your evaluation research to decide what you need to refine and how? A good general approach is to work backward from outcomes – asking “but why?” – regarding why each previous phase failed to produce the results you wanted. Using the "But why?" Method to examine outcomes
If the program had the impact you expected, but no outcomes, perhaps you’ve chosen the wrong behaviors or factors to target, and need to rethink your problem analysis and related intervention. There are other plausible explanations: your intervention wasn’t in place long enough, the effects are delayed, your measures are insensitive to what is being achieved, etc.
If your program did go as planned – you met your deadlines and did what you intended to do in the way you intended to do it – and you failed to achieve your goals, there’s a good chance that your planning was the problem. You may have aimed at insufficient risk and/or protective factors, as mentioned above, or you may have chosen ineffective methods to influence the right ones...or both. There are other possibilities that could be picked up by a process evaluation as well, many of which have already been suggested – treatment of participants, language or other communication issues, lack of cultural competence, etc. Identifying and correcting such problems can help a program reach success.
In SummaryThe purpose of an evaluation and the research that goes into it is not just to tell you whether or not your intervention has been a success. The real value of evaluation research lies in its ability to help you identify and correct problems – as well as to celebrate progress. Evaluation can pinpoint the strengths of your program, and help you to protect and enhance those strengths and make them even stronger. By examining the three elements of an intervention – process, impact, and outcomes – your evaluation can tell you whether you did what you had planned; whether what you did had the influence you expected on the behaviors and factors you intended to influence; and whether the changes in those factors led to the intended outcomes. That knowledge can show you what you might change to improve your program, as well as the overall effectiveness of the intervention. And, the information can be used to celebrate the accomplishments you are making along the way. What are the things that we should consider in doing and making research a success?A research project can only be a success if it is sufficiently embedded in existing literature and knowledge on the subject. This can be achieved by doing a complete and coherent literature review.
What makes a good research study?Good research is replicable, reproducible, and transparent. Replicability, reproducibility, and transparency are some of the most important characteristics of research. The replicability of a research study is important because this allows other researchers to test the study's findings.
What type of research evaluates the best ways to prevent diagnose and treat adverse health issues that affect individuals and families quizlet?Clinical research evaluates the best ways to prevent, diagnose, and treat adverse health issues that adversely affect individuals and families.
Which of the following is the first step in the marketing research process?The first step in the marketing research process is defining the problem or the question your research is trying to answer, followed by developing a research plan to answer that question, collecting and analyzing the data, and then producing a report.
|