Physical Activity Evaluation Handbook: Department of Health and Human Services Centers For Disease Control and Prevention
Physical Activity Evaluation Handbook: Department of Health and Human Services Centers For Disease Control and Prevention
Physical Activity Evaluation Handbook: Department of Health and Human Services Centers For Disease Control and Prevention
DEPARTMENT OF HEALTH AND HUMAN SERVICES Centers for Disease Control and Prevention
DEPARTMENT OF HEALTH AND HUMAN SERVICES Centers for Disease Control and Prevention National Center for Chronic Disease Prevention and Health Promotion
This publication was produced by the National Center for Chronic Disease Prevention and Health Promotion of the Centers for Disease Control and Prevention. Centers for Disease Control and Prevention Jeffrey P. Koplan, MD, MPH, Director National Center for Chronic Disease Prevention and Health Promotion James S. Marks, MD, MPH, Director Division of Nutrition and Physical Activity William H. Dietz, MD, PhD, Director Technical Information and Editorial Services Branch Christine Fralish, MLIS, Chief Amanda Crowell, Writer-Editor Compiled by Sarah Levin, MS, PhD Nancy E. Hood
For more information, contact Division of Nutrition and Physical Activity 4770 Buford Highway N.E. Mailstop K-46 Atlanta, GA 30041-3717 (770) 488-5692
Suggested Citation US Department of Health and Human Services. Physical Activity Evaluation Handbook. Atlanta, GA: US Department of Health and Human Services, Centers for Disease Control and Prevention; 2002.
Information about the Web sites of nonfederal organizations are provided solely as a service to our users. This information does not constitute an endorsement of these organizations by CDC or the federal government, and none should be inferred. The CDC is not responsible for the content of these Web sites. Publication support was provided by Palladian Partners, Inc. under contract no. 200-98-0415.
Preface
The landmark 1996 publication, Physical Activity and Health: A Report of the Surgeon General, identified substantial health benefits of regular physical activity. In January 2000, Healthy People 2010 released a set of 10 priority health indicators that include physical activity as one of the major concerns for public health attention. The Physical Activity and Health Branch of CDCs Division of Nutrition and Physical Activity recently partnered with other national organizations to develop guidelines for increasing physical activity across an array of settings and populations. These include Promoting Better Health for Young People Through Physical Activity and Sports. Available at http://www.cdc.gov/nccdphp/dash/presphysactrp. Promoting Physical Activity: A Guide for Community Action. Available at http://www.cdc.gov/nccdphp/dnpa/pahand.htm. National Blueprint: Increasing Physical Activity Among Adults Age 50 and Older. Available at http://www.cdc.gov/nccdphp/dnpa/pr_blueprint.htm. Increasing Physical Activity: A Report on Recommendations of the Task Force on Community Preventive Services. Available at http://www.cdc.gov/nccdphp/dnpa/physical/ recommendations.htm. We hope the recommendations and strategies described in these and other resources will help users improve existing programs and develop new approaches. As innovative programs emerge and evolve, ongoing program evaluation must be used to Measure the effectiveness of new and enhanced interventions. Determine whether funds and other resources are being used efficiently. Assess the appropriateness and effectiveness of recommended interventions in different settings and populations. Demonstrate accountability and influence policy makers. Evaluate the effects of comprehensive state approaches. This handbook provides tools for state and local agencies and community-based organizations that are evaluating physical activity programs. We hope these tools will help users demonstrate program outcomes and continuously improve physical activity promotion programs. The goal is clear: we need to get moving! Program evaluation will enhance our knowledge of the resources, methods, and strategies necessary to increase physical activity.
William H. Dietz, MD, PhD Director, Division of Nutrition and Physical Activity National Center for Chronic Disease Prevention and Health Promotion, CDC
1
Contents
Introduction ....................................................................................................................................................................5 Six Steps for Evaluating Physical Activity Programs ............................................................................................9 Step 1: Engage Stakeholders ......................................................................................................................................9 Step 2: Describe or Plan the Program ......................................................................................................................13 Step 3: Focus the Evaluation ......................................................................................................................................20 Step 4: Gather Credible Evidence..............................................................................................................................23 Step 5: Justify Conclusions ........................................................................................................................................29 Step 6: Ensure Use and Share Lessons Learned......................................................................................................32
Appendices ....................................................................................................................................................................37 Appendix 1: Program Evaluation Standards and How They Apply To the Six Steps of Program Evaluation ....37 Appendix 2: Guide to Community Preventive Services Recommendations ..............................................................41 Appendix 3: Theories and Models Used in Physical Activity Promotion ..............................................................43 Appendix 4: How to Write SMART Objectives........................................................................................................47 Appendix 5: Indicators and Measurement Resources ............................................................................................49 Appendix 6: Sample Case Studies ............................................................................................................................55
Introduction
Recognition of the importance of physical activity has reached a new height in America. In fact, physical activity was recently named as one of the 10 leading health indicators in Healthy People 2010.1 Consequently, the imperative to evaluate our physical activity programs is greater than ever.
Why?
Physical activity programs must be evaluated to reflect on our progress, see where were going and where weve come from, share what weve learned with our colleagues, put money to nonduplicative use, and improve our programs. After all, we will be held accountable. Program evaluation can be used to Influence policy makers and funders. Build community capacity and engage communities. Share what works and what doesnt work with other communities. Ensure funding and sustainability. Program evaluation can be conducted using these six major steps: Engage stakeholders. Describe or plan the program. Focus the evaluation. Gather credible evidence. Justify conclusions. Ensure use and share lessons learned.
What Is Evaluation?
Evaluation is the systematic examination and assessment of features of an initiative and its effects, in order to produce information that can be used by those who have an interest in its improvement or effectiveness. 2
US Department of Health and Human Services. Healthy People 2010. 2nd ed. With Understanding and Improving Health and Objectives for Improving Health. 2 vols. Washington, DC: US Government Printing Office; 2000. WHO European Working Group on Health Promotion Evaluation. Health Promotion Evaluation: Recommendations to Policymakers. Copenhagen: World Health Organization; 1998.
Program evaluation differs from basic research in that its primary aim is not to add to a body of knowledge but to learn how to improve a program. Other distinctions include the following: Evaluation is controlled by those involved (the stakeholders) instead of being rigorously designed by an investigator. The steps of evaluation vary considerably from those of basic research. Standards of evaluation include usefulness, feasibility, accuracy, and fairness rather than internal and external validity. Evaluation assesses merit, worth, and importance rather than emphasizing associations. Evaluation is holistic and flexible by design to allow for changes and unexpected circumstances rather than being tightly controlled. Evaluation methods are both quantitative and qualitative. Evaluation is ongoing rather than being limited to a specific timeframe. The scope is broad, in an attempt to be integrative, rather than narrowly focused. Judgments from evaluation depend on agreed-upon or specifically stated values of a stakeholder rather than being value-free. Use of the data is imperative not just to further knowledge and help improve similar programs through publication, but also to build capacity or improve a program.
How?
In 1999, CDC published the Framework for Program Evaluation in Public Health (available on-line at http://www.cdc.gov/epo/mmwr/preview/mmwrhtml/rr4811a1.htm).3 The publication outlines six steps for program evaluationengage stakeholders, describe the program, focus the evaluation design, gather credible evidence, justify conclusions, and ensure use and share lessons learned. This handbook uses the Framework for Program Evaluation in Public Health, its companion, An Evaluation Framework for Community Health Programs,4 and Promoting Physical Activity: A Guide for Community Action5 as guiding documents to outline these six steps as they relate to physical activity program evaluation.
Centers for Disease Control and Prevention. Framework for Program Evaluation in Public Health. MMWR 1999;48(No RR-11). The Center for the Advancement of Community Based Public Health. An Evaluation Framework for Community Health Programs. Durham, NC: The Center for the Advancement of Community Based Public Health; 2000. US Department of Health and Human Services. Promoting Physical Activity: A Guide for Community Action. Atlanta, GA: US Department of Health and Human Services, Centers for Disease Control and Prevention; 1999.
Introduction
Features unique to this handbook include We challenge you to think outside the box when you consider your own evaluation plans. We provide KidsWalk-to-School examples to illustrate the main points. CDCs KidsWalk-to-School is a community-based program that aims to increase opportunities for daily physical activity by encouraging children to walk to and from school in groups accompanied by adults. We provide a worksheet that you can photocopy and use to help you apply each step to your physical activity programs. We include appendices to provide more detail on certain aspects of program evaluation in relation to physical activity programming, including evaluation indicators and case studies (see Appendices 16). For additional evaluation links and resources, visit the American Evaluation Associations Web site at http://www.eval.org/EvaluationLinks/links.htm.
Standards
Thirty standards provide the guiding principles for your evaluation (see Appendix 1).6 The standards are based on four key questions that you should ask yourself throughout the six steps of program evaluation. Is the evaluation Useful? Feasible? Accurate? Fair? Will the amount and type of information you collect meet the needs of those who intend to use the evaluation findings? Will the evaluation be practical, doable, and realistic? Will the evaluation findings be correct? Will the evaluation be conducted with awareness of the rights of the people involved in the program?
All standards cannot be achieved equally in every situation. However, some standards must always be preserved. Although an accurate measurement of physical activity might not be feasible because of its cost or complexity, you can never skimp on fairness. Likewise, an evaluation is not worth doing if the results will not be used.
The Joint Committee on Standards for Educational Evaluation. The Program Evaluation Standards: How to Assess Evaluations of Educational Programs. 2nd ed. Thousand Oaks, CA: Sage Publications; 1994. 7
Use this list to help you identify a master list of stakeholders.Your evaluation stakeholders will be a subset of all program stakeholders. Community sector Target audience members. Community residents. Youth. Government sector National, state, and local elected officials. Regional or local planning commissions. State or county departments of education. State or county departments of parks and recreation. State departments of tourism. Law enforcement agencies. Public housing communities. Health sector Wellness councils or physical activity coalitions. Physicians in private practice. Physical and occupational therapists. Insurance companies. National and state nursing and medical associations. National and state health education associations. Education sector Universities and colleges. Technical schools. State and local chapters of professional teachers and administrators associations. Students. Transportation and environmental development sector U.S. Environmental Protection Agency. National and state highway traffic and safety officials. Professional associations and environmental advocacy groups.
10
Business sector Chamber of Commerce. Professional sports teams. Large and small businesses and industries. Media and communication sector Television stations. Radio station managers. Professional journal editors. Health and fitness publication editors. Recreation sector National, state, and local parks. Walking, hiking, or running clubs. State games associations (e.g., Senior Games and Corporate Games). Sports governing bodies and state athletic associations. Religious sector Clergy and ministerial associations or councils. Youth groups. Church-owned recreation facilities, camps, etc. Voluntary or service organizations sector National associations and foundations. Parent-teacher associations. Graduate students in applicable programs. Special public or private foundations. Economic development agencies.
KidsWalkto-School Stakeholders
Implementers Parents. School teachers and staff. Parent/teacher organizations. Local health department.
Partners 4-H clubs. Boys and Girls clubs. CDC. Parents. Kids. Neighbors.
Participants
11
2. Describe how you will assess your stakeholders interests, needs, resources, and contributions throughout the planning process.
3. Identify the people who will use the results of the evaluation and be involved in most evaluation decisions (i.e., evaluation stakeholders).
12
Stage of Development
The three general program stages are planning, implementation, and maintenance.Your programs stage of development will affect the entire evaluation planning process, starting with the program description. If your program is in the planning stage, you might want to conduct a needs assessment (sometimes called a formative evaluation) to determine the extent of the problem that you want to address or the need that your program might meet. For a program that is already being implemented or maintained, your evaluation planning process will focus more on measuring the implementation of program activities and identifying the expected outcomes for program participants and the contextual factors that affect the process or outcomes of the program. All steps in planning your evaluation will be tailored to your programs stage of development.
Statement of Problem
These questions help define the problem and the corresponding need for the program. Each question includes a hypothetical answer. What is the nature of the problem? Physical activity is one of 10 leading health indicators for the nation (Healthy People 2010). What is the magnitude of the problem (including subpopulations)? According to the state Youth Risk Behavior Surveillance System (YRBSS), only 45% of children in grades 912 perform the recommended level of physical activity per week. What are the consequences of the health problem? Physical inactivity leads to many chronic diseases or conditions, such as obesity, cardiovascular disease, and osteoporosis.
13
What causes the problem? According to local school district data, only 40% of students are enrolled in physical education (PE) each semester. Think Outside the Box
Did you notice the multiple sources of data used to describe the problem? Can you think of other sources?
What changes or trends are occurring? According to the school principal, PE enrollment has dropped and fewer children walk or ride the bus to school each year because more parents are dropping them off at school.
A Logic Model
Describes the core components of the program. Illustrates the connection between program components and expected outcomes. Includes pertinent information regarding program context (i.e., influential factors).
Clarify program strategy. Justify why the program will work. Assess the potential effectiveness of an approach. Identify appropriate outcome targets (and avoid overpromising). Set priorities for allocating resources. Incorporate findings from research and demonstration projects. Make midcourse adjustments and improvements in your program. Identify differences between the ideal program and its real operation. Specify the nature of questions being asked in the evaluation. Organize evidence about the program. Make stakeholders accountable for program processes and outcomes. Build a better program.
14
ACTIVITIES Events or actions (e.g., workshops, curriculum development, training, social marketing, special events, advocacy)
OUTPUTS Direct products of program (e.g., number of people reached or sessions held)
INITIAL OUTCOMES Short-term effects of program (e.g., knowledge, attitude, skill, and awareness changes)
This approach, also called reverse logic, starts with desired outcomes and requires you to work backwards to develop activities and inputs. Usually used in the planning stage, this approach ensures that program activities will logically lead to the specified outcomes if your arrow bridges are well-founded.You will ask the question,How?as you move to the left in your logic model. This approach is also helpful for a program in the implementation stage that still has some flexibility in its program activities.
15
Example
What is the desired long-term outcome? Youth will incorporate the recommended daily amount of physical activity into their lifestyle. How? What is the desired intermediate outcome?
Youth will gain increased skills and additional physical activity in school. How? Physical education curricula will be modified. How?
Left-to-Right Logic Model This approach, also called forward logic, may be used to evaluate a program in the implementation or maintenance stage that does not already have a logic model. Start by articulating the program inputs and activities. To move to the right in your model, you must ask the question,Why? You can also think of this approach as an If , then progression. Example
What are the existing inputs? What are the existing activities? What are the desired short-term outcomes? Staff, incentives, materials. Why?
Employees attitudes will improve and their knowledge about the recommended daily level of physical activity will increase. Why?
What is the desired intermediate outcome? What is the desired long-term outcome?
16
Logic models can take many shapes and sizes. At the end of this chapter you will find a generic logic model that includes a variety of physical activity program activities. You could use this model as a starting point, but feel free to change the design and put your own twist on your model. There is no one correct way to create or display a logic model.
Physical education teachers will be taught how to modify their curricula to incorporate more lifelong physical activities in a coordinated way with other courses.
* In addition to the logic model, you might also need to create SMART (specific, measurable, achievable, relevant, time-bound) objectives for both process and outcome measures (e.g., In the first semester, weekly walks from five different locations will be held.). See Appendix 4.
17
No. of employees participating No. of work site environmental and policy changes to support physical activity
Increase in choices available for recreation or transportation (e.g., new paths, classes, or flex time)
Major holidays, competing interests of target populations, history of poor coalition efforts, lack of school board support for physical activity, support of physician counseling by the American Medical Association INFLUENTIAL FACTORS
18
2. Plan or describe the program. We know our end goal, so we will work right-to-left and ask, How?
OR
We know what we have to put into the program, so we will work left-to-right and ask, Why?
19
Sample Uses
Assess the level of community interest in a physical activity program, and use that information to plan a physical activity program. Identify barriers to and facilitators of physical activity in schools, and use that information to advocate for school health policies. Monitor the implementation of a youth program, and use the results to enhance the physical activity component of the program. Survey the target audience that your physical activity message is reaching, and use that information to improve the content and delivery of a physical activity media message.
Improve a program
Measure the extent to which your performance indicators are met, and use these results to apply for additional funding. Use information about which employees benefited most from a work site wellness program to target future efforts more effectively.
Evaluation Questions
Think Outside the Box
Though many of your evaluation questions will be answered by measuring the activities or outcomes from your logic model, encourage stakeholders not to limit their questions. Evaluation questions should ask more than whether outcomes were obtained.
To focus the evaluation, stakeholders indicate what questions they believe the evaluation should answer. Encourage stakeholders to generate a long list of questions, which will then be prioritized based on the stage of your programs development, available resources and the intended uses of the results. The final list should include some questions that are acceptable to all stakeholders. Ask your stakeholders what they want or need to know about Program activities. Initial, intermediate, and long-term program outcomes. Program participants. Larger effects of the program on organizations or communities. External factors that influence the program.
20
KidsWalkto-School
Parents wanted to know if their children could safely walk to school. The principal wanted to know if walking to school had an effect on students school performance in academics and discipline. Although the parents question could be answered with process measures, the principals questions would take outcome measures (pretest or posttest). Parents could use the evaluation to give them peace of mind and determine how and whether their children would walk to school. The principal could use the evaluation to justify putting resources into the KidsWalk-to-School program.
Outcome Evaluation Outcome evaluation (sometimes called impact or summative evaluation) measures the effects of the program on the short-term, intermediate, or long-term outcomes in your logic model. Outcome evaluations should be conducted only when a program is mature enough to potentially produce the desired outcomes. Usually, programs in the maintenance stage are the only ones that can realistically expect outcomes. However, you may be able to ask questions about short-term outcomes for a program in the implementation stage. Sample Questions
What did we accomplish? Did we achieve our outcomes? Why or why not? What is different as a result of our actions? What can we learn from the participants who dropped out of the program? How expensive was the program compared with other physical activity interventions? Is the program as effective as or more effective than similar programs? What went right? What went wrong? What could we do differently next time to achieve better outcomes? Were there any unintended effects of the program? Were there external influences that could have enhanced or hindered the achievement of expected outcomes?
21
2. List all potential uses for the evaluation results (be as specific as possible).
3. Identify whether a process or outcome evaluation (or a combination) is most appropriate for your programs stage of development. Then, list all potential evaluation questions. Many of your evaluation questions will come directly from the program logic model.
4. Go back to questions 2 and 3 and put a star beside the uses and evaluation questions that you think are most important and acceptable to stakeholders.
22
23
Administrative records. Program attendance lists. Asset and needs assessments. Local, state, or national government reports. Observations Direct observations of physical activity behavior. Direct observations of environment and/or physical activity facilities. Indirect observations via video camera or infrared light counter. Existing Data State and national Behavioral Risk Factor Surveillance System (BRFSS). State and national YRBSS. National Health and Nutrition Examination Survey (NHANES III). National Health Interview Survey (NHIS). School fitness testing. Crime reports. University-based surveys. Phone book.
24
Number of days walked or biked to school in past week Childrens attitudes towards walking to school (threequestion scale for parents and children) Childrens scores on traffic safety test Community members knowledge of physical activity recommendations Community members intentions to exercise Community members exercise in past 7 days Community cohesion scale
Community household survey (before and after the program or after the program only) Key informant interviews
Description of original barriers to walking Description of barriers to walking after the program Quantity and quality of advocacy efforts
25
Design
After identifying and prioritizing the evaluation questions, indicators, data sources, and performance indicators, you must decide on an evaluation design. A randomized trial is the most rigorous design, but is probably not feasible or appropriate for a communitybased physical activity program. Less rigorous designs have strengths and weaknesses and should be combined to maximize the effectiveness of the evaluation design; they also are commonly used to evaluate physical activity programs. Choose your evaluation design with your available expertise, resources, and timeline in mind.
Logistics
The methods, timing, and infrastructure for collecting and handling evidence must consider Steps 13. The logistics of data collection should particularly consider the cultural context of the program and protect the privacy of the data sources and confidentiality of the information. For example, the sex and race or ethnicity of a person taking measurements for a body mass index (BMI) might need to be matched to the sex and race or ethnicity of the participant. Survey respondents must be told that their individual responses will never be identified by their names.
26
Agreements
Agreements specify roles and responsibilities so that the evaluation is effectively and efficiently conducted. Elements of the agreement include purpose, users, uses, questions and methods, end products, time line, and budget. Ethical considerations throughout the evaluation process should be discussed in the agreement process (see Appendix 1). The formality of the agreements will depend on the needs and characteristics of the stakeholders, but written documents are recommended even for less formal agreements.
27
Evaluation Questions
Indicators
Data Sources
Performance Indicators
28
Analyze Data
Enter the data into a computer (e.g., using EpiInfo, a free database available on-line at http://www.cdc.gov/epiinfo). Check for data entry errors. Tabulate the data (e.g., calculate the number of participants, percentage of participants meeting physical activity recommendations, percentage of participants who walked to school every day). Stratify data (e.g., by community, age, race or ethnicity, income level, fitness level). Make comparisons (e.g., differences between pretests and posttests or between a comparison and intervention community). Present data in a clear and uncomplicated format. Think Outside the Box
Analyzing data requires expertise in data management and statistical testing. If you do not have this expertise among your staff or stakeholders, be creative in forming partnerships. Many university graduate students are looking for evaluation projects and might provide the expertise you need free of charge. If you have a larger budget, an evaluation consultant can bring years of experience to your analysis. Evaluation staff in local, state, or federal health departments or nongovernmental organizations could be helpful.
Interpret Results
What do the numbers, frequencies, averages, and statistical test results actually say about your program? Are your results similar to what you expected? If not, why do you think they are different? Are there alternative explanations for your results? How do your results compare with those of similar programs? What are the limitations of your evaluations (e.g., potential biases, generalizability of results, reliability, validity)? How well does your evaluation reflect the program as a whole? If you used multiple indicators to answer the same evaluation question, did you get similar results? Will others interpret the findings in an appropriate manner?
29
Judgments
By comparing the interpretation of your results to agreed-upon standards, you can make judgments about the program based on the purpose(s) and intended uses of the evaluation. Although not explicitly stated, the standards for making judgments have been discussed throughout the evaluation process as the stakeholders have taken the following steps: Set performance indicators. Performance indicators are standards in and of themselves. Decisions about what measures should be taken and how much they should change over time will be used to judge the process and outcome results of the evaluation. Developed a logic model. For some stakeholders, the fidelity of program implementation, as outlined by the logic model, is critical. If stakeholders insisted on a detailed logic model, this could indicate that the implementation process is significant to them. They might judge a program more harshly if the process evaluation indicated problems with implementation. Prioritized evaluation questions. In prioritizing the evaluation questions, stakeholders make their values known. If stakeholders prioritized feasibility, for example, a program might show positive outcomes but be judged according to how practical the continuation of the program is. Made decisions regarding their involvement. Some stakeholders, perhaps a funder or other resource provider, might want to judge the results of the program evaluation solely by whether resources were used efficiently. If the evaluation results showed an increase in participants levels of physical activity, but the program was not costeffective, these stakeholders would judge it differently than a stakeholder involved primarily to promote behavioral change. Although forming these judgments might not be easy, the consensus-building process will help stakeholders understand the basis for the recommendations in Step 6, thereby helping ensure the future use of evaluation results.
KidsWalkto-School
What would happen to your KidsWalk-toSchool program if a child got hurt while walking or bicycling to school? Would this one incident be the basis for the judgments made about the program? We hope not. We hope the process of working through the previous four steps with the stakeholders (and remembering the standards of usefulness, feasibility, accuracy, and fairness) leads to balanced judgments.
30
2. Are your results similar to what you expected? If not, why do you think they are different?
What are the limitations of your evaluations (e.g., potential biases, generalizability of results, reliability, validity)? How well does your evaluation reflect the program as a whole?
If you used multiple indicators to answer the same evaluation question, did you get similar results?
3. Against what standards will you compare your interpretations in forming your judgments?
31
Recommendations
Recommendations for continuing, expanding, redesigning, or abandoning a physical activity program might follow straight from the judgments; however, you should also consider competing priorities and alternatives. Recommendations Should Be
Action oriented. Relevant. Useful.
Tips Consider your stakeholders values and align recommendations when possible. Share draft recommendations with stakeholders and solicit feedback. Relate your recommendations to the original purposes and uses of the evaluation. Target your recommendations appropriately for each audience. Potential audiences for your recommendations Schools. Workplace owners. Parents. National agencies and organizations. Health insurance agencies. Advocacy groups. Traffic safety planners and enforcers. State legislators. City councils. Community-based organizations and programs. State health department officials. Police departments. Nonprofit health and service organizations.
32
Communication
At this point, you have decided what to recommend and who needs to hear the recommendations, but how will you effectively share this information? Your strategy should consider both format and channels. Format Reports summarizing your evaluation results should be easy to understand and appropriate for the intended audience. Depending on your audiences, you may have to prepare more than one report. Some tips include Summarize the evaluation plan and procedures. List the strengths and weaknesses of the evaluation. List the pros and cons of each recommendation. Present clear and succinct results in tables and graphs. Summarize the stakeholders roles and involvement in both the project and the follow-up plans. Channels Decide how you will get your information to the intended audiences.You may use Mailings. Web sites. Community forums. Media (television, radio, newspaper). Personal contacts. Listservs. Organizational newsletters. Think Outside the Box
Evaluation results can be communicated in ways other than traditional written reports, including Oral presentations. Diagrams and charts. Illustrations. Success stories. Newspaper articles. Radio reports. Local news stories. Fact sheets.
Follow Up
Because of the effort required, reaching justified conclusions and making sound recommendations can seem like an end in itself. However, active follow up is needed to Remind stakeholders and the audience of the intended uses of the evaluation results. Prevent lessons learned from being lost or ignored when complex program or policy decisions are made. Prevent misuse of results by ensuring that evidence is applied to the questions that were the evaluations central focus and that the results are not taken out of context.
33
34
Format.
Channel.
3. Who will ensure follow up with users of the evaluation findings, and how will that be accomplished?
Who.
How.
35
Appendix 1
Program Evaluation Standards and How They Apply To the Six Steps of Program Evaluation*
Program Evaluation Standards
Utility Standards Utility standards are intended to ensure that an evaluation will serve the information needs of intended users. Stakeholder identification: Persons involved in or affected by the evaluation should be identified so that their needs can be addressed. Evaluator credibility: The persons conducting the evaluation should be both trustworthy and competent to perform the evaluation, so that the evaluation findings achieve maximum credibility and acceptance. Information scope selection: Information collected should be broadly selected to address pertinent questions about the program and be responsive to the needs and interests of clients and other specified stakeholders. Values identification: The perspectives, procedures, and rationale used to interpret the findings should be carefully described so that the bases for value judgments are clear. Report clarity: Evaluation reports should clearly describe the program being evaluated, including its context and the purposes, procedures, and findings of the evaluation, so that essential information is provided and easily understood. Report timeliness and dissemination: Significant interim findings and evaluation reports should be disseminated to intended users so that the information can be used in a timely fashion. Evaluation impact: Evaluations should be planned, conducted, and reported in ways that encourage follow-through by stakeholders, so that the likelihood that the evaluation will be used is increased. Feasibility Standards Feasibility standards are intended to ensure that an evaluation will be realistic, prudent, diplomatic,and frugal. Practical procedures: The evaluation procedures should be practical to keep disruption to a minimum while needed information is obtained. Political viability: The evaluation should be planned and conducted with anticipation of the different positions of various interest groups, so that their cooperation may be obtained and possible attempts by any of these groups to curtail evaluation operations or to bias or misapply the results can be averted. Cost-effectiveness: The evaluation should be efficient and produce information of sufficient value that the resources expended can be justified. Step 3: Focus the evaluation. Step 3: Focus the evaluation. Step 1: Engage stakeholders. Step 1: Engage stakeholders.
Step 6: Ensure use and share lessons learned. Step 6: Ensure use and share lessons. Step 6: Ensure use and share lessons learned.
* The Joint Committee on Standards for Educational Evaluation. The Program Evaluation Standards: How to Assess Evaluations of Educational Programs. 2nd ed. Thousand Oaks, CA: Sage Publications; 1994. 37
38
Appendix 1
Step 2: Describe or plan the program. Step 2: Describe or plan the program. Step 3: Focus the evaluation.
Step 5: Justify conclusions. Step 6: Ensure use and share lessons learned. Steps 16: Continually evaluate the strengths and weaknesses of your evaluation.
39
Appendix 2
Guide to Community Preventive Services* Recommendations
In 2001, the Task Force on Community Preventive Services published recommendations on evidence-based interventions to promote physical activity. Based on systematic reviews of the literature, these recommendations provide guidance to organizations and agencies that are planning or conducting programs to increase physical activity. However, the recommendations are based on a limited number of well-controlled interventions in specific settings with selected populations. Therefore, the implementation and effectiveness of a program in your specific environment should still be evaluated. Some interventions reviewed by the Community Guide revealed insufficient evidence to support a recommendation, but only recommended or strongly recommended interventions are presented here.
Intervention Intervention Description Task Force Recommendation for Use
Strongly recommended
Informational Approaches Community-wide campaigns Large-scale, high-intensity, communitywide campaigns with sustained high visibility. Messages regarding physical activity behavior are promoted through television, radio, newspaper columns and inserts, and trailers in movie theaters. Motivational signs placed close to elevators and escalators encouraging use of nearby stairs for health benefits of weight loss. Percentage of persons active. Estimated energy expenditure. Time spent in physical activity. Scaled activity scores.
Point-of-decision prompts
Recommended
Percentage of persons taking stairs instead of elevators or escalators (settings included train, subway, and bus stations; shopping malls; and university libraries).
Behavioral and Social Approaches Individually adapted health behavior change programs Programs tailored to the persons readiness for change or specific interests. Designed to help participants incorporate physical activity into their daily routines by teaching them behavioral skills, including goal-setting and self-monitoring, building social support, behavioral reinforcement (self-reward and positive self-talk), structured problem-solving, and relapse prevention. May be delivered in group settings or by mail, telephone, or directed media. Strongly recommended Minutes spent in physical activity. Energy expenditure.
* Centers for Disease Control and Prevention. Increasing Physical Activity: A Report On Recommendations of the Task Force on Community Preventive Services. MMWR 2001;50(No. RR-18):116. Also see the Guide to Community Preventive Services Web site at http://www.thecommunityguide.org.
41
Intervention
Intervention Description
Behavioral and Social Approaches (continued) School-based physical education (PE) Modified curricula and policies to increase the amount of moderate or vigorous activity, the amount of time spent in PE class, or the amount of time students are active during PE class. Interventions included changing the activities taught or modifying the rules of the game so that students are more active. Focus is on changing physical activity behavior through building, strengthening, and maintaining social networks that provide supportive relationships for behavior change. Strategies include creating new social networks or working within preexisting networks in a social setting (e.g., the workplace), setting up a buddy system, contracting with another person to complete specified levels of physical activity, or establishing walking groups or other groups to provide friendship and support. Minutes per week spent in moderate to vigorous physical activity (MVPA). Percentage of class time spent in MVPA. Estimated energy expenditure.
Social support interventions in community settings (does not include family settings)
Strongly recommended
Environmental and Policy Approaches Creation of or enhanced access to places for physical activity combined with informational outreach activities Access to places for physical activity can be created or enhanced by building trails or facilities or by reducing barriers to such places. Certain programs also provide training in using equipment and incentives (e.g., risk factor screening and counseling or other health education activities). Work site programs were also included in this category. Strongly recommended Percentage of persons exercising on X days per week. Self-reported exercise scores. Energy expenditure.
42
Appendix 3
Theories and Models Used in Physical Activity Promotion
As you are planning or describing your program, referring to individual, interpersonal, or community-level theories that relate to health behavior change is sometimes useful. For example, these theories could support the arrow bridges in your logic model or help you identify potential points of intervention. Because the theories and models presented here are supported by varying levels of research, use them as one piece of your planning puzzle.
Theory/Model
Individual Level Health belief model
Summary
Key Concepts
For people to adopt recommended physical activity behaviors, their perceived threat of disease (and its severity) and benefits of action must outweigh their perceived barriers to action.
Perceived susceptibility Perceived severity Perceived benefits of action Perceived barriers to action Cues to action Self-efficacy
In adopting healthy behaviors (e.g., regular physical activity) or eliminating unhealthy ones (e.g., watching television), people progress through five levels related to their readiness to changeprecontemplation, contemplation, preparation, action, and maintenance. At each stage, different intervention strategies will help people progress to the next stage.
Precontemplation Contemplation Preparation Action Maintenance Skills training Cognitive reframing Lifestyle rebalancing Exposure Attention Liking/interest Comprehension Skill acquisition Yielding Memory storage Information search and retrieval Decision Behavior Reinforcement Postbehavior consolidation
Relapse prevention
Persons who are beginning regular physical activity programs might be aided by interventions that help them anticipate barriers or factors that can contribute to relapse.
Informationprocessing paradigm
The impact of persuasive communication, which can be part of a social marketing campaign to increase physical activity, is mediated by three phases of message processingattention to the message, comprehension of the content, and acceptance of the content.
43
Theory/Model
Interpersonal Level Social learning/ social cognitive theory
Summary
Key Concepts
Health behavioral change is the result of reciprocal relationships among the environment, personal factors, and attributes of the behavior itself. Self-efficacy is one of the most important characteristics that determine behavioral change.
For behaviors that are within a persons control, behavioral intentions predict actual behavior. Intentions are determined by two factorsattitude toward the behavior and beliefs regarding others peoples support of the behavior.
Attitude toward the behavior Outcome expectations Value of outcome expectations Subjective norms Beliefs of others Desire to comply with others
Peoples perceived control over the opportunities, resources, and skills needed to perform a behavior affect behavioral intentions, as do the two factors in the theory of reasoned action.
Attitude toward the behavior Outcome expectations Value of outcome expectations Subjective norms Beliefs of others Desire to comply with others Perceived behavioral control
Social support
Often incorporated into interventions to promote physical activity, social support can be instrumental, informational, emotional, or appraising (providing feedback and reinforcement of new behavior).
44
Appendix 3
Theory/Model
Community Level Community organization model
Summary
Public health workers help communities identify health and social problems, and they plan and implement strategies to address these problems. Active community participation is essential. Effective interventions must influence multiple levels because health is shaped by many environmental subsystems, including family, community, workplace, beliefs and traditions, economics, and the physical and social environments.
Key Concepts
Social planning Locality development Social action Multiple levels of influence Intrapersonal Interpersonal Institutional Community Public policy
Ecological approaches
Certain processes and strategies might increase the chances that healthy policies and programs will be adopted and maintained in formal organizations.
Definition of problem (awareness stage) Initiation of action (adoption stage) Implementation of change Institutionalization of change
People, organizations, or societies adopt new ideas, products, or behaviors at different rates, and the rate of adoption is affected by some predictable factors.
Sources 1. Alcalay R, Bell RA. Promoting Nutrition and Physical Activity Through Social Marketing: Current Practices and Recommendations. Davis, CA: Center for Advanced Studies in Nutrition and Social Marketing, University of California, Davis; 2000. 2. National Institutes of Health. Theory at a Glance: A Guide for Health Promotion Practice. Bethesda, MD: National Institutes of Health, National Cancer Institute; 1995. 3. US Department of Health and Human Services. Physical Activity and Health: A Report of the Surgeon General. Atlanta, GA: US Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion.
45
Appendix 4
How to Write SMART Objectives
For many grants and reports, you might have to write goals and objectives. This handbook mentions goals briefly, using the words outcomes and indicators, but does not use the term objectives. However, throughout the process of evaluation planning, all of the decisions necessary for writing program goals and objectives have been made.
Program Goal
In Step 2, you designed a logic model for your program that probably included a goal or mission statement. If not, review your logic model and the description of the problem that the program is trying to address. Compose a phrase or short sentence that captures the overarching, ideal purpose of your program. This is your goal.
Program Objectives
To formulate strong program objectives, use information from your logic model to write SMART (specific, measurable, achievable, relevant, and time-bound) objectives.You can write either process or outcome objectives by using the information in your logic model. Process objectives include content from the activities column of your logic model. Outcome objectives include content from the outcomes columns of your logic model. Other components of the evaluation planning process that will help you write SMART objectives include evaluation questions, data sources, and performance indicators.You may also borrow the Healthy People 2010* objectives or link your local objectives with these national objectives. Healthy People 2010 Objectives for Physical Activity As national priorities for physical activity promotion, these objectives may be used as the long-term objectives for your program. Physical activity is a leading health indicator for the United States. To monitor progress for Healthy People 2010, the physical activity indicator is being measured by the following two objectives: 22.7 Increase the proportion of adolescents who engage in vigorous physical activity that promotes cardiorespiratory fitness 3 days per week for 20 minutes per occasion. Increase the proportion of adults who engage regularly, preferably daily, in moderate physical activity for at least 30 minutes per day.
KidsWalkto-School Examples
Process Objective In the first semester of KidsWalk-to-School, 20 community volunteers will commit to participating in organized walks to school. Outcome Objective By the end of this school semester, the number of students walking to school will increase by 20%.
22.2
* US Department of Health and Human Services. Healthy People 2010. 2nd ed. With Understanding and Improving Health and Objectives for Improving Health. 2 vols. Washington, DC: US Government Printing Office; 2000.
47
Additional physical activity objectives include Physical Activity in Adults 22.1 22.3 Reduce the proportion of adults who engage in no leisure-time physical activity. Increase the proportion of adults who engage in vigorous physical activity that promotes the development and maintenance of cardiorespiratory fitness 3 days per week for 20 minutes per occasion. Increase the proportion of adults who perform physical activities that enhance and maintain muscular strength and endurance. Increase the proportion of adults who perform physical activities that enhance and maintain flexibility. Increase the proportion of adolescents who engage in moderate physical activity for at least 30 minutes on 5 of the previous 7 days. Increase the proportion of the nations public and private schools that require daily physical education for all students. Increase the proportion of adolescents who participate in daily school physical education. Increase the proportion of adolescents who spend at least 50% of school physical education class time being physically active.
22.10 Increase the proportion of adolescents who view television 2 hours on a school day. Access 22.11 (Developmental) Increase the proportion of the nations public and private schools that provide access to their physical activity spaces and facilities for all persons outside of normal school hours (i.e., before and after the school day, on weekends, and during summer and other vacations). 22.12 Increase the proportion of work sites offering employer-sponsored physical activity and fitness programs. 22.13 Increase the proportion of trips made by walking. 22.14 Increase the proportion of trips made by bicycling.
48
Appendix 5
Indicators and Measurement Resources
Common Individual-Level Indicators for Physical Activity
These indicators can be used to measure individual-level outcomes of your physical activity program. This list is not comprehensive. Make sure you choose indicators that are realistic for your program and that can be measured using available resources.
Measure
Direct Measures Metabolic equivalent (MET)* intensity levels (MET-minutes per day or week) Light: <3 METs Moderate: 36 METs Vigorous: >6 METs Questionnaire The Compendium of Physical Activities* lists 605 specific activities that are each assigned an intensity level based on the rate of energy expenditure (EE), expressed as METs. One MET is considered a resting metabolic rate while sitting quietly. By expressing self-reported minutes of activities in MET-minutes, you create a standardized physical activity measure that you can compare with other MET-minutes of activity. Calculate from a past weeks recall of physical activity as follows: MET-mins/day = (frequency x time x intensity) / 7 days Minutes of physical activity per day or week Questionnaire For minutes or MET-minutes, it may be helpful to separate the following types of physical activity for respondents: jobrelated; transportation; housework, house maintenance, and caring for family; and recreation, sport, and leisure-time. Note that raw minutes of physical activity do not include the intensity of the activity. Calculate from a past weeks recall of physical activity as follows: Minutes/day = (frequency x time) / 7 days Steps walked per day or week Energy expenditure (EE) per day or week Pedometer Accelerometer Simple, relatively inexpensive tool to assess mobility. Accelerometer measures two or three dimensions of movement. Software can calculate EE based on the persons age, sex, height, and weight.
Source of Data
Comments
49
Measure
Indirect Measures Waist circumference Waist-hip ratio
Source of Data
Comments
Tape measure Tape measure Equals the circumference of the waist divided by the circumference of the hips. BMI = weight (kg) / height (m)2
The American College of Sports Medicine has established and published valid protocols for all of these tests to measure aerobic fitness. VO2 max can be estimated from heart rate or measured directly. Time to complete one measured mile is an indirect measure of fitness. FitnessGram provides a complete protocol for youth fitness testing.
Aerobic fitness (field measure) Youth fitness scores Intervening Measures Knowledge
Questionnaire
Do respondents know the recommended levels/frequency of physical activity? Do they know the different recommendations for moderate versus vigorous activity? How do respondents feel about being physically active? What do they think will happen if they increase their levels of physical activity? How confident are they about their ability to do physical activity? Respondents might be at different stages in changing their behavior. Different interventions are more appropriate for different stages of change, and progress can be measured by assessing progression through the stages.
Attitudes
Questionnaire
Questionnaire
* Ainsworth BE, Haskell WL, Whitt MC, et al. Compendium of Physical Activities: An Update of Activity Codes and MET Intensities. Med Sci Sports Exerc 2000;32(suppl 9):S498S516. International Physical Activity Questionnaire. Available on-line at http://www.ipaq.ki.se. FitnessGram. Available on-line at http://www.cooperinst.org/ftgmain.asp.
50
Appendix 5
Information
Percentage of health-care providers that routinely advise patients to exercise more. Availability of materials in work sites linking physical activity to cardiovascular disease. Percentage of schools offering curricula in grades K12. Number of media reports dealing with physical activity. Point-of-purchase education materials.
Environmental
Miles of walking trails per capita in schools. Number of physical activity facilities per capita in schools. Availability of facilities to community members (e.g., how many, hours of operation). Number of programs for physical activity offered in community. Number of agencies in community that sponsor physical activity events or programs. Level of enforcement of pedestrian/driver responsibilities (e.g., jaywalking, yielding to pedestrians). Zoning/development regulations that require or promote smart growth. Score on pedestrian walkability scales.
Behavioral outcome Observations of usage (e.g., in malls, trails). measures Membership in physical activity organizations (e.g., YMCAs, YWCAs, health clubs).
Sales of selected physical activity items (e.g., sports equipment, videos). * Cheadle A, Sterling TD, Schmid TL, Fawcett SB. Promising Community-Level Indicators for Evaluating Cardiovascular Health-Promotion Programs. Health Educ Res 2000;15:109116. 51
Measurement Resources
These resources are intended to help you develop data collection instruments to measure your selected indicators. Because some of these instruments have been tested for reliability and validity, you can improve the quality of your data collection by using them. Also, using items from an existing survey allows comparison of your responses with others. However, be careful to select items that actually measure the indicators your program is designed to affect. No one tool from this list is likely to be the most appropriate data collection instrument for your evaluation.You might need to combine items from several surveys or combine an environmental checklist with a questionnaire designed to assess behavior change. Also, some tools might be more appropriate for program planning than evaluation data collection. Review the examples critically as you develop your own data collection instruments and plans. National Health and Nutrition Examination Survey (NHANES), CDC. Physical Activity and Physical Fitness Questionnaire. Questions address activities related to transportation, daily activities, and leisure-time activities. Available on-line at http://www.cdc.gov/nchs/about/major/nhanes/questexam.htm. Behavioral Risk Factor Surveillance System, CDC. Physical activity questions. Using this national survey allows you to compare your results with the same questions at the state and national level. Available on-line at http://www.cdc.gov/nccdphp/brfss/brfsques.htm. A Collection of Physical Activity Questionnaires for Health Related Research. Seventeen of these complete questionnaires are used to survey the general population, four are used for older adults, and seven are used as part of major population-based surveys. Med Sci Sports Exerc 1997;29(suppl 6). International Physical Activity Questionnaire. Four internationally comparable questionnaires that measure adult levels of physical activity. Available on-line at http://www.ipaq.ki.se/. CDC KidsWalk-to-School Guide (Walk-to-School Survey and Walkable Routes to School Survey). Can be used to measure behavior and environmental changes for any program that promotes kids walking to school. Available on-line at http://www.cdc.gov/nccdphp/dnpa/kidswalk/kidswalk_guide.htm. HeartCheck (New York Department of Health). Used to assess work site facilities, practices, and policies that support a heart-healthy lifestyle. PDF file available. Contact Lori King at (518) 473-0673 or by E-mail at [email protected].
52
Appendix 5
Local Index of Transit Availability (LITA) Manual, Local Government Commission. Outlines a system for rating transit availability in various parts of metropolitan areas. Available on-line at http://www.lgc.org/freepub/land_use/lita/lita_manual.html. Walkability Checklist, Partnership for a Walkable America, Pedestrian and Bicycle Information Center and U.S. Department of Transportation. Simple checklist allows you to rate environmental walkability factors as you walk around your neighborhood. Available on-line at http://www.walkinginfo.org/walkingchecklist.htm. Promoting Active Communities Award, Community Self-Assessment Inventory. Governors Council on Physical Fitness, Health, and Sports. Michigan Fitness Foundation. Assessment checklist includes the following categories: policies and planning, pedestrian and bicycle safety and facilities, community resources, work sites, schools, and public transportation. Call 1-800-434-8642 for more information.
53
Appendix 6
Sample Case Studies
Case Study 1: Active Play Project
This evaluation case study is an example of a program designed to achieve school-based physical education, which is an intervention strongly recommended by the Task Force on Community Preventive Services to promote physical activity (see Appendix 2). Step 1: Engage Stakeholders During the planning of the project, project staff conducted the following activities to gain stakeholder involvement from the beginning: Contacted school principals with a letter and a follow-up telephone call to assess their interest in the project and enlist their support. Visited community health workers at the local health department to assess current, related programming efforts and to inform them about the Active Play project. Conducted focus group with parents to understand their feelings about physical activity related to their childrens health. Interviewed students in groups of two or three to learn what activities they enjoy. Additional stakeholders for the evaluation included Implementers: Teachers (both classroom and physical education); researchers who planned the project. Partners: Funder (a local foundation). Step 2: Describe or Plan the Program Several school districts in the state were identified by annual school height and weight surveys as having significantly higher rates of overweight and obesity than other districts. Nationally, almost 1 out of every 5 students is overweight; in these school districts, almost 1 out of 4 students is overweight. Therefore, schools and university-based researchers came together to plan a pilot project targeted at increasing students activity levels at school. Several schools from one of the districts with students at high risk were selected for the pilot project. The current evaluation was conducted during the implementation of the yearlong pilot project. Note that the evaluation was planned simultaneously with the project planning, and key evaluation stakeholders were involved from the first meeting. The logic model outlines project activities and expected outcomes.
55
Step 3: Focus the Evaluation The purpose of the evaluation of the Active Play pilot project was to identify ways to improve the project and to measure short-term outcomes. The project was in its first implementation year, so measuring longer-term impacts was not appropriate for this evaluation. The evaluation was used to create an annual report for the local funder, who would use it to determine whether to continue funding the project. The implementers used the evaluation to make informed changes to the project, which was likely to continue even if the funding decreased after the pilot year. After meeting with each stakeholder, the evaluators compiled the following evaluation questions: Were the project components implemented as planned? Did students become more active as a result of the project? What were the reactions of students and teachers to the project?
No. of teachers trained Teachers reactions to training Recess periods designated for active play No. of specific recess activities performed No. of additional opportunities for physical activity during class
56
Appendix 6
Step 4: Gather Credible Evidence This evaluation used a one-sample pretest and posttest.
Evaluation Questions
Were the project components implemented as planned?
Indicators
No. of teachers trained No. of minutes provided for specific physical activities during PE class No. of recess periods designated for active play No. of additional opportunities for physical activity during class
Data Sources
Training sign-in logs Observations of recess and PE classes (using SOFIT*) Teacher implementation checklist
Performance Indicators
80% of PE and classroom teachers trained 50% increase in minutes provided 20% increase in active play recess periods 15% increase in opportunities for physical activity
Percentage of time spent in moderate to vigorous physical activity in PE class Percentage of students who meet recommended levels of physical activity per day
Observations of recess and PE classes (using SOFIT) Accelerometer counts (worn by students) Interviews with students about their physical activity during the past day Posttraining evaluation forms Interviews with students
50% increase in minutes active in PE class 20% increase in students who get recommended physical activity per day
Not applicable
Step 5: Justify Conclusions Researchers analyzed the data and provided preliminary interpretations. Generally, results indicated that project components were implemented as planned and reactions of students and teachers to the Active Play project were positive. However, the increase in the number of active play recess sessions did not meet the performance indicator as indicated by the implementation checklists for classroom teachers. Looking at outcomes of the project, the number of students who achieved the recommended amounts of physical activity per day only increased by 5%. This increase was not significantly higher than preintervention levels and was well below the performance indicator of a 20% increase. Active minutes increased 10% as part of PE classes, which again was not significantly higher than the number of active minutes measured before the Active Play project. In interpreting these results, stakeholders had to make some decisions about which standards were most important for judging the data. To facilitate this process, stakeholders were brought together to review the findings and to make recommendations based on the data.
* McKenzie TL, Sallis JF, Nader PR. SOFIT: System For Observing Fitness Instruction Time. J Teach Phys Educ 1991;11:195205. 57
Step 6: Ensure Use and Share Lessons Learned As expected, stakeholders went back and forth in their opinions regarding the strength of the positive feelings associated with the project versus the nonsignificant behavioral outcomes. Some quotes from the meeting help to illustrate the perspectives of different stakeholders. Elementary school principal I think that this is a great project and we should make improvements based on the evaluation. Were moving in the right directionthe numbers show that kids are more active. This is a project that teachers and students like. Its fun for the kids and it challenges teachers to try something new. Community health worker The problem is that the project only focuses on schools. When kids go home after school, their parents dont encourage them to be activekids think its a treat to get to sit in front of the TV for 4 hours every night. Physical education teacher I dont know what else we can do besides offer time for kids to be active. One of the biggest issues is that the kids are only in PE 2 days a week. The only thing that matters to the school is proficiency tests these days. Classroom teacher It was hard sometimes to get kids organized during recess to play structured games. They have structure all day. Recess is supposed to be a time for free play, for creativity and doing what they want to do, not what someone tells them to do. That was hard for me. University researcher Even though there were some positive benefits to the project, we need to ask ourselves if those benefits are worth the time and money put into the project, because the outcomes that we wanted to see were not seen. Despite these differing perspectives, stakeholders compiled a short, concrete list of recommendations for improving the project. Each person was given an opportunity to suggest changes, then the group voted on which changes could be made and which recommendations were priorities. Unfortunately, the foundation did not support the project for another year because behavioral outcomes were not supported by the evaluation. Nonetheless, based on the relationships between stakeholders that were built during the year-long project planning, implementation, and evaluation, the project continued. The university provided the minimal funds needed for additional training, and university staff conducted the training as part of their community service requirements. PE teachers from nearby schools attended the training based on positive feedback they heard from other PE teachers in the pilot schools.
58
Appendix 6
The eight components are health education, physical education, health services, nutrition services, health promotion for staff, counseling and psychological services, healthy school environment, and parent/community involvement.
59
Public and Professional Education. Funds have been allocated to develop consistent health behavior messages across the state to raise awareness and motivate behavior change. This educational campaign will contain multiple components, including (but not limited to) traditional media, Internet-based approaches, and grassroots education. Surveillance and Evaluation. Funds have been allocated to create the Hawaii Outcomes Institute (HOI). This group will conduct an independent evaluation of HHI, create community health profiles, and serve as a data warehouse for health-related data in Hawaii. To create measurable geographical categories, HHI divided the state into 46 distinct geographical regions based on high school catchment areas. These divisions were used for school and community programs and facilitated evaluation because the amount of exposure a person could get from the program could be calculated by zip code. Step 3: Focus the Evaluation Because of the complexity of HHI, DOH sponsored a conference for international physical activity experts to help design the evaluation. As a result of this 3-day conference, eight recommendations for evaluation were proposed. Allow HOI to centrally guide the evaluation. Focus the evaluation on a limited number of target communities. Focus the major survey collection efforts on the Hawaii BRFSS. Form a technical advisory committee soon. Dont compromise quality for speed in entering the field. Keep the evaluation design simple. Keep the reporting requirements for community grants simple. Focus, focus, focus. The final words of advice from the committee were Do fewer evaluations better. Do good process evaluation always, good impact evaluation sometimes. The HHI evaluation team has been working for the last year to implement the recommendations of the expert panel. The evaluation is centrally guided by HOI under the direction of Jay Maddock, PhD, and Claudio Nigg, PhD, University of Hawaii. The evaluation design is simple. Process data is collected from all grantees using the University of Kansas (UK) Community Tool Box and tools developed by HOI. Intense,highlight evaluations are being conducted on six school and six community grantees. To supplement these data, a cross-sectional, longitudinal survey will be conducted in January 2002 and every 6 months thereafter. This survey will measure the mediators of change including stage of change, self-efficacy, perceived environment, attitude, subjective norm and benefits, and barriers for the three target behaviors. BRFSS will be used as the main behavioral outcome assessment, with the other 49 states serving as comparison groups. Tumor registry and hospital data will be used to measure the long-term impact of the program.
60
Appendix 6
Levels of Evaluation
Distal Profiles Hospital discharge data Tumor registry BRFSS Mediators survey (zip code) Knowledge/attitude/behavior Immediate Highlight communities/schools Moderators and process data
School interventions
Process evaluation
Outcome evaluation
Outcome evaluation INITIAL OUTCOMES Population shift in stage of change Changes in mediators (perceived environment/social norms)
INTERMEDIATE OUTCOMES Increase the no. of people getting 30 minutes of physical activity most days Decrease state level of obesity and overweight
61
Step 4: Gather Credible Evidence A multilevel design was implemented to measure the effectiveness of the HHI. This included Process analysis of all grantees. Further in-depth analysis of highlight schools and communities. A statewide survey to measure initial outcomes (i.e., stage of change, knowledge, attitude, perceived environment). BFRSS (main behavioral outcome; sample size = 6,000). Morbidity and mortality indicators (hospital data, Hawaii Tumor Registry).
Evaluation Questions
Were the project components implemented as planned?
Indicators
CDCs eight components implemented in schools Percentage of community action plans completed Media penetration
Data Sources
University of Hawaii process tracking UK Community Toolbox Media survey
Performance Indicators
At least 6 of 8 CDC components implemented in all grantee schools All communities have implemented at least one structural or environmental change 50% recall of HHI message
Percentage of smokers Percentage of people physically active at least 30 minutes a day most days of the week Percentage of people eating 5 fruits and vegetables a day
BRFSS
Step 5: Justify Conclusions Data will be analyzed in waves over the next several years. The first component will be an analysis of treatment fidelity. Process data from the three program areas will be analyzed, and each of the 46 catchment areas will be rated on the intensity of their intervention. A statewide summary for the end of each year (starting in 2002) will be developed to assess overall exposure to the program. Once this is complete, the mediators survey will be analyzed to assess movement in the stages of change and other relevant behaviors on the target variables. The survey is designed to yield reliable estimates for all six of the islands in the state and to compare communities with grants to control communities. Finally, BRFSS data will be compared longitudinally with the other 49 states to assess trend changes in
62
Appendix 6
the target behaviors. With population-based data, any significant change in the prevalence of the target behaviors (+1%) will have an important impact on the health of the state. For instance, a 1% decrease in the statewide prevalence of physical inactivity will equate to 8,700 people statewide. Step 6: Ensure Use and Share Lessons Learned With a large project like HHI, key stakeholders must be kept interested and motivated. Although we are just beginning our evaluation, we have developed several strategies to ensure continued success and share lessons learned. Grantees. Because of the numerous school and community grantees, we must maintain enthusiasm for the program, celebrate successes, and share lessons learned. Our evaluation of the highlight schools and communities will be used to feed information back to other grantees on what does and does not work well. This process should provide continual feedback to the grantees. Also, successes by the grantees will be highlighted in many ways, through community newsletters, grantee meetings, and public education. We feel these are important efforts to help grantees feel they are learning from each other and are not working in isolation. This information will also be fed back to DOH to guide future calls for proposals. HHI staff. Because of the many people at DOH and other organizations working on this project, feeding back information on successes and barriers is important. In addition to timely reporting of results, we are implementing a yearly survey with key stakeholders to assess their biggest successes and challenges of the past year and to ask them what could be done to make HHI more effective. This information will then be fed back to the team using summary data. Legislators and community members. HOI will develop a yearly summary of the progress of HHI to highlight the years major accomplishments. The summary will be delivered to state legislators and interested community members to inform them of HHIs progress and future directions. In addition, periodic press releases will be written to inform the public of major milestones. Professional dissemination. HOI staff will prepare technical reports, conference presentations and reports, peer-reviewed publications, and book chapters to keep health professionals informed of HHIs progress. We believe informing public health officials throughout the country about methods to evaluate change in statewide programs is important, and this will be a cornerstone of our effort in this step of the evaluation. This case study was prepared by Jay Maddock, PhD, and Claudio Nigg, PhD, University of Hawaii; and Angela Wagner, MPH, Hawaii State Department of Health. The authors would like to acknowledge the Hawaii DOH, which funded this evaluation through the Tobacco Settlement Fund; Bruce Anderson, PhD, and Virginia Pressler, MD, of HOI; the members of the HHI team who dedicated long hours to the development of the HHI; and Susan Jackson for her helpful comments on a earlier version of this case study.
63
64
Appendix 6
The resulting Take Our Trail campaign was conducted for 3 months in late spring 2001 by the health department and the Heart Health coalition. The campaign kicked off with a 3-mile Family Fun Walk, with T-shirts and refreshments donated by local businesses. For the length of the campaign, signs were strategically placed in busy areas throughout the community to raise community members awareness of the trail. A small, simple brochure was developed and provided to all programs in the local health department to distribute to their clients, as well as to clinics, physician offices, church leaders, and the Heart Health coalition. The brochure contained information on the importance of physical activity, tips to increase walking, safety, the trail, and who to contact for walking club information. The local television station created a public service announcement to promote the trail and the importance of regular physical activity during the evening news. The public transportation system placed signs inside their buses encouraging riders to Take Our Trail. The Heart Health coalition helped develop walking clubs at work sites, churches, and social organizations. These clubs established times and days for club members to meet and walk together on the trail. Local law enforcement officials agreed to patrol the walking trail periodically. The coalition also worked with local businesses, city government, and churches to raise money to enhance the trail, adding amenities such as lights, benches, mile markers, painted lanes, and a water fountain.
Safety of trail, hours of daylight, weather, trail and park maintenance INFLUENTIAL FACTORS
GOAL Improved health and quality of life and reduced chronic diseases
65
Step 3: Focus the Evaluation The primary purposes of this evaluation were to determine whether a promotional campaign would increase use of existing trails and whether trail use increases the number of persons who meet recommended levels of physical activity. Results of the evaluation would be used to make decisions about conducting a similar campaign in another community and about funding trail construction in additional communities. Therefore, the evaluation needed to include a) process measures for potentially replicating the campaign in the future and b) short-term outcome measures to see if behavior change or intentions to change behavior resulted from the campaign. Long-term physical activity behavior change also needed to be measured. Stakeholders agreed on the following four primary evaluation questions: What activities were actually conducted as part of the Take Our Trail campaign? Did trail use increase as a result of the Take Our Trail campaign? Who uses the trailboth before and after the campaign? To what extent do trails increase physical activity levels of community members? Step 4: Gather Credible Evidence Because two communities already had trails in place, the evaluation work group (composed of a lead staff person from the local health department and volunteer stakeholders identified in Step 1) decided to conduct a quasi-experimental trial. By conducting the Take Our Trail campaign in one community but not the other, they could determine whether trail use appeared to increase because of the campaign. If the promotion proved effective, the control community would conduct a similar campaign. A third, geographically distinct, sociodemographically similar community with no walking trail or campaign was used as an additional comparison group for measuring the longterm effects of trails on physical activity behavior. Stakeholders spent several meetings discussing and prioritizing indicators to measure their four primary evaluation questions and brainstorming about the best way to collect the necessary data. Two public health graduate students from a nearby university were recruited to plan and coordinate data collection as a project for an evaluation course. Additionally, several high school seniors in each community were recruited to help count and interview walkers as part of their community service requirement for graduation. The evaluation plan consisted of the following components: Trail usage evaluation. Process evaluation techniques were employed in the two communities with a walking trail. A multipurpose electronic counter was installed at
The multipurpose counters developed for subsequent intervention work in southeastern Missouri also included a card reader. Persons in the intervention group would swipe their card when they initiate and completed trail use. The length of time they spent on the trail and their pattern of use could be determined. This information allows individual tailoring of intervention messages.
66
Appendix 6
each of the walking trails before campaign commencement to monitor usage with laser technology by day and time. Counter data were collected 1 month before, during, and 1 month after the promotional and enhancement campaign. Walking trail counter data was cross-referenced with weather and local events. Throughout the campaign, the graduate students periodically visited the walking trails to count how many people were using the walking trail at specific times, to compare this information with the counter data and to document demographic characteristics of trail users. These visits varied by time and day. Trail user interviews. Graduate students were paired with the high school volunteers to randomly interview trail users 1 month before, during, and 1 month after the promotional and enhancement campaign. Data included walking, trail use, and other physical activity behavior; assessment of how the person found out about the trail and awareness of campaign materials; trail likes and dislikes; individual perception of increased walking since the trail existed; and positive and negative community consequences of having the trail. Stakeholder interviews. Additional stakeholders (e.g., church leaders and physicians) were interviewed about trail usage in the community and perceived consequences (both positive and negative) of the trails existence. Event logs. An event log system was developed to track all events that occurred in each community 1 month before, during, and 1 month after the promotional campaign and trail enhancement activities. First, everyone involved in the campaign (e.g., health department, Heart Health coalition) recorded activities on paper by hand. Recorded data included events at the walking trail, enhancements of the trail, formation of walking clubs, walking club meeting times and number of participants, and any other walking-related activities. These logs were then entered into a word processing program, and activities were categorized and coded by research assistants. Sample categories included services provided and community changes. Finally, coded data were used to make Microsoft Excel graphs to illustrate changes in different types of activities over the course of the campaign. Graduate students summarized these data for comparison between the two communities, and these data were used in conjunction with the counter data to explain increases or decreases in trail use. Media review. The graduate students were instructed to listen to PSAs, watch the evening news, and read newspaper articles to identify announcements relevant to the trail campaign. Staff members at health departments and clinics, physicians, and church leaders were surveyed to determine whether they had received Take Our Trail brochures and distributed them. Long-term behavioral outcome evaluation. Immediately before the Take Our Trail campaign began, a modified BRFSS survey composed of questions regarding walking behavior, chronic disease outcomes, and physical activity was randomly administered by telephone to a cross section of the two communities with a trail, as well as the community without a trail. The survey was administered again 1 year after this baseline data was collected.
67
Evaluation Questions
What activities were actually conducted as part of the Take Our Trail campaign? Number of bus signs Number of PSAs
Indicators
Data Sources
Event logs Media review
Number of newspaper articles Number of minutes of television coverage/promotion Number of brochures distributed Number of community events held at trail Number of community members at trail events Number of walking clubs formed Number of trail enhancements (e.g., benches, water fountains, restrooms, lights, mile markers, painted lanes) Hours of trail patrol by police force
Did trail use increase as a result of the Take Our Trail campaign?
Number of users before, during, and after the campaign in Take Our Trail community Number of users before, during, and after the campaign in control community with a trail Busiest time for trail use Awareness of campaign materials and messages
Who uses the trail, both before and after the campaign? How much do trails, increase physical activity levels of community members?
Interviews with walkers on the trail Key stakeholder interviews Electronic counter with card reader
Percentage of community who achieved recommended levels of physical activity before and after the campaign in communities with trails Percentage of community who achieved recommended levels of physical activity before and after the campaign in the control community without a trail Trail users perceptions of the effects of the trail on their physical activity behavior
Step 5: Justify Conclusions In general, the 3-month walking trail counter results indicated increased trail usage in the Take Our Trail community. The Take Our Trail community had a 35% increase in trail use between 1 month before and 1 month after the campaign, compared with a 10% increase in the community without the campaign. Initial walking trail counter data indicated that trail usage was highest on weekday mornings and lowest at night, on weekends, and in inclement weather. Data from the walking trail counter also indicated that trail usage was higher during Take Our Trail events in the campaign community. Usage increased more when walking clubs were formed in both communities (several walking clubs formed naturally in the control community and were recorded in the event log system), but the
68
Appendix 6
increase in the Take Our Trail community was significantly higher. In the final month of counter data collection, lunchtime trail usage increased, coinciding with formation of work site walking clubs. In addition, Sunday afternoon and Wednesday evening usage increased when church-based walking clubs were formed. Interviews with stakeholders indicated that persons in the campaign community felt safer while walking, compared with the community with a trail and no campaign, because of walking with partners (e.g., walking clubs), trail lights, and police patrols. Approximately 60% of trail users in both communities indicated an increase in walking since the trail existed. Most walkers and stakeholders felt the trail was an asset to their community and a source of community pride because it provided a free place for people to exercise. All types of people used the trail. Walkers were more likely to be women, older adults, athletes recovering from injuries, and persons with medical conditions that required a lowimpact activity. Those who used the trail generally felt safe while using it. The perception of safety increased in the Take Our Trail community after lights were added and police surveillance increased. Trail users in the Take Our Trail community had more positive responses to the interview question about trail likes and dislikes than did the comparison community. When asked how they became aware of the trail, most respondents indicated that they lived or worked near the trail or had heard about it at church or work or from friends or family. Some learned about the trail from their doctors. Few trail users had seen the fliers or PSAs and were generally unaware of the promotional campaign. The 1-year follow-up phone survey indicated a 5% increase in the number of persons meeting the physical activity recommendations in the Take Our Trail community, a 2% increase in the other community with a trail, and a 1% decrease in the community without a trail. Although these numbers are small, they could result in larger changes if the trends continue. For example, in 3 years, the community without a trail could have a 3% total decrease in the number of persons meeting the physical activity recommendations, whereas the Take Our Trail community could have a 15% increasea substantial improvement over the current rate. Step 6: Ensure Use and Share Lessons Learned Results of this evaluation indicated that construction of walking trails increased physical activity and implementation of a campaign to promote trail usage increased physical activity more by increasing use of the new trail. These findings were shared with DOT, with a recommendation to build additional walking trails and support campaigns aimed at increasing trial usage. The report to DOT also suggested that the focus of these campaigns should include community-wide involvement in promoting the trail and walking and enhancement of the trails. The most effective way to reach people is through the organizations they are affiliated with and through members of their social networks. Increasing safety and security is a must.
69
Another positive, unexpected result resulted from this evaluation. Community members, church leaders, and civic leaders worked together to determine methods for providing indoor walking areas to be used during cold winter months and other times when the weather prohibits outdoor walking. This included several churches and a community center installing marked indoor walking areas in their buildings and allowing access to nonmembers. This case study was prepared by Rashida Dorsey; Robyn A. Housemann, PhD, MPH; Imogene Wiggs, MBA; Ross C. Brownson, PhD; and Bernard Malone, MPA, of the Saint Louis University Prevention Research Center and Missouri Department of Health and Senior Services.
70