PROGRAM
EVALUATION
GUIDE
Tribal Motor Vehicle Injury Prevention
3
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
TRIBAL MOTOR VEHICLE INJURY PREVENTION
PROGRAM EVALUATION GUIDE
2022
DEVELOPED FOR:
National Center for Injury Prevention and Control
Centers for Disease Control and Prevention
DEVELOPED BY:
Albuquerque Area Southwest Tribal Epidemiology Center
Albuquerque Area Indian Heath Board, Inc.
This document was produced by the Albuquerque Area Southwest Tribal Epidemiology Center
through an Interagency Agreement between the National Center for Injury Prevention and Control,
Centers for Disease Control and Prevention, and the Indian Health Service,
Ofce of Public Health Support.
4
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
TABLE OF CONTENTS
Introduction 5
How to Use this Guide: Why? What? When? Who? How?
Framework for Tribal Motor Vehicle Injury Prevention Program Evaluation 11
Step 1 Engaging Partners 13
Step 2 Describing the Program 15
Step 3 Focusing the Evaluation Design 22
Step 4 Gathering Credible Evidence 26
Step 5 Justifying Conclusions 41
Step 6 Ensuring Use and Sharing Lessons Learned 45
Appendix A Evaluation Standards 49
Appendix B Additional Resources 53
Glossary 55
Acknowledgements 59
5
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
HOW TO USE THIS GUIDE
WHY WAS THIS GUIDE DEVELOPED?
This guide was developed to share best practices for evaluating Tribal Motor Vehicle Injury Prevention
(also called Tribal Trafc Safety) Programs. It is a companion document to the Tribal Motor Vehicle
Injury Prevention (TMVIP) Best Practices Guide 2016, which focuses on evidence-based strategies
to reduce the burden associated with motor vehicle crashes in American Indian and Alaska Native
communities. These strategies include increasing seat belt use, increasing child safety seat use,
and decreasing alcohol-impaired driving. This guide aims to assist Tribes and Tribal Organizations
that are implementing these proven strategies with key methods for evaluating their programs.
Specic examples are provided throughout the guide that pertain directly to Tribal Trafc Safety
Programs. At the same time, the evaluation principles and framework throughout the guide are
universal, and may also be useful for evaluating other public health programs in Indian Country.
HOW TO NAVIGATE THROUGH THIS GUIDE
This guide was developed to be an easy-to-use electronic document. Several hyperlinks—words
highlighted in color and underlined that can be clicked on—are included throughout the document.
Clicking on these hyperlinks will take you to a variety of external resources (e.g., websites, PDF
documents, or other resources) for more in-depth information about a specic topic, method, or
technique.
There is also a list of resources at the end of this guide that can be used to help strengthen
knowledge and skills in program evaluation.
6
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Tribal Motor Vehicle Injury Prevention Programs: What? Why? Who? When?
WHAT IS PROGRAM EVALUATION?
Definition: Program evaluation is the systematic collection of information about the activities,
characteristics, and outcomes of programs to make judgments about the program, improve program
effectiveness, and/or inform decisions about future program development.
Tribal Trafc Safety Programs are invaluable in American Indian/Alaska Native communities to
protect all generations from the harms of motor vehicle crash injuries, which continue to be a
leading cause of mortality across the lifespan. But how do we know if these programs are working?
And even if they are, how can we improve them to make them even better for tribal communities?
Good program evaluation can help us answer these questions, and provides results that can be
used to make our programs the best programs possible!
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Tribal Motor Vehicle Injury Prevention Programs: What? Why? Who? When?
WHY DO PROGRAM EVALUATION?
Most Tribal Trafc Safety Programs aim to change behavior in one or more populations of focus.
At the same time, they also strive to create an environment that reinforces sustained adoption of
these changes, with the intention that changes in environments and behaviors will prevent injuries.
Through evaluation, programs can track these changes and, with careful evaluation designs, assess
the effectiveness and impact of a particular intervention, strategy, and/or activity for producing these
changes.
Program evaluation enables programs to:
Create the best possible programs
Identify what is working well
Learn from mistakes
Modify program strategies and activities as needed
Monitor progress toward program goals
Assess the success of the program in achieving its short-term, intermediate,
and long-term outcomes
The ultimate purpose of program evaluation is to monitor progress toward your program’s goals and
use the information to improve your programs. The evaluation results can be used to demonstrate
the effectiveness of your program, identify ways to improve your program, modify program elements,
demonstrate accountability, and justify investment of resources.
✔ To demonstrate that resources are
being well spent and that the program
is effective
✔ To compare outcomes with those of
previous years
✔ To compare actual outcomes with
intended outcomes
✔ To justify the need for additional resources
and support
✔ To support annual and long-range planning
✔ To focus attention on issues important
to your program
✔ To promote your program
✔ To retain or increase resource investment
✔ To provide direction for program staff
✔ To identify training and technical
assistance needs
✔ To nd opportunities for continuous
quality improvement
MORE REASONS TO PRIORITIZE PROGRAM EVALUATION
7
•
•
•
•
•
•
8
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Tribal Motor Vehicle Injury Prevention Programs: What? Why? Who? When?
WHO SHOULD DO PROGRAM EVALUATION?
Program evaluation is an essential component of all Tribal Trafc Safety Programs. There are several
options that should be considered when determining who will lead evaluation activities in your
program.
Internal/External Evaluators: Traditionally, program evaluation was led by an external individual who
was not afliated with your program. The rationale for this approach was to ensure that the evaluator
would be objective and not introduce bias into your evaluation results. However, over the past
decade there have been new movements in program evaluation, such as empowerment evaluation,
which provides communities with the tools and knowledge that allow them to monitor and evaluate
their own performance. Neither approach is right or wrong, but you should know if your funding
source has specic requirements that determine who can evaluate your program.
Team Approach: Good evaluation requires a combination of skills that are rarely found in one
person. A preferred approach is to build an evaluation team that includes internal program staff,
external partners, and possibly consultants or contractors with evaluation expertise.
An initial step in the formation of an evaluation team is to decide who will be responsible for
planning and implementing evaluation activities. One or two persons can be selected as the
lead evaluator (or co-leaders) to coordinate program evaluation efforts. Leader(s) should be
responsible for overseeing evaluation activities, including planning and budgeting for evaluation,
developing program objectives, addressing data collection needs, reporting ndings, and working
with consultants. Although the leader(s) should have the skills necessary to competently coordinate
evaluation activities, he or she can also look elsewhere for technical expertise to design and
implement specic tasks.
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Tribal Motor Vehicle Injury Prevention Programs: What? Why? Who? When?
✔ Has experience in the type of evaluation
needed
✔ Can do both quantitative and qualitative
data collection and analysis
✔ Is aware of existing data sources
✔ Works successfully with a wide variety of
partners
✔ Incorporates evaluation into all program
activities
✔ Listens to your needs and ideas
✔ Educates program personnel in designing
and conducting the evaluation
✔ Honors important cultural considerations
for your program/community
✔ Keeps all data condential (consider using
a MOA or data sharing agreement)
✔ Gives staff and partners a full report of
ndings
✔ Agrees that the Tribal Trafc Safety Program
has sole ownership of the evaluation data
✔ Maintains neutrality and understands that
the ultimate purpose of program evaluation
is to make our programs the best programs
possible
CHARACTERISTICS OF A GOOD EVALUATOR
If you plan to include an external evaluator as part of your evaluation team, important factors to
consider include their level of professional training, experience, and ability to meet your needs. Overall,
it is important to nd an evaluator whose approach to evaluation best ts your program’s needs and
goals. Be sure to check all references carefully before you enter into a contract with any evaluator. Also
consider executing a Memorandum of Agreement (MOA) or data sharing agreement that outlines how
the condentiality of your data will be protected. Some places to nd experienced evaluators include
health departments, universities, non-prot organizations, and Tribal Epidemiology Centers.
TRIBAL EPIDEMIOLOGY CENTERS
TIP
There are 12 Tribal Epidemiology Centers (TECs) across the country.
TECs provide training and technical assistance to tribes and urban Indian
organizations in a variety of program evaluation related activities, including:
Developing evaluation plans Collecting new data Interpreting results
Designing survey instruments Analyzing data Developing reports
Gathering existing data
TECs can be a valuable partner in your program evaluation activities. If you are not familiar
with the TEC in your area, you can learn more at www.tribalepicenters.org.
9
•
•
•
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Tribal Motor Vehicle Injury Prevention Programs: What? Why? Who? When?
WHEN SHOULD I EVALUATE?
Program evaluation should occur before, during, and after your program. Plans for evaluation should
be a part of planning for the implementation of the program. The evaluation program helps establish
a baseline from which changes in participants or the impact on the community can be tracked.
Evaluation during program implementation can help you make immediate changes/adjustments in
the program as it is progressing. This helps staff nd the strengths and weaknesses in the program
while it is still going on and is helpful for program improvement. Evaluation after the completion of
a program is valuable to sum up what has occurred, assess the extent to which you have achieved
your target outcomes, identify successes and lessons learned, and reect on the value of the
program.
BEFORE
Conduct needs
assessment
Establish baselines
and targets
Create evaluation
plan that aligns with
your program goals,
objectives,
and outcomes
AFTER
Assess achievement
of outcomes
Compare actual
results to targets
and standards
Identify successes
and lessons learned
DURING
Process evaluation
Monitor if program
activities are being
implemented as intended
Assess progress toward
short-term outcomes
Make immediate
adjustments as the
program is progressing
Improve quality
continuously
10
•
•
•
•
•
•
•
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Framework for Evaluating Tribal Trafc Safety Programs
DESCRIBE
THE PROGRAM
JUSTIFY
CONCLUSIONS
ENSURE USE
AND SHARE
LESSONS
EVALUA
FOCUS
DESIGN
TION
HOW TO EVALUATE YOUR TRIBAL TRAFFIC SAFETY PROGRAM
The remainder of this guide is designed to help you answer this nal question: “How do I evaluate
my Tribal Traffic Safety Program?” The framework described below is a practical, non-prescriptive tool
that summarizes the important elements of program evaluation. It was developed by the Centers for
Disease Control and Prevention (CDC), and offers a way to understand and improve your program
evaluation using methods that are useful, feasible, proper, and accurate. The framework contains
two related dimensions: 1) steps in evaluation practice, and 2) standards for good evaluation.
A FRAMEWORK FOR PROGRAM EVALUATION
ENGAGE
PARTNERS
STANDARDS
UTILITY
FEASIBILITY
PROPRIETY
ACCURACY
GATHER
CREDIBLE
EVIDENCE
11
12
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Framework for Evaluating Tribal Trafc Safety Programs
The six connected steps of the framework are actions that should be a part of any evaluation:
1. Engage partners
2. Describe the program
3. Focus the evaluation design
4. Gather credible evidence
5. Justify conclusions
6. Ensure use of findings and share lessons learned
These steps are meant to be adaptable, not rigid. They are intended to serve as starting points
around which Tribal Trafc Safety Programs can tailor an evaluation to best meet their needs.
Sensitivity to each program’s unique context (e.g. culture, history, capacity, and organizational
climate) is essential for impactful evaluation.
The second part of the framework is a basic set of standards to assess the quality of evaluation
activities. There are 30 specific standards, organized into the following four groups:
Utility: Who needs the evaluation results? Will the evaluation provide relevant information in a
timely manner for them?
Feasibility: Are the planned evaluation activities realistic given the time, resources, and
expertise at hand?
Propriety: Does the evaluation protect the rights of individuals and the welfare of those involved?
Does it engage those most directly affected by the program, such as participants or the
surrounding community?
Accuracy: Will the evaluation produce ndings that are valid and reliable, given the needs of
those who will use the results?
Each of these steps and standards will be explored in detail throughout the remainder of this guide.
•
•
•
•
13
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 1: Engaging Stakeholders
STEP 1: ENGAGING PARTNERS
Partners are people or organizations that are invested in the program or that are interested in the
results of the evaluation or what will be done with results of the evaluation.
Key partners for evaluations of public health programs typically fall into three major groups:
Those involved in program operations, i.e., program management, staff, and partnering programs
Those served or affected by the program, i.e., tribal members
Those who are intended users of the evaluation findings, i.e., tribal leaders, program partners,
funding agencies, and other persons in a position to make decisions about the program
WHY INVOLVE PARTNERS IN
EVALUATION?
Evaluation cannot be done in isolation. Almost
everything done in programs to improve
community health and wellness involves
partnerships. Therefore, program evaluation
must consider the different values and interests
held by the partners. Partners must be part
of the evaluation to ensure that their unique
perspectives are understood. When partners are
not appropriately involved, evaluation ndings
are likely to be ignored, criticized, or resisted.
However, if they are part of the process, people
are likely to feel a good deal of ownership for the
evaluation process and results. That’s why this
evaluation cycle begins by engaging partners.
GIVE PRIORITY TO PARTNERS WHO:
Can increase the credibility of your efforts or
your evaluation.
Are responsible for day-to-day implementation
of the activities that are part of the program.
Will advocate for or authorize changes to the
program that the evaluation may recommend.
Will support or authorize the continuation or
expansion of the program.
•
•
•
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 1: Engaging Stakeholders
TYPES OF PARTNERS IN TRIBAL TRAFFIC SAFETY PROGRAMS
Although each Tribal Trafc Safety Program is unique, some common partners may include, but are
not limited to:
Tribal leaders I/T/U Health Facilities Emergency Medical Services
Schools Environmental Health State and Government
Head Start Community Members Women Infants and Children (WIC) Program
Day Care Centers Bureau of Indian Affairs Community Health Representatives (CHR)
Law Enforcement Indian Health Service Transportation Department
TIPS FOR ENGAGING PARTNERS
✔ Include a diverse group of partners.
✔ Identify key areas for partner input.
✔ Create a plan for partner engagement.
✔ Involve partners at the beginning, middle, and end of your program evaluation activities.
✔ Include partners at various levels based on their preferences, interests, and expertise:
Direct involvement in designing and conducting the evaluation
Advisory role providing feedback and guidance on program evaluation activities and
reports through routine communication
✔ Keep partners informed regarding the progress of the evaluation through periodic meetings,
reports and other preferred means of communication.
✔ Partners are more likely to support the evaluation and act on results and recommendations
if they are involved in the evaluation process.
TIP
14
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
15
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 2: Describing the Program
STEP 2: DESCRIBING THE PROGRAM
This step centers upon developing a comprehensive description of your Tribal Trafc Safety Program
that outlines all of its key components and intended outcomes. Completing this step will help you
focus your evaluation on the most central and important questions. This step can either follow the
partner engagement step or precede it. It is important to note that in this step you are describing
the program and not the evaluation. You can use a tool called a logic model for this purpose (see
page 19), but a program description can be developed without using this or any tool. Either way, a
comprehensive program description includes the following components:
Need. What is the big public health problem you aim to address with your program?
Focus. Which individual groups, programs, departments, or organizations need to change or take
action to ensure progress on the public health problem?
Outcomes. How and in what way do these groups, programs, departments, or organizations need
to change? What action specically do they need to take?
Activities. What will your program and its staff do to move these groups, programs, departments,
or organizations to change/take action?
Outputs. What tangible capacities or products will be produced by your program’s activities?
Resources/Inputs. What is needed from the larger environment in order for the activities to be
successful?
Each of these components will be further described in the remainder of this section.
NEED FOR PROGRAM
The need is the public health or other problem addressed by the program. For most Tribal Trafc
Safety Programs the public health problem is motor vehicle crash injuries and deaths. You
might dene the need, in terms of its consequences for your tribe or community, the size of the
problem overall, the size of the problem in various segments of the population (e.g., children and
adolescents), and/or signicant changes or trends in the rate of motor vehicle crash injuries. For
example, the need for a particular Tribal Trafc Safety Program might be to reduce the high rate of
motor vehicle crash injuries to American Indian children due to insufcient child passenger safety
restraint use (i.e., use of weight, height, and age appropriate car seats, booster seats, and seat
belts among children).
FOCUS
Focus on the various audiences the program needs to engage in order to make progress on the
public health problem. For Tribal Trafc Safety Programs this might include tribal members (e.g.,
parents of infants, children, adolescents, adults, elders, or the whole community), as well as
key programs or departments (e.g., law enforcement, Community Health Representatives, tribal
leadership, etc.). Reducing motor vehicle crash injuries will require not only individual-level actions by
tribal members (e.g., using seat belts, car seats, and booster seats), but also community, system,
and environmental level actions such as improvements to roads, increased signage, enforcement of
laws, distribution of child safety seats, community education, etc.
•
•
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 2: Describing the Program
OUTCOMES
Outcomes are the changes in someone or something (other than the program and its staff) that
you hope will result from your program’s activities. For Tribal Trafc Safety Programs, the ultimate
outcome is often ambitious and long-term, such as reducing motor vehicle crashes with injuries or
decreasing motor vehicle-related fatalities. A strong program description provides details not only
on the intended long-term outcomes, but also on the short-term and intermediate outcomes that
precede it.
SAMPLE SHORT-TERM, INTERMEDIATE-TERM, AND LONG-TERM
OUTCOMES FOR TRIBAL TRAFFIC SAFETY PROGRAMS
OUTCOME TYPE DEFINITION EXAMPLES
Short-Term
The initial expected changes in
your population(s) of focus after
implementing certain activities
or interventions (e.g., changes in
knowledge, skills, and attitudes).
Increased tribal member skills in
installing car seats
Increased knowledge about safe driving
and the harms of distracted and alcohol-
impaired driving
Enhanced collaboration/communication
among program partners (e.g., Tribal
Trafc Safety Program, law enforcement,
CHR program, and schools)
Intermediate-
Term
The interim results that provide
a sense of progress toward
reaching the long-term objectives
(e.g., changes in behavior,
practices, actions, and policy)
Intermediate outcomes specify
what individuals do with the
short-term outcomes.
Increased use of seat belts among tribal
members
Increased use of car seats and booster
seats for children
Increased citations for speeding,
impaired driving, and failure to use
seat belts
Primary seat belt law passed and
implemented in the tribal community
Long-Term
Long-term outcomes are those
that result from the achievement
of your short- and intermediate-
term outcomes. Typically, but
not always, these mirror the goal
statement, and are achieved only
after the program has been in
place for some time.
Decreased motor vehicle crash-related
mortality
Fewer motor vehicle crashes that result in
injury
Improved quality of life
Increased life expectancy
16
•
•
•
•
•
•
•
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 2: Describing the Program
ACTIVITIES
Activities are the actions taken by the program and its staff to achieve the desired outcomes in
the groups of focus. Typical program activities might include outreach, training, education, service
delivery, collaborations and partnerships, and health communication. Most Tribal Trafc Safety
Programs will engage in numerous activities to achieve their desired outcomes, such as those
highlighted in the text box below.
SAMPLE ACTIVITIES OF TRIBAL TRAFFIC SAFETY PROGRAMS
✔ Conduct car seat installations
✔ Conduct community education/
presentations
✔ Conduct observational seat belt checks
✔ Provide classroom education in high
schools about the hazards of distracted
driving
✔ Review existing tribal laws and policies that
relate to trafc safety
✔ Recruit partners to participate in a
community trafc safety coalition
✔ Implement a community scan for road and
signage improvement needs
✔ Identify training opportunities for
program staff
✔ Coordinate safe driving courses
✔ Gather existing data on motor vehicle
crashes and injuries in the community
✔ Advertise program events
✔ Recruit participants
✔ Implement sobriety checkpoints
✔ Create and disseminate small media (e.g.,
posters, digital stories, brochures, fact
sheets, radio PSAs, etc.)
✔ Conduct surveys
✔ Share information on evidence-based
trafc safety policy interventions.
17
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 2: Describing the Program
SAMPLE OUTPUTS FOR TRIBAL TRAFFIC SAFETY PROGRAMS
✔ Consultations with partners or other
partners around evidence-based trafc
safety policies
✔ Seat belt, DUI, and speeding checks
conducted
✔ Car seats checked
✔ Coalition meetings conducted
✔ Community presentations conducted and
participants reached
✔ Car seats installed
✔ Outreach events/health
fairs conducted
✔ Tribal members participating in safe driving
course
✔ Memoranda of Agreement executed among
partners
✔ Programs/departments represented on
coalition
✔ Staff trained as child passenger safety
technicians
✔ Materials developed and disseminated
Outputs are the direct products of activities, usually some sort of tangible deliverable.
Outputs can be viewed as activities redened in tangible or countable terms. For example,
the number of:
18
19
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 2: Describing the Program
RESOURCES/INPUTS
These are the people, money, infrastructure, and other resources needed—usually from outside
the program—to implement program activities effectively. It is important to include inputs in the
program description because accountability for resources to funders and partners is often a focus
of evaluation. Just as important, the list of inputs is a reminder of the type and level of resources
on which the program is dependent. If intended outcomes are not being achieved, look to the
resources/inputs list for one reason why program activities could not be implemented as intended.
Examples of Resources/Input for Tribal Traffic Safety Programs
Funding
Program staff
Partners (e.g., police, tribal leaders, schools, CHRs, etc.)
Infrastructure
Materials & equipment
Data
LOGIC MODEL
Once the components of the program description have been
identied (i.e., need, focus, outcomes, activities, outputs,
resources/inputs), a visual depiction might help to summarize the relationship among the
components of your Tribal Trafc Safety Program. The logic model requires no new thinking about the
program; rather, it converts the raw material in the program description into a picture of the program.
This visual tool can help with both strategic planning and program evaluation.
•
•
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 2: Describing the Program
In other words, logic models are graphic depictions of the relationship between a program’s activities
and its intended outcomes. Two words in this denition bear emphasizing:
Relationship: Logic models convey not only the activities that make up the program, but also
how they relate to each other, as well as their relationship with your outputs and outcomes.
Intended: Logic models depict intended outcomes of a program’s activities. As the starting
point for evaluation and planning, the model serves as an “outcomes road map” that shows the
underlying logic behind the program, i.e., why it should work.
LOGIC MODEL COMPONENTS
Logic models come in many different shapes and sizes. At a minimum, they should include
the following components from your program description:
Inputs: Resources that go into the program and on which it is dependent to execute
its activities
Activities: Actual events or actions done by the program and its staf
Outputs: Direct products of program activities, often measured in countable terms
(e.g., the number of education sessions held, number of car seats installed, etc.)
Outcomes: The changes that result from the program’s activities and outputs, often
in a sequence expressed as short-term, intermediate-term, and long-term outcomes
20
•
•
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 2: Describing the Program
SAMPLE LOGIC MODEL FOR TRIBAL TRAFFIC SAFETY PROGRAMS
INPUTS ACTIVITIES OUTPUTS OUTCOMES
• Program Staff
• Car seat clinics
• Number of car
Short-Term Intermediate Long-Term
• Increased
• Increased use
• Decrease in
• Partners
• Safe driving
seats inspected
number of
of car seats
motor vehicle
(police, school,
classes
& installed
vehicles with
• Increased use
crashes
WIC, CHR,
• Media
• Number
appropriate car
of seat belts
• Decrease in
transportation)
campaign
of classes
seats
• Increased
motor vehicle
• Grant funding
• Seat belt
conducted
• Increased
enforcement of
crash-related
• Data (State
observation
• Number
knowledge
seat belt and
injuries and
Vital Records,
EMT logs, IHS
• Car seat
observation
• Policy
development
of class
participants
• Number of
community
members
about safe
driving
practices
• Increased
community
car seat use
death
• Increased life
expectancy
and quality of
life for AI/AN
reached
awareness
people
through media
about trafc
campaign
safety
• Number of
• Improved
observations
surveillance of
completed
seat belt and
• Number of
car seat use
policies drafted
• Enhanced tribal
trafc safety
policies
21
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 3: Focusing the Evaluation Design
STEP 3: FOCUSING THE EVALUATION DESIGN
After completing Steps 1 and 2, you and your partners should have a clear understanding of your
Tribal Trafc Safety Program and are ready to focus the evaluation. This includes determining the
most important evaluation questions and the appropriate design for the evaluation. Rather than
waiting to conduct evaluation when the program ends and asking, “Did the program work?”, the
framework outlined here views evaluation as an ongoing activity over the life of a program that asks,
“Is the program working?”
EVALUATION TYPES
There are several types of evaluations. Some of the most common types include: formative,
summative, process, outcome, and impact. In this guide, we will focus on two complementary
evaluation types that all Tribal Trafc Safety Programs should conduct: process and outcome.
PROCESS EVALUATION
Process evaluation assesses whether or not a program is operating as intended and identies
opportunities to strengthen it. In process evaluation, you examine whether the planned activities
are taking place, who is conducting the activities, who is reached through the activities, and whether
sufcient inputs have been allocated or mobilized. You will also examine whether program activities
have resulted in certain outputs. Remember, outputs are the direct products of the program activities
which provide evidence that the activities were accomplished.
Process evaluation should be conducted throughout the life of your program.
The results of your process evaluation will strengthen your ability to report on your program
accomplishments and can be used to improve future activities.
PROCESS EVALUATION
Process evaluation helps you to track program information related to who, what, when,
where, and how? Such as:
What has your program done?
When did your program activities take
place?
Where did your program activities take
place?
Who received services?
How much or for how long did they
receive these services?
What are the barriers/facilitators to
implementating your program activities?
22
•
•
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 3: Focusing the Evaluation Design
OUTCOME EVALUATION
Outcome evaluation measures the degree to
which your Tribal Trafc Safety Program is being
effective in meeting it’s objectives. Outcome
evaluation measures a program’s results and
helps determine whether a program or strategy
produced the changes it intended to achieve.
In other words, the main goal of outcome
evaluation is to determine whether or not the
program contributed to benecial effects on
your specic population. Outcome evaluation
assesses the short-term, intermediate-term, and
long-term effects of your program, and might
include things such as:
Changes in people’s attitudes and beliefs
Changes in risk or protective behaviors
Changes in the environment, including
policies, enforcement of laws, roads and
signage, and programs/departments
Changes in morbidity and mortality
PROCESS AND OUTCOME EVALUATION GO TOGETHER!
It is important to note the usefulness of conducting process evaluation while you are also
implementing outcome evaluation. For example, if the outcome evaluation shows that
your Tribal Trafc Safety Program did not produce the expected results, it may be due to
program implementation issues. These potential issues will be much easier to identify
if you have been measuring them through process evaluation activities. Likewise, if your
process evaluation demonstrates that your program is only reaching one subgroup of your
population of focus (e.g., females), you would be less likely to expect change for other
subgroups (e.g., males) in the community during outcome evaluation. Therefore, process
and outcome evaluation are complementary, and it is recommended that all Tribal Trafc
Safety Programs conduct both simultaneously!
23
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 3: Focusing the Evaluation Design
DEVELOPING EVALUATION QUESTIONS
At this point, it is important to develop evaluation questions to focus and guide your program
evaluation activities. Your evaluation questions should reect the basic question of whether or not
your program worked as intended.
Many different questions can be selected to guide your program evaluation. Most can t into four
main categories:
QUESTION CATEGORIES POSSIBLE QUESTIONS
PLANNING AND IMPLEMENTATION:
How well was your Tribal Trafc Safety
Program planned, and how well was that
plan put into practice?
Were your program’s activities put into
place as originally intended?
How many events were conducted?
Who participated?
Was there diversity among participants?
Why do participants enter and leave your
program?
Do those most in need of help receive
services?
Are community members satised with
your program?
ASSESSING ATTAINMENT OF OBJECTIVES:
How well has the Tribal Trafc Safety
Program met its stated objectives?
How many people participated?
Were activities being implemented
according to your planned timelines?
Did program activities yield your
expected outputs?
IMPACT ON PARTICIPANTS:
How much and what kind of a difference has
the Tribal Trafc Safety Program made for its
population of focus?
How has behavior changed as a result of
participation in the program?
What new knowledge or skills have been
obtained by participants?
IMPACT ON THE COMMUNITY:
How much and what kind of a difference has
the Tribal Trafc Safety Program made in the
community as a whole?
What new policies resulted from the
program?
Were there any changes in infrastructure
or the environment?
What new resources or infrastructure
has been created by the program?
24
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 3: Focusing the Evaluation Design
PRIORITIZED EVALUATION QUESTIONS
Are important to your program and partners.
Address important program needs.
Can be answered with available resources, including funds and staff capacity.
Reect program goals and objectives.
Are realistic and can be answered within the available timeframe.
Provide information to make program improvements.
TIP
SAMPLE EVALUATION QUESTIONS FOR TRIBAL TRAFFIC SAFETY PROGRAMS
Did the program follow the basic plan for implementation?
What services were provided through the program (e.g., outreach, education, car seat
installations/distribution, seat belt checks, coalition building, policy development,
new signage, safe driving classes, DUI checks, etc.)?
What was the quality of the services provided through the program?
How much did the community participate in the program?
What are the characteristics of the individuals who participated in the program?
What is the participants’ satisfaction with the program?
Who still needs to be recruited to participate?
What are staff members’ perceptions of the program?
What are the planning products of the programs (e.g., logic models, strategic plan, etc.)?
What materials were developed through the program (e.g., video, brochures, curriculum, etc.)?
How many program materials have been distributed?
What were the barriers to participation in the program?
What were the barriers to the implementation of the program?
What resources are still needed to implement the program as planned?
How much progress did the program make toward short-term, intermediate-term and
long-term outcomes?
What changes occurred among individuals that participated in the program?
What community-level/system-level changes occurred as a result of the program?
What educational opportunities were used to discuss evidence-based policy options
around trafc safety with partners and policymakers?
25
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 4: Gathering Credible Evidence
STEP 4: GATHERING CREDIBLE EVIDENCE
Now that you have created your evaluation questions, your next task is to gather the evidence to
answer them. Your evaluation should strive to collect information that will convey a well-rounded
picture of your Tribal Trafc Safety Program. Therefore, you should plan to gather multiple types of
evidence to evaluate your program.
THE TWO MOST IMPORTANT STEPS TO GATHERING CREDIBLE EVIDENCE ARE TO:
1. Select meaningful indicators (sometimes called performance measures).
2. Select appropriate data collection methods and sources to track your indicators.
Indicators are specic, observable, and measurable characteristics or changes that show the
progress a program is making toward achieving a specied outcome. In other words, your indicators
are the specic item(s) that you will measure to answer your evaluation questions (e.g., attendance,
events conducted, meetings convened, people trained, new materials developed, changes observed,
etc.). Indicators reect aspects of the program that are meaningful for monitoring. They can be
related to your process or outcome evaluation. Some examples of indicators that can be dened and
tracked by your Tribal Trafc Safety Program are highlighted in the table below.
SAMPLE INDICATORS FOR TRIBAL TRAFFIC SAFETY PROGRAM
Process Indicators Outcome Indicators
Attendance/participation Changes in participant behavior
Participant characteristics (age, sex, job title, etc.) Changes in knowledge, attitudes, and beliefs
Number of events/activities conducted Changes in policies
Number of meetings held Changes in health status
Number of data use agreements established
Changes in enforcement of laws (seat belt
warnings/citations, child safety seat warnings/
citations, DUI arrests)
Number of assessments/observations
conducted
Changes in quality of life
Number of materials developed & distributed Changes in services or practices
Number of staff trained Changes in road conditions
Number of classes held Changes in trafc signage
26
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 4: Gathering Credible Evidence
Indicators should be developed for
activities (process indicators) and for
outcomes (outcome indicators).
There can be more than one indicator for
each activity or outcome.
The indicator must be clear and specic in
terms of what it will measure.
Indicators should meaningfully address
your evaluation questions.
Indicators should align with your program
goals and objectives.
Select a mix of indicators at the
individual-level (e.g., change in participant
knowledge), community-level (e.g.,
number of car seats distributed throughout
community), and systems/environmental
levels (e.g., change in policies,
availability of safe driving classes, roads
improvements/signage, enforcement of
seat belt, car seat, speeding, and impaired
driving).
KEEP THE FOLLOWING TIPS IN MIND WHEN SELECTING YOUR INDICATORS:
TIP
Another helpful tip for selecting indicators is to look at the SMART objectives you have established
for your program. If your program objectives were written to be specic, measurable, action-oriented,
realistic, and time-bound (so-called “SMART” objectives), they will likely include indicators as well.
SMART OBJECTIVES
Have you ever wondered why so many grants require you to include SMART objectives?
S = Specific
M = Measurable
A = Action-oriented (achievable)
R = Realistic
T = Timebound
Not only do SMART objectives provide details of your program plan, they also lay the foundation
for your program evaluation by:
indicating specic approaches that will be implemented.
identifying key indicators or performance measures.
establishing baselines and targets for your indicators.
outlining timeframes for completion of key program components.
27
•
•
•
•
•
•
•
•
•
•
28
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 4: Gathering Credible Evidence
SELECTING DATA COLLECTION METHODS AND SOURCES
Now that you have determined the activities and outcomes that you want to measure and the
indicators that you will use to measure progress on them, you need to select data collection
methods and sources for your indicators. A key decision is whether there are existing data
sources—secondary data—that can be gathered to measure your indicators or whether you need to
collect new data—primary data. Most likely you will use both!
SECONDARY DATA SOURCES
Secondary data are data that have already been collected by someone else. Depending on your
evaluation questions and indicators, some secondary data sources might be appropriate. For example,
existing data sources that might be particularly useful for Tribal Trafc Safety Programs include:
Local:
EMT/EMS logs
Tribal Departments of Transportation
Police logs, reports, records
Community surveys and needs assessments
IHS/Tribal/Urban Indian Health facility
Maps
electronic health records
State:
Vital records/statistics (mortality, death certicates)
Hospital discharge data (motor vehicle crashes that resulted in hospitalization)
Emergency department data (motor vehicle crashes that resulted in an ED visit)
State Departments of Transportation/State Highway Safety Ofces
National:
NHTSA’s Fatality Analysis Reporting System (FARS), data regarding fatal injuries suffered in
motor vehicle trafc crashes
CDC’s Web-based Injury Statistics Query and Reporting System (WISQARS), an interactive,
online database that provides fatal and nonfatal injury data
CDC’s Behavioral Risk Factor Surveillance System (BRFSS), a national survey of health-related
risk behaviors among adults, including seat belt use and drinking and driving data
CDC’s Youth Risk Behavior Surveillance System (YRBSS), a national survey of health-related
risk behaviors among adolescents, including seat belt use, texting and driving, and drinking and
driving data
Inventory of National Injury Data System, a comprehensive list of 45 different federal data
systems operated by 16 different agencies and 3 private injury registry systems that provide
nationwide injury-related data
CDC’s Pregnancy Risk Assessment Monitoring System (PRAMS), a system that collects state-
specic, population-based data on maternal attitudes and experiences before, during,
and shortly after pregnancy
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
29
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 4: Gathering Credible Evidence
Reach out to your Tribal Epidemiology Center or
State Department of Health for technical support
TIP
in accessing secondary data that may be helpful
to evaluate your Tribal Traffic Safety Program.
PRIMARY DATA SOURCES
Primary data are data that are collected by your evaluation team to answer your evaluation
questions. Primary data can be quantitative or qualitative. It is important that data are collected in
an ethical manner. Please see Propriety Standards on Page 50 for additional information.
Quantitative data are data that can either be counted or compared on a numeric scale. These
data are used to measure a problem or address the questions of “what?” or “how many?”.
Qualitative data are data concerned with descriptions, which can be observed but cannot
be computed. Qualitative data describes qualities or characteristics. It is collected using
questionnaires, interviews, or observation, and frequently appears in narrative (words) form. Data
collected include transcripts of interviews and focus groups and eld notes from observation of
certain program activities.
Primary data collection methods for program evaluation typically fall into several broad categories.
Among the most common that are used to evaluate Tribal Trafc Safety Programs include:
Surveys
Key Informant Interviews
Focus Groups
Observation
Document Review
The remainder of this section will provide denitions of each primary data collection type and tips for
how you can incorporate them into the evaluation of your Tribal Trafc Safety Program.
•
•
•
•
•
•
•
30
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 4: Gathering Credible Evidence
1. SURVEYS
A survey is a way of collecting information to understand the views of the community or group in
which you are interested. Survey instruments or questionnaires ask questions in a standardized
format that allows consistency and the ability to aggregate (combine) responses. Most survey
questions are quantitative and closed-ended, requiring participants to select an answer from a
given list. However, surveys may also include some open-ended (qualitative) questions that allow
participants to provide an answer in their own words. Some common types of surveys used in
program evaluation include:
Pre-Post Test – A survey instrument given to participants before and after a training, an
intervention, or a particular period of time to assess changes in participant knowledge, attitudes,
beliefs, and/or intentions. The same survey is administered at two different time points, and the
changes are evaluated to determine impact.
Satisfaction Survey – A satisfaction survey is typically administered only after an event, training,
or intervention to assess participant satisfaction with program services and activities.
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 4: Gathering Credible Evidence
WHO SHOULD I SURVEY?
Deciding who you will survey (your sample)
depends on the purpose of the survey. If you are
trying to evaluate the impact of your project on
people who have participated in your activities,
you should administer the survey to those indi-
viduals who have participated in one or more of
your program activities. Similarly, if you are trying
to determine the impact of your project on spe-
cic groups of people, such as youth, parents,
adults, women, etc., you may want to just survey
people in the community with those characteris-
tics. If you are trying to understand the impact
of your program on the entire community, you
should administer the survey to a much broader
group of participants, including those who did
not participate in project specic activities. This
could be a random sample of community mem-
bers, where everyone in your community has an
equal chance of being selected to participate.
Random samples are often drawn from existing
lists of community members (e.g., voter registration, tribal enrollment/census, mailing lists, IHS/
Tribal/Urban Indian Health facility users, etc.), or community maps (i.e., random selection of homes
throughout the community). Other options include: 1) census, where all community members are
selected to participate; 2) convenience sample, where survey participants that are easy to reach
are selected (e.g., surveying participants gathered at an event, at the post ofce, in the clinic, etc.);
and/or 3) volunteers, where individuals self-select to participate.
SAMPLE TYPE PROS CONS
Census or Random Sample
More representative of the
whole community
More expensive/resource
intensive
More time consuming
Convenience Sample
or Volunteers
Quicker
Less expensive/resource
intensive
Usually not representative
of the whole community
Results cannot be
generalized
31
•
•
•
•
•
•
•
32
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 4: Gathering Credible Evidence
10 BEST PRACTICES FOR CREATING EFFECTIVE SURVEYS
1. Keep your survey short. In most cases, people are doing you a favor by taking your survey.
What better way to respect their time than by not taking up too much of it? You’ll be rewarded
with a higher completion rate as well as more thoughtful responses for the questions you end
up including.
2. Keep your questions clear and simple. Be specic and concrete in developing your questions
and be sure to avoid technical words or jargon that are commonly used among your Tribal Trafc
Safety Program staff and colleagues, but might not be well known to your community members.
3. Ask only questions that will help you meet your goal. Every question you include should have a
well-dened purpose and a strong reason for being there. Remove any questions that appear to
duplicate one another or will not provide information that is helpful to your evaluation.
4. Place the more personal questions to the end. Structure your survey like a conversation.
Would you start an exchange by asking someone how old they are? Probably not. Instead, you’d
engage in small talk rst, and gradually move on to more personal topics. Similarly, keep your
early set of questions light and straightforward and then slowly move towards more personal
questions (often taking the form of demographic questions).
5. Focus on using closed-ended questions. What do we mean by closed-ended questions? We’re
talking about questions that use pre-populated answer choices for the respondent to choose
from—like multiple choice or checkbox questions. These questions are easier for respondents
to answer and provide you with quantitative data to use in your analysis. Open-ended questions
(qualitative) ask the respondent for feedback in their own words. Since open-ended questions
can take much longer to answer, try to only include 1-2 of them at the end of your survey.
6. Consider including a survey incentive. If you’re keen on getting a lot of responses, an incentive
in some form can prove helpful. Potential incentives range from entering respondents into a
drawing to giving respondents a gift card if they answer all of your questions.
7. Don’t ask leading questions. In other words, try not to put your own opinion into the question.
Doing so can inuence the responses in a way that doesn’t reect participants’ true experiences.
For example, a leading question might ask: You are satisfied with our trainings, aren’t you?
The right question to ask in this context would be:
How satisfied are you with our training?
F Very Satised
F Somewhat Satised
F Neither Satised nor Dissatised
F Somewhat Dissatised
F Very Dissatised
33
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 4: Gathering Credible Evidence
8. Keep your answer choices balanced. Using answer choices that lean a certain way can result in
respondents providing inauthentic feedback. Let’s revisit the prior example. Here’s how a set of
unbalanced answer choices (that lean towards being too positive) can look for that question:
F Very Satised
F Somewhat Satised
F Neither Satised nor Dissatised
And here is how they would look once balanced:
F Very Satised
F Somewhat Satised
F Neither Satised nor Dissatised
F Somewhat Dissatised
F Very Dissatised
9. Stay away from asking double-barreled questions. Double-barreled questions ask for feedback
on two separate things within a single question.
Here’s an example: “How would you rate the quality of our training and brochures?”
How would the respondent answer this question? Would they address the quality of the
training? The quality of the brochures that you developed? Maybe they’d skip the question out of
confusion. Also, how would you interpret their answer? Were they responding about the training,
the brochures, or both? You can x a double-barreled question by either choosing one thing to
ask or by breaking the question up into two separate questions.
10. Preview and pilot test your survey before you use it. Imagine using your survey only to realize
that you forgot to add a question. Or that you didn’t include a few essential answer choices for
one of the questions you asked. In either case, you’ll probably end up being frustrated and get
results that fall short of what you need. To prevent any mishaps in your survey design, preview
your survey and share it with others so they can catch any mistakes you might not nd on your
own. Even better, pilot test your survey instruments with a small group of participants similar to
the group of focus to assess its clarity, comprehensibility, and cultural appropriateness.
TIP
When developing a new survey, always look for existing questions or instruments
that have been previously tested and can be adapted or used as-is!
34
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 4: Gathering Credible Evidence
2. KEY INFORMANT INTERVIEWS
Key informant interviews are a qualitative data collection method that, like open-ended questions
in a survey, allow you to obtain an individual’s response in their own words. Interviews differ from
surveys in that they elicit more detailed qualitative data and allow you to interact with the person
to better understand their response. Interviews may be conducted in-person or over the phone.
Interviewing is useful when you want more in-depth information about a person’s knowledge,
attitudes/beliefs, or behaviors. Key informant interviews can be useful for Tribal Trafc Safety
Programs looking to gather in-depth information from a select number and type of individuals, such
as tribal leaders, program administrators, law enforcement, partnering agencies, etc.
3. FOCUS GROUPS
Like interviews, focus groups allows you to collect qualitative data. However, unlike interviews, in
which data are collected by one-on-one interactions, focus groups provide data about a particular
topic through small group discussions. Focus groups are an excellent method for obtaining opinions
about programs and services. They produce information from many people in a short period of time,
so they can be an effective method when information is needed quickly. A facilitator guides the group
based on a predetermined set of topics. The facilitator creates an environment that encourages
participants to share their perceptions and points of view. Focus groups typically are conducted
with participants who share common traits (e.g., gender, adolescents, parents, people who have
participated in your project activities, people who have not participated in your project activities).
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 4: Gathering Credible Evidence
Develop a guide with key themes or
questions that you would like to explore in
each interview or focus group. This will be
used by the facilitator or interviewer, not
shared with the participants.
Keep the guide to no more than 10 key
themes or questions.
List subthemes or probes under each
theme or question. You can use these
probes if the topics you want to cover
do not emerge on their own during the
interview or focus group.
Audio record each session (with permission
from participants) to make sure that you
capture everything that was said.
Take notes even if you are recording so that
you have a backup.
Aim to keep interviews to no more than one
hour in length, and focus groups under 1.5
hours.
Have at least two people assist with the
focus group if possible—one to moderate
the session and the other to take notes
and handle logistics.
TIPS FOR CONDUCTING KEY INFORMANT INTERVIEWS AND FOCUS GROUPS
TIP
4. OBSERVATION
Observation is a way of gathering data by watching behavior and events and noting physical
characteristics in their natural setting. Observation allows you to gather information about a program
as the program’s activities occur. It can collect quantitative data, qualitative data, or a mix of both.
Examples could be observing services being provided, training sessions, meetings, community
member behaviors (e.g., seat belt/car seat use), the environment (e.g., trafc patterns), etc. It is
best for observation to be done in an unobtrusive manner to ensure that you will witness what really
takes place. Some types of observation that are useful for evaluating Tribal Trafc Safety Programs
include:
a. Seat belt/car seat checks These observational activities use a formal protocol to nd out
how many people in your community are using seat belts or car seats.
b. Windshield/walking survey Windshield surveys are systematic observations made from
a moving vehicle. Walking surveys are systematic observations made on foot. Either can
help you better understand specic conditions that your Tribal Trafc Safety Program might
be trying to impact such as trafc patterns, road conditions, signage, road safety, and
infrastructure needs.
Prior to conducting an observation, always develop a checklist of what you
plan to observe so if you become distracted, you won’t forget what to look for!
35
•
•
•
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 4: Gathering Credible Evidence
5. DOCUMENT REVIEW
Document review might also be helpful when evaluating your Tribal Trafc Safety Program. You can
review meeting minutes, sign-in sheets, progress reports, medical records, logs, maps, laws, and
policies to learn more about the activities of your program and its reach and impact.
To ensure that you have plenty of documents to review, be sure to collect data at each of your Tribal
Trafc Safety Program activities and events, including sign-in sheets, meeting minutes, satisfaction
surveys, etc. It’s also a good idea to debrief with your staff shortly after each major activity to
discuss what worked well and what needs improvement for the future. Try using the plus/delta
technique, an easy-to-use feedback tool.
TIPS FOR DOCUMENT REVIEW OF POLICIES/LAWS
Rather than just evaluating whether or not specic laws and policies exist in your community, it is
helpful to review the specic elements of these laws and policies to determine their alignment with
proven, evidence-based practices. The list below shows some examples of criteria that can be used
to evaluatie laws and policies.
DOCUMENTING LAWS FOR IMPAIRED DRIVING
F Is there a law? Is it a Tribal law, or does it defer to state law?
F How is the term “impaired” dened (that is, alcohol, drugs, prescription medication)?
F Does the law describe how “impairment” must be measured, and if so, are those measurements
different from state laws and best practices?
F What are the consequences for rst-time offenders? To what degree do those consequences
differ from state laws and best practices?
F What are the consequences for repeat offenders? To what degree do those consequences differ
from state laws and best practices?
F Does the law include a “child endangerment” clause if children are in the vehicle at the time of
arrest?
F Does the law include “court diversion” or alternate sentencing options, and if so, who manages
those programs?
F Does the law stipulate how funds generated from nes or court fees are used (that is, do those
funds support on-going trafc safety efforts)?
36
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 4: Gathering Credible Evidence
DOCUMENTING LAWS FOR SEAT BELT AND CHILD SAFETY SEAT USE
F Is there a law? Is it a Tribal law, or does it defer to state law?
F Is the law a primary or secondary enforcement law?
F Who is covered by the seat belt law (i.e., front seat and rear seat occupants, or front-seat only)?
F What is covered by the child safety seat law (i.e., ages, location of seat, and/or types of seats)?
F What are the nes or other penalties for not using a restraint and how do they compare to those
at the state level and best practices? Do the nes differ for non-use of a child safety seat versus
improper use of a child safety seat?
F Does the law describe increased consequences for repeat offenders?
F Does the law include a “court diversion” program and if so, who manages the court diversion
program?
F Does the law stipulate how funds generated from citations or court fees are used (that is, do
those funds support on-going trafc safety efforts)?
37
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 4: Gathering Credible Evidence
PROS & CONS OF DATA COLLECTION/SOURCE TYPES
DATA COLLECTION/
SOURCE TYPE
PROS CONS
Surveys
Standardized questionnaires
that ask predetermined
questions
• Easy to do with a large number of people
• Easy to conduct analysis
• Easy to repeat at a later time (pre-post)
• Allows control over the content that is
being measured
• Can be highly accurate
• Allows for comparisons with other/larger
populations when survey items come
from existing instruments
• Little opportunity to explore issues in
depth
• Low response rates are common
• Time-consuming if you need a large
number of participants
• Relatively high cost
• Relatively slow to design, implement,
and analyze
• Accuracy depends on who and how many
people participate
Key Informant
• Possible to explore issues in depth
• Interviewer’s presence and
Interviews
• Personalized approach
• Low cost (assuming relatively few)
characteristics might bias results
• Small sample; not generalizable
Structured or unstructured
one-on-one directed
conversations with key
• Allows respondents dene what is
important and in their own words
• Can clarify responses through probes
• Can be time consuming to set up
interviews with busy informants
• Might be difcult to analyze and
summarize ndings due to large
individuals or leaders in a
• May lead you to other data sources and
volume of transcripts and diverse
community
other key informants
perspectives
Focus Groups
Structured interviews with
small groups of like individuals
using standardized questions,
follow-up questions, and
exploration of other topics that
arise to better understand
participants
• Less expensive
• Rapid data collection
• Faster than one-on-one interviews
• Can clarify participant responses through
probing (follow-up questions)
• Often generate new/fresh ideas
• Allow respondents dene what is
important and in their own words
• Requires a skilled facilitator to keep
conversation owing and everyone
involved
• Can be time consuming to assemble
groups
• Less control over process than key
informant interviews
• Difcult to collect sensitive information
• Limited generalizability
• Might be difcult to analyze and
summarize ndings
38
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 4: Gathering Credible Evidence
DATA COLLECTION/
SOURCE TYPE
PROS CONS
Observations
• Natural setting
• Does not rely on people’s willingness
or ability to provide information
• Observer might bias behavior of
participants
• Difcult to generalize observations to
Watching behaviors or
• Allows you to directly see what people
the larger populations
events or noting physical
do rather than relying on what people
• Does not increase your understanding
characteristics
say they did
of why people behave as they do
Document Review
• Low cost
• Might be difcult to access
• Might not be current or updated
Reviewing documents, records,
• Relatively quick
• Often allows for historical
• Data are limited to what has already
been collected
or archives held by your
comparisons or trends
• Might not answer your specic
project or partnering agencies
evaluation questions
Gathering
Secondary Data
• Less expensive
• Less time consuming
• Might be immediately available for use
• Data are limited to what has already
been collected
• Might not answer your specic
evaluation questions
• Data sharing limitations
• Data might not be tribe-specic
39
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 4: Gathering Credible Evidence
SELECT SOURCES OF EVIDENCE FOR TRIBAL TRAFFIC SAFETY PROGRAM EVALUATION
Who might you survey or interview?
Program participants (or nonparticipants)
Staff, program managers, administrators
General public
Partner agency staff
Tribal leaders
What might you observe?
Coalition meetings
Educational events/trainings
Car seat checks/installations
Seat belt use
Signage
Road conditions
Trafc patterns
Which documents might you analyze?
Meeting minutes
Sign-in sheets
Event and activity logs
Previous evaluation reports
Road maps
Photographs
News articles
Medical records or other les
Law enforcement logs
Tribal policies and protocols
Progress reports
USING MIXED METHODS APPROACHES FOR PROGRAM EVALUATION
A mixed methods approach combines at least one qualitative and one quantitative data
source or collection method for program evaluation.
Selecting multiple data sources provides an opportunity to include different perspectives
regarding the program and might enhance your evaluation’s credibility. Sometimes a single
method is not sufcient to accurately measure an activity or outcome because the thing
being measured is complex. Using multiple methods helps increase the accuracy of the
measurement and the certainty of your conclusions when the various methods yield similar
results. Combining methods maximizes the strengths and minimizes the limitations of
each method, which will provide you and your partners increased condence in the overall
ndings.
40
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
41
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 5: Justifying Conclusions
STEP 5: JUSTIFYING CONCLUSIONS
This step includes:
Analyzing the evidence gathered in step 4
Making claims about your Tribal Trafc Safety Program based on the analysis
Justifying claims by comparing the evidence against standards selected by your program
Making recommendations for continuing, expanding, modifying, or terminating the program
ANALYZYING EVIDENCE
Data analysis is the process of evaluating data using analytical and statistical tools to discover
useful information and aid in decision making. There are several data analysis methods, most of
which can be categorized as quantitative or qualitative data analysis.
QUANTITATIVE DATA ANALYSIS: KEY STEPS
Enter the data into a database and check for errors. If you are using secondary data such as BRFSS
or PRAMS, the data have already been checked, entered, and tabulated by those conducting the
survey. Select the computer program you will use and determine who will enter and analyze the data,
and determine who will enter, check, tabulate, and analyze the data.
Tabulate the data to provide information (such as a number or percentage) for each indicator. Some
basic calculations include determining:
The total number of participants, trainings, The number of times a certain value
activities, etc. appears (frequency)
The number of participants achieving the The difference between the highest score
desired outcome and lowest score observed (range)
The percentage of participants achieving the The lowest/smallest score in a data set
desired outcome (min.)
The percent increase (or decrease) observed The highest/largest score in a data set
The average (mean) score
(max.)
For more specic information about analyzing quantitative evaluation data click here or check out
the additional resources listed in the appendix.
•
•
•
•
•
•
•
•
•
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 5: Justifying Conclusions
QUALITATIVE DATA ANALYSIS: KEY STEPS
Transcribe your interviews or focus groups from your notes and recordings.
Become familiar with the data. It is important to read and re-read the data, writing down
impressions, looking for meaning and determining which pieces of data have value.
Focus the analysis. In this step, it is important to identify key questions that you want to answer
through the analysis. One approach would be to focus the analysis on the answers to a particular
question or topic. Another way to focus the analysis would be to examine the data as it relates to a
particular group (e.g., elders, teens, women, etc.).
Categorize the data and create a framework. This is often referred to as coding or indexing the
data. Begin by identifying ideas, concepts, behaviors, key words, and phrases that frequently emerge
in your data. A code is then assigned to those pieces of data in an effort to label the data and make
it easier to organize and retrieve. A coding plan helps to provide a framework that will structure, label
and, dene the data.
Identify patterns and make connections. Review codes and patterns of codes to identify themes
and relationships between themes.
Interpret the data and explain findings. After themes, patterns, connections, and relationships
are identied, attach meaning and signicance to the data. It can be helpful in this process to
develop lists of key ideas, create diagrams, or use models to explain the ndings. It is important to
remember that qualitative data are not suited to make generalizations across a population.
For more specic information about analyzing qualitative evaluation data, click here or check out the
additional resources listed in the appendix.
TRIANGULATION
Triangulation involves combining different sources of information in your evaluation to reach
a larger understanding of the impact of your Tribal Trafc Safety Program. These different
sources might include:
Quantitative data (surveys)
Qualitative data (focus groups and interviews)
Primary data (data that you collect)
Secondary data (existing data that you use)
One of the main benets of triangulation is that it uses a range of data to provide more
comprehensive insight into your program and that is particularly useful as a way to validate
and strengthen your evaluation ndings.
42
•
•
•
•
43
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 5: Justifying Conclusions
STANDARDS
Once you have analyzed your evaluation data, the next step is to interpret the ndings to make
claims and judgements about your Tribal Trafc Safety Program. This step is accomplished by
comparing the evidence against standards selected by your program. Standards establish a
comparison by which your program can be judged and be considered successful. The box below lists
possible standards you might use for determining the progress and/or success of your program:
Needs of the participants/community
Change in performance since baseline
Program mission
Professional or existing standards (e.g.,
Program SMART objectives
Healthy People 2030, Countermeasures
that Work: A Highway Safety
Performance by previous or similar
Countermeasure Guide for State Highway
programs
Safety Offices, The Guide to Community
Performance by a comparison group
Preventive Services)
Targets or xed criteria of performance
Mandates, policies, statutes, regulations,
Change in performance over time
laws
Your standards for performance and outcomes should be determined at the beginning of your
program, based on achievable targets, and aligned with you goals and SMART objectives. It is also
important to note that multiple standards can be applied to a given program.
INTERPRETING THE FINDINGS AND MAKING JUDGEMENTS AND RECOMMENDATIONS
Interpretation is the effort of guring out what your ndings mean about your Tribal Trafc Safety
Program. Evaluation evidence must be interpreted to determine the practical signicance of what
has been learned. Judgements are statements concerning the merit, worth, or signicance of the
program. They are formed by comparing the ndings and interpretations regarding your Tribal Trafc
Safety Program against one or more of your selected standards. Recommendations are actions for
consideration resulting from the evaluation. Sharing draft recommendations, soliciting reactions from
multiple partners, and presenting options instead of directive advice will increase the likelihood that
recommendations are relevant and well-received.
•
•
•
•
•
•
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 5: Justifying Conclusions
Interpret evaluation results with the goals
of your program in mind.
Keep your audience in mind. What do they
need and want to know?
Consider the limitations of the evaluation.
Consider the possible biases of the
individuals conducting the evaluation.
Consider alternative explanations for your
results.
Compare program outcomes with those of
previous years.
Compare actual outcomes with intended
outcomes.
Explore how your results compare with
those of similar programs.
Check to see if the different data collection
methods used to measure your progress
show similar results.
Evaluate whether or not your results
are similar to what you expected. If not,
consider why they may be different.
Keep all recommendations relevant and
feasible.
TIPS TO REMEMBER WHEN INTERPRETING YOUR FINDINGS
AND MAKING JUDGEMENTS & RECOMMENDATIONS
TIP
44
•
•
•
•
•
•
•
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 6: Ensuring Use and Sharing Lessons Learned
STEP 6: ENSURING USE AND SHARING LESSONS LEARNED
Lessons learned in the course of an evaluation do not automatically translate into informed decision-
making and appropriate action. This nal step is therefore critical to ensure that the evaluation
processes and ndings are used and disseminated appropriately.
REMEMBER YOUR PURPOSE!
The ultimate purpose of program evaluation is to use the information to improve your Tribal Trafc
Safety Programs to maximize its benets for the community. The purpose(s) you identied early in
the evaluation process should guide the use of the evaluation results. The evaluation results should
be used to demonstrate the effectiveness of your program, identify ways to improve your program,
modify program planning, demonstrate accountability, and justify funding.
ADDITIONAL PURPOSES OF PROGRAM EVALUATION
Demonstrate to legislators or other
partners that resources are being well
spent and that the program is effective
Aid in forming budgets and to justify the
allocation of resources
Compare outcomes with those of
previous years
Compare actual outcomes with intended
outcomes
Suggest realistic intended outcomes
Support annual and long-range planning
Focus attention on issues important
to your program
Ppromote your program
Identify partners for collaboration
Enhance the image of your program
Retain or increase funding
Provide direction for program staff
Identify training and technical assistance
needs
Document the level of success in
achieving objectives
Identify areas of the program that need
improvement
Mobilize community support
Redistribute or expand the locations
where the intervention is carried out
Focus program resources on a specic
population
45
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 6: Ensuring Use and Sharing Lessons Learned
WHO WILL USE THE EVALUATION RESULTS?
Many different people and organizations will be interested in your evaluation ndings, including:
Partners – Remember your work in step 1 to identify those people or organizations that
are invested in your Tribal Trafc Safety Program or that are interested in the results of the
evaluation. Although each Tribal Trafc Safety Program is unique, some common partners might
include, but are not limited to: tribal leaders, schools, head start, day care centers, Community
Health Representatives (CHR), WIC, transportation departments, law enforcement, I/T/U health
facilities, EMTs, environmental health, community members, BIA, IHS, and state and county
partners.
Funders – Most funding agencies want to know how their investment in your program is going.
They might also use your evaluation ndings to assess accountability, improve their grantmaking,
disseminate successful programs to other grantees, increase the state of knowledge, assess
the quality or impact of grants, and plan new programs.
You – In addition to these important partners, you and your Tribal Trafc Safety Program
team should use your evaluation ndings to celebrate successes, troubleshoot challenges or
unintended outcomes, and continuously improve the quality of your program.
CONTINUOUS QUALITY IMPROVEMENT (CQI)
Continuous Quality Improvement (CQI) is the process of using your process and outcome
evaluation data to determine what worked well, where there is room for improvement, and
what changes may be needed. CQI takes advantage of what you have learned over time
from your evaluation to improve the program for the future without starting over from the
beginning. It helps all staff involved to keep your program fresh and a good t for your
participants and your community. CQI should be performed throughout the life of your
program to assess what went well and what should be improved. You will continually use
the results from your evaluation to determine whether you met your goal and desired
short-, intermediate-, and long-term outcomes. The results will prepare you to decide
whether and how to revise your goals and desired outcomes, reassess t and capacity,
and revise your work plan. The process then repeats itself over time to assess whether the
changes you make prove to be effective.
46
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 6: Ensuring Use and Sharing Lessons Learned
DEVELOPING AN EVALUATION REPORT:
Your evaluation report should clearly, succinctly, and impartially communicate all parts of the
evaluation. The report should be written so that it is easy to understand by your audience. It need
not be lengthy or technical.
Common Evaluation Report Outline:
Executive Summary a short summary of the
report that can be shared separately or as
part of the full evaluation report
Background and Purpose
• Program background
• Program description
• Key evaluation questions/focus
Evaluation Methods
• Design
• Sampling procedures
• Measures or indicators
• Data collection procedures
• Analysis
• Limitations
Results
Discussion and Recommendations
TIPS FOR WRITING YOUR EVALUATION REPORT
Tailor the report to your audience; you may
need a different version of your report for
each segment of your audience.
Present clear and succinct results.
Include a summary.
Describe essential features of the program
(e.g., logic models).
Explain the focus of the evaluation and its
limitations.
Summarize the evaluation plan and
procedures.
List the strengths and weaknesses of the
evaluation.
Organize the report logically and include
appropriate details.
Discuss recommendations for action with
their advantages, disadvantages, and
resource implications.
Verify that the report is unbiased and
accurate.
Remove technical jargon.
Use examples, illustrations, graphics, and
stories.
Prepare and distribute reports on time.
Distribute reports to as many partners as
possible
Acknowledge partners and participants.
47
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
STEP 6: Ensuring Use and Sharing Lessons Learned
DISSEMINATION: SHARING RESULTS AND LESSONS LEARNED FROM EVALUATION
Dissemination involves communicating evaluation ndings or lessons learned to relevant audiences
in a timely, unbiased, and consistent manner. Disseminating the evaluation ndings for your Tribal
Trafc Safety Program is a critical step in the overall process that is sometimes overlooked. To avoid
making this common mistake, work with your team to create a dissemination plan that maps the
following essential details:
Who is the key audience?
What approaches and formats will be used to disseminate ndings—hard copy print, electronic,
oral presentations, briengs, mailings, etc.?
How, where, and when will ndings be used?
Who is responsible for dissemination?
What resources are available to accomplish the work?
What are the follow-up activities to ensure that the evaluation ndings are used?
How will follow-up activities be monitored?
SOME METHODS FOR GETTING YOUR EVALUATION INFORMATION
TO YOUR AUDIENCE INCLUDE:
ORAL PRESENTATIONS –
Tribal councils, partners
meetings, community
groups, etc.
MAILINGS
WEB SITES
COMMUNITY FORUMS
MEDIA –
television, radio, newspaper
PERSONAL CONTACTS
EMAIL LISTSERVS
ORGANIZATIONAL
NEWSLETTERS
48
•
•
•
•
•
•
•
49
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Appendix A
APPENDIX A: STANDARDS FOR EFFECTIVE EVALUATION
The steps in the CDC Evaluation Framework used throughout this guide are informed by a set
of standards for evaluation. The standards provide practical guidelines to follow when having to
decide among evaluation options at each step in the framework. The steps and standards are used
together throughout the evaluation process. For each step, a subset of relevant standards should be
considered. Furthermore, the standards can be applied while planning an evaluation and throughout
its implementation.
The four categories discussed below include a total of 30 specic standards.
STANDARD 1: UTILITY
Utility standards ensure that information needs of evaluation users are satised. Seven utility
standards address such items as identifying those who will be impacted by the evaluation, the
amount and type of information collected, the values used in interpreting evaluation ndings, and the
clarity and timeliness of evaluation reports. The following utility standards ensure that an evaluation
will serve the information needs of intended users:
• Partner identification. Persons involved in or affected by the evaluation should be identied so
that their needs can be addressed.
• Evaluator credibility. The persons conducting the evaluation should be trustworthy and
competent in performing the evaluation for ndings to gain maximum credibility and acceptance.
• Information scope and selection. Information collected should address pertinent questions
regarding the program and be responsive to the needs and interests of clients and other
specied partners.
• Values identification. The perspectives, procedures, and rationale used to interpret the ndings
should be carefully described so that the bases for value judgments are clear.
• Report clarity. Evaluation reports should clearly describe the program being evaluated, including
its context and the purposes, procedures, and ndings of the evaluation so that essential
information is provided and easily understood.
• Report timeliness and dissemination. Substantial interim ndings and evaluation reports should
be disseminated to intended users so that they can be used in a timely fashion.
• Evaluation impact. Evaluations should be planned, conducted, and reported in ways that
encourage follow-through by partners to increase the likelihood of the evaluation being used.
50
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Appendix A
STANDARD 2: FEASIBILITY
Feasibility standards ensure that the evaluation is viable and pragmatic. The three feasibility
standards emphasize that the evaluation should employ practical, nondisruptive procedures; that the
differing political interests of those involved should be anticipated and acknowledged; and that the
use of resources in conducting the evaluation should be prudent and produce valuable ndings. The
following feasibility standards ensure that an evaluation will be realistic, prudent, diplomatic, and
frugal:
• Practical procedures. Evaluation procedures should be practical so that you can keep disruption
of the program to a minimum while still being able to obtain needed information.
• Political viability. During planning and while the evaluation is being conducted, consideration
should be given to the varied positions of interest groups so that their cooperation can be
obtained and possible attempts by any group to curtail evaluation operations or to bias or
misapply the results can be averted or counteracted.
• Cost-effectiveness. The evaluation should be efcient and produce valuable information to justify
expended resources.
STANDARD 3: PROPRIETY
Propriety standards ensure that the evaluation is ethical (i.e., conducted with regard for the rights
and interests of those involved and affected). Eight propriety standards address such items as
developing protocols and other agreements for guiding the evaluation; protecting the welfare
of human subjects; weighing and disclosing ndings in a complete and balanced fashion; and
addressing any conicts of interest in an open and fair manner. The following propriety standards
ensure that an evaluation will be conducted legally, ethically, and with regard for the welfare of those
involved in the evaluation as well as those affected by its results:
• Service orientation. The evaluation should be designed to assist organizations in addressing
and serving effectively the needs of the participants.
• Formal agreements. All principal parties involved in an evaluation should agree in writing to their
obligations (i.e., what is to be done, how, by whom, and when) so that each must adhere to the
conditions of the agreement or renegotiate it.
• Rights of human subjects. The evaluation should be designed and conducted in a manner that
respects and protects the rights and welfare of human subjects.
• Human interactions. Evaluators should interact respectfully with other persons associated with
an evaluation, so that participants are not threatened or harmed.
• Complete and fair assessment. The evaluation should be complete and fair in its examination
and recording of strengths and weaknesses of the program so that strengths can be enhanced
and problem areas addressed.
51
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Appendix A
• Disclosure of findings. The principal parties to an evaluation should ensure that the full
evaluation ndings with pertinent limitations are made accessible to the persons affected by the
evaluation and any others with expressed legal rights to receive the results.
• Conflict of interest. Conict of interest should be handled openly and honestly so that the
evaluation processes and results are not compromised.
• Fiscal responsibility. The evaluator’s allocation and expenditure of resources should be prudent,
ethically responsible, and follow sound accounting procedures.
STANDARD 4: ACCURACY
Accuracy standards ensure that the evaluation produces ndings that are considered correct. Twelve
accuracy standards include such items as describing the program and its context; articulating in
detail the purpose and methods of the evaluation; employing systematic procedures to gather valid
and reliable information; applying appropriate qualitative or quantitative methods during analysis and
synthesis; and producing impartial reports containing conclusions that are justied. The following
accuracy standards ensure that an evaluation will convey technically adequate information regarding
the determining features of merit of the program:
• Program documentation. The program being evaluated should be documented clearly and
accurately.
• Context analysis. The context in which the program exists should be examined in enough detail
to identify probable inuences on the program.
• Described purposes and procedures. The purposes and procedures of the evaluation should be
monitored and described in enough detail to identify and assess them.
• Defensible information sources. Sources of information used in a program evaluation should be
described in enough detail to assess the adequacy of the information.
• Valid information. Information-gathering procedures should be developed and implemented to
ensure a valid interpretation for the intended use.
• Reliable information. Information-gathering procedures should be developed and implemented to
ensure sufciently reliable information for the intended use.
• Systematic information. Information collected, processed, and reported in an evaluation should
be systematically reviewed and any errors corrected.
• Analysis of quantitative information. Quantitative information should be analyzed appropriately
and systematically so that evaluation questions are answered effectively.
• Analysis of qualitative information. Qualitative information should be analyzed appropriately and
systematically to answer evaluation questions effectively.
• Justified conclusions. Conclusions reached should be explicitly justied for partners’
assessment.
52
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Appendix A
• Impartial reporting. Reporting procedures should guard against the distortion caused by
personal feelings and biases of any party involved in the evaluation to reect the ndings fairly.
• Meta-evaluation. The evaluation should be formatively and summatively evaluated against these
and other pertinent standards to guide its conduct appropriately and, on completion, to enable
close examination of its strengths and weaknesses by partners.
Source: Joint Committee on Standards for Educational Evaluation. Program evaluation standards: how
to assess evaluations of educational programs. 2nd ed. Thousand Oaks, CA: Sage Publications, 1994.
53
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Appendix B
APPENDIX B - ADDITIONAL RESOURCES
Selected Web-based Resources on Program Evaluation
Centers for Disease Control and Prevention: www.cdc.gov/eval/
Community Tool Box, University of Kansas: ctb.ku.edu/
RAND Corporation – Getting to Outcomes:
www.rand.org/pubs/tools/TL259/introduction.html
University of Wisconsin Cooperative Extension: Evaluation Resources:
www.uwex.edu/ces/pdande/
W.K. Kellogg Foundation:
www.wkkf.org/resource-directory/resource/2017/11/wk-kellogg-foundation-step-by-step-guide-
to-evaluation
Selected Publications on Program Evaluation
Connell JP, Kubisch AC, Schorr LB, Weiss, CH. New approaches to evaluating community initiatives.
New York, NY: Aspen Institute, 1995.
Fawcett SB, Paine-Andrews A, Francisco VT, Schulz J, Ritchter KP, et al. Evaluating community
initiatives for health and development. In: Rootman I, Goodstadt M, Hyndman B, et al., eds.
Evaluating Health Promotion Approaches. Copenhagen, Denmark: World Health Organization (Euro),
1999.
Fawcett SB, Sterling TD, Paine Andrews A, Harris KJ, Francisco VT, et al. Evaluating community
efforts to prevent cardiovascular diseases. Atlanta, GA: Centers for Disease Control and Prevention,
National Center for Chronic Disease Prevention and Health Promotion, 1995.
Fetterman DM, Kaftarian SJ, Wandersman A. Empowerment evaluation: Knowledge and tools for self-
assessment and accountability. Thousand Oaks, CA: Sage Publications, 1996.
54
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Appendix B
Fetterman DM, Kaftarian SJ, Wandersman A. Empowerment evaluation principles in practice. New
York: Guilford, 2005.
Joint Committee on Standards for Educational Evaluation. Program evaluation standards: how to
assess evaluations of educational programs. 2nd ed. Thousand Oaks, CA: Sage Publications, 1994.
Patton MQ. Utilization-focused evaluation. Thousand Oaks, CA: Sage Publications, 1997.
Rossi PH, Freeman HE, Lipsey MW. Evaluation: A systematic approach. Newbury Park, CA: Sage
Publications, 1999.
55
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Appendix Glossary
APPENDIX GLOSSARY
Accountability: The responsibility of program managers and staff to provide evidence to partners
and funding agencies that a program is effective and in conformance with its coverage, service,
legal, and scal requirements.
Accuracy: The extent to which an evaluation is truthful or valid in what it says about a program,
project, or material.
Activities: The actual events or actions that take place as a part of the program using inputs, such
as funds, technical assistance and other types of resources to produce specic outputs.
Baseline Data: The initial information collected about the condition or performance of subjects
prior to the implementation of an intervention, against which progress can be compared at strategic
points during and at completion of the program.
Bias: A point of view that inhibits objectivity.
Coding: To translate a given set of data or items into descriptive or analytic categories to be used
for data labeling and retrieval.
Comparison group: A group not exposed to a program or treatment. Also referred to as a control
group.
Comprehensiveness: Full breadth and depth of coverage on the evaluation issues of interest.
Continuous Quality Improvement (CQI): An ongoing process that evaluates how a program works
and ways to improve its processes.
Control group: In quasi-experimental designs, a group of subjects who receive all inuences
except the program in exactly the same fashion as the treatment group (the latter called, in some
circumstances, the experimental or program group). Also referred to as a non-program group.
Data Material: Gathered during the course of an evaluation which serves as the basis for
information, discussion, and inference.
Data collection method: The way facts about a program and its outcomes are amassed. Data
collection methods often used in program evaluations include literature search, le review, natural
observations, surveys, expert opinion, and case studies.
Descriptive statistical analysis: Numbers and tabulations used to summarize and present
quantitative information concisely.
Dissemination: The communication of the actions--by written, oral, and/or audio-visual reporting--of
evaluators to foster knowledge of the evaluation ndings among all right-to-know audiences.
56
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Appendix Glossary
Evaluation design: The logical model or conceptual framework used to arrive at conclusions about
outcomes.
Evaluation plan: A written document describing the overall approach or design that will be used to
guide an evaluation. It includes what will be done, how it will be done, who will do it, when it will be
done, why the evaluation is being conducted, and how the ndings will likely be used.
Executive summary: A nontechnical summary statement designed to provide a quick overview of the
full-length report on which it is based.
Focus group: A group of people selected for their relevance to an evaluation that is engaged by a
trained facilitator in a series of discussions designed for sharing insights, ideas, and observations
on a topic of concern.
Indicator: A specic, observable, and measurable characteristic or change that shows the progress
a program is making toward achieving a specied outcome.
Inputs: Resources that go into a program in order to perform the activities successfully.
Interviews: Guided conversations between a skilled interviewer and an interviewee that seek to
maximize opportunities for the expression of a respondent’s feelings and ideas through the use of
open-ended questions and a loosely structured interview guide.
Instrument: Device that assists evaluators in collecting data in an organized fashion, such as a
standardized survey or interview protocol; a data collection device.
Key informant: Person with background, knowledge, or special skills relevant to topics examined by
the evaluation; sometimes an informal leader or spokesperson in the population of focus.
Logic model: A systematic and visual way to present the perceived relationships among the
resources you have to operate the program, the activities you plan to do, and the changes or results
you hope to achieve.
Mixed-method evaluation: An evaluation design that includes the use of both quantitative and
qualitative methods for data collection and data analysis.
Natural observation: A data collection method that involves on-site visits to locations where a
program is operating. It directly assesses the setting of a program, its activities, and individuals who
participate in the activities.
Objective: A specic, measurable description of an intended outcome.
Objectivity: The expectation that data collection, analysis and interpretation will adhere to standards
that eliminate or minimize bias; objectivity insures that outcome or evaluation results are free from
the inuence of personal preferences or loyalties.
57
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Appendix Glossary
Outcome evaluation: The systematic collection of information to assess the impact of a program,
present conclusions about the merit or worth of a program, and make recommendations about
future program direction or improvement.
Outcomes: Changes in attitudes, values, behaviors or conditions between baseline measurement
and subsequent points of measurement. Changes can be immediate, intermediate or long-term; the
results/effects expected by implementing the program’s strategies.
Outputs: The direct products of program activities; immediate measures of what the program did.
Partners: People or organizations who have a direct or indirect interest in the development
intervention or its evaluation and/or what will be done with results of the evaluation.
Population: All the persons in a particular group.
Pre/post tests: Standardized methods used to assess change in subjects’ knowledge and capacity
to apply this knowledge to new situations. The tests are administered prior to implementation of the
program and after completion of the program’s activities. Determines performance prior to and after
the delivery of an activity or strategy.
Primary data: Data collected by an evaluation team specically for the evaluation study.
Process evaluation: The systematic collection of information to document and assess how a
program was implemented and operates.
Program evaluation: The systematic collection of information about the activities, characteristics,
and outcomes of programs to make judgments about the program, improve program effectiveness,
and/or inform decisions about future program development.
Program goal: A statement of the overall mission or purpose(s) of the program.
Qualitative data: Non-numerical data rich in detail and description that are usually presented in a
textual or narrative format, such as data from case studies, focus groups, interviews or document
reviews. Used with open-ended questions, these data have the ability to illuminate evaluation
ndings derived from quantitative methods.
Quantitative data: Numeric information, focusing on things that can be counted, scored and
categorized; used with close-ended questions, where participants have a limited set of possible
answers to a question. Quantitative data analysis utilizes statistical methods.
Questionnaire: Highly structured series of written questions that is administered to a large number
of people; questions have a limited set of possible responses.
Random assignment: A process by which the people in a sample to be tested are chosen at random
from a larger population; a pool of eligible evaluation participants are selected on a random basis.
58
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Appendix Glossary
Reliability: The extent to which a measurement, when repeatedly applied to a given situation
consistently produces the same results if the situation does not change between the applications.
Reliability can refer to the stability of the measurement over time or to the consistency of the
measurement from place to place.
Representative sample: A segment or group taken from a larger body or population that mirrors in
composition the characteristics of the larger body or population.
Resources: Assets available and anticipated for operations. They include people, equipment,
facilities, and other things used to plan, implement, and evaluate programs.
Sample: A segment of a larger body or population.
Sample size: The number of units to be sampled.
Secondary data: Data collected and recorded by another (usually earlier) person or organization,
usually for different purposes than the current evaluation.
Standard: A principle commonly agreed upon by experts in the conduct and use of an evaluation for
the measure of the value or quality of an evaluation (e.g., accuracy, feasibility, propriety, utility).
Statistical analysis: The manipulation of numerical or categorical data to predict phenomena, to
draw conclusions about relationships among variables or to generalize results.
Subjectivity: Subjectivity exists when the phenomena of interest are described or interpreted in
personal terms related to one’s attitudes, beliefs or opinions.
Surveys: A data collection method that involves a planned effort to collect needed data from a
sample (or a complete census) of the relevant population. Uses structured questions from specially
designed instruments to collect data about the feelings, attitudes, knowledge, and/or behaviors
of individuals. The relevant population consists of people or entities affected by the program (or of
similar people or entities).
Triangulation: Using multiple data sources independent of one another to help verify the validity of
evaluation ndings so results are not informed (and possibly skewed) by only one or two sources.
Having more sources helps ensure ndings are replicable and unbiased.
Validity: The degree of accuracy of a measurement. For survey instruments, validity refers to the
degree to which the instrument measures what it is intended to measure.
59
Tribal Motor Vehicle Injury Prevention Program Evaluation Guide
Acknowledgements
ACKNOWLEDGEMENTS
The contents of this guide were developed and adapted from several existing sources including:
Centers for Disease Control and Prevention
CDC Program Performance and Evaluation Office
Framework for Program Evaluation in Public Health
Introduction to Program Evaluation for Public Health Programs: A Self-Study Guide
Tribal Motor Vehicle Injury Prevention Program Best Practices Guide
Community Tool Box, University of Kansas
Joint Committee on Standards for Educational Evaluation
SUGGESTED CITATION:
Albuquerque Area Southwest Tribal Epidemiology Center (2022). Tribal Motor Vehicle Injury
Prevention: Program Evaluation Guide. Albuquerque, NM.
This guide was developed by the Albuquerque Area Southwest Tribal Epidemiology Center via Indian
Health Service Cooperative Agreement Number U1B1IHS0013-12-01, with support from the Centers
for Disease Control and Prevention. The content is solely the responsibility of the authors and does
not necessarily represent the ofcial views of the Indian Health Service or the Centers for Disease
Control and Prevention.
For additional information about the guide please contact the Albuquerque Area Southwest Tribal
Epidemiology Center (AASTEC) at [email protected]
