Contents Introduction Of Special Interest: To Producers To Broadcasters To Funders To Community Leaders Model Campaigns Chapter 1: POV's High Impact TV Chapter 2: Television Race Initiative Chapter 3: Take this Heart Chapter 4: Positive: Life with HIV Perspectives from Partners Chapter 5: For Filmmakers Chapter 6: For Broadcasters Chapter 7: For Nonprofits Chapter 8: For Grantmakers Strategic and Practical Advice Chapter 9: On Media Chapter 10: On Evaluation Case Studies Contact and Resource List

by Ellen Schneider and Melanie Piersol

The purpose of evaluation is to give funders and project directors an assessment of whether and how the goals of the project were met, a determination of what was successful and what wasn’t, and an analysis of why and why not.

 

–Debra Franco, Copperfield Associates, Inc.

Evaluation is a roadmap to the future, pointing out new directions for a project, documenting readiness for expansion, or even outlining how a few small changes might transform an unsuccessful project into a winner.

 

–Fran Cooper, Communication Sciences Group

WHY EVALUATE?

Funders want it. Corporate sponsors demand it. Yet evaluation is the word some nonprofits dread the most. Increasingly it is becoming a key ingredient in media projects, and while it may seem daunting, it can lead to surprising, long-term, positive growth.

At P.O.V., we have always considered ourselves something of a public interest media laboratory and our High Impact Television projects provide an opportunity to experiment and learn valuable lessons about this field. At the end of each campaign, we would routinely gather anecdotal reports and feedback from campaign partners. But still that nagging question remained: had we really been as successful as we thought; and more importantly, why or why not?

Based on our experience with the Getting to The Heart of the Matter campaign and more recently the Television Race Initiative, we propose that there are several good reasons for conducting an evaluation:

1. The evaluation process helps clarify project goals.
2. It documents your successes–even those you do not know about.
3. It uncovers the flaws in a campaign, which can help you to refine future projects.
4. It helps forge stronger relationships with project partners.
5. The lessons learned will inform others working in the field.

BEYOND SUCCESS AND FAILURE

It is important to understand what evaluation is and what it is not. It is not a glossy overview of the successes of a project, nor is it about impressive numbers and statistics–although they can certainly be part of it. And most importantly, evaluation is not a determination of success or failure.

"Most people are afraid that we’re going to tell them they’re doing a bad job, and then they’re going to lose their funding," explained Dr. Irene Goodman, president of Goodman Research Group in Boston. "We see evaluation as helping the client determine what’s working and what’s not, and then helping to make it better."

Certainly funders expect results–but not perfection. Debra Franco of Copperfield Associates, Inc., the group that conducted the
evaluation of Getting to The Heart of the Matter urged, "Mistakes are just as important as successes." Tripp Somerville, program director for The Northwest Area Foundation, concurs and encouraged us to "move away from the judgmental context." He shared the foundation’s approach to evaluation:

"Evaluation provides a critical outside assessment that is just not possible when you are enmeshed in the day-to-day work of the project. It’s about culling the lessons learned from your experience. But if you don’t share these lessons with your colleagues then you’re really doing them a disservice."

So above all, evaluation is a learning tool. But where do you start?

FIRST, DEFINE YOUR PROJECT’S GOALS

The most critical factor in a successful evaluation is having well-defined goals for your project. That means you should be thinking about evaluation in the earliest stages of the project. "Often we find that project directors have very lofty goals which are admirable but not really measurable," said Goodman. "One of our first tasks is helping them define measurable goals and objectives."

To define your goals simply ask yourself the following question: "What specific outcomes would you like to point to at the end of the project?" Then plot out the steps you will take to reach each goal. These steps are your project’s objectives.

EVALUATION TIPS, TYPES AND TOOLS

Once you have clearly defined your project’s goals and objectives, you can begin to frame the evaluation. Franco suggests asking yourself these questions when developing a framework for your evaluation process:

1. What do you want to learn from the evaluation?
2. Who can best provide the feedback you want to gather?
3. How can you best use evaluation tools to elicit this information from these individuals?

There are four types of evaluations. It is important to understand the basics abut each one, and to choose the type(s) that are most appropriate for your project.

PROCESS EVALUATIONS focus on the steps that lead up to the desired outcome. A process evaluation measures program elements such as the number of people who attend an event, or call a hotline.

OUTCOME EVALUATIONS document short-term results, such as the number of individuals who sign up for a new program as a result of an event or the number of people who demonstrate "increased awareness" of an issue as a result of the efforts of your project. The Heart of the Matter campaign conducted an outcome evaluation one year after the broadcast.

FORMATIVE EVALUATION is designed to assess strengths and weaknesses in pilot programs, and to maximize the opportunities for success if a project is expanded. The Television Race Initiative is taking this approach.

IMPACT EVALUATION is more comprehensive, and focuses on long-range results, e.g., changes in station policy and in community behavior that can be attributed to the project. Impact evaluation is very expensive and long-term.

"Distinctions [among evaluation approaches] are important because projects generally over-state their goals, so that they are both unrealistic and impossible to measure," notes Dr. Fran Cooper, the Television Race Initiative evaluator. "A good evaluator should help you state goals and objectives that are concrete and measurable."

There are three basic tools for conducting evaluations: surveys, interviews and case studies. Surveys can be conducted by telephone or mail and are a good way of getting information from a large group of people. Survey questions can be used to gather both qualitative and quantitative data. The numbers may or may not be statistically valid, depending on many factors, including the size of the sample you are surveying and its relationship to the questions you are asking. Any well-structured survey can still be extremely useful. However, if you have not studied survey research, you need a professional to help you design your survey.

Interviews and case studies are always anecdotal and will never be statistically valid, but they are excellent sources of information about the process. Interviews are generally most useful to gather data about the perceptions of project participants. Case studies are used more often when the goal is to explain to others how to implement a similar project, especially how to capitalize on the successes and avoid the failures of the project under study. Debra Franco noted that "Anything that adds richness and depth, talks about motivation and perception–anything with a human aspect that is not just a measurable outcome–is going to be well-served by gathering anecdotal information." The problem with anecdotal information is that it should not be used to draw firm conclusions.

HIRING AN EVALUATOR

If you have the resources, you may want to consider hiring a professional evaluator (an individual or a firm). Bring the evaluator into the project as early as possible. Decisions that you make from the very beginning can either complicate or simplify the evaluation process and have a big effect on the quality of data you are able to collect and the cost of gathering that data. Hiring a professional evaluator affords certain advantages:

1. EXPERTISE. Professional evaluators have the experience and know-how to develop an evaluation plan that will realistically assess the success of your project.

2. TIME. Evaluation can be time-consuming, and few of us can afford to remain focused on a project after it is completed.

3. OBJECTIVITY. Professional evaluators have the objectivity both to recommend appropriate measurement tools and use them in ways that will not influence responses.

4. GREATER RESPONSE RATE. The response rate to an evaluator’s questionnaires and requests for interviews, case studies, etc., is generally much higher than if you do it yourself. Being approached by a professional evaluator usually indicates to people that you highly value their input, and thus they are much more likely to cooperate.

5. MORE ACCURATE RESPONSES. Professional evaluators can elicit more accurate responses from evaluation participants. They are less likely to ask leading questions, and because they have a certain degree of distance from the project, respondents feel more comfortable sharing their true perspectives.

6. CREDIBILITY. All of the above make an evaluation conducted by a professional evaluator much more credible than one conducted by the project team.

Again, an evaluator’s role is not to state whether an effort succeeded–rather, their goal is to guide the learning process. If you choose to work with a professional evaluator, look around to find the right match. Use the following questions to help guide you in making your decision:

DOES THE INDIVIDUAL OR FIRM HAVE:

• experience with media projects?
• a clear understanding of your project goals?
• a realistic plan for evaluating the project?
• How open is the evaluator to your input and collaboration in the design of the evaluation tools?
• How much will it cost?

Evaluating media is substantially different from other projects, so if at all possible, find someone with expertise in that area. "Make sure they’ve done something, if not in film or television, something in the same arena," Franco suggested. "Otherwise you’ll have to spend too much time explaining and translating."

Whoever you choose should listen to your needs and work closely with you to design the evaluation. Fran Cooper emphasizes, "Find an evaluator who is willing to work in partnership with you to improve your project, not someone who wants to keep you at arms length from the evaluation." You know more about what is meaningful to you and, as Franco noted, "a professional evaluator will design objective questions that get to the heart of your concerns in ways that are measurable. But of course, you will want to have input about the important questions." The most critical factor in a successful evaluation is having well-defined goals for your project.

Cost can certainly be a major consideration, and evaluators will either charge on an hourly basis or a flat fee for the entire project. Keep in mind that an hourly arrangement can be both costly and limiting. An hourly rate, Franco noted, limits how much a person can get done. "There’s always more work involved than appears up-front," she explained. "So be clear about your needs and expectations from the outset." She also urged: "Do as much up-front work as you can so you don’t waste their time. Define the parameters of the project, outline your goals and budget, and put it all down on paper. If you have $50,000, you can do $50,000 worth of evaluation." Cooper adds, "As a rule of thumb, professional evaluators generally recommend that the evaluation budget should be 10%—15% of the project budget."

CONDUCTING AN EVALUATION BY YOURSELF

For some of us, given the scope of the project or budget constraints, hiring a professional evaluator is not always a viable option. For do-it-yourself evaluators, Debra Franco offered this advice:

Do all of your thinking before you begin your project and carefully strategize what you can accomplish. It’s much more important to do a well-planned, small evaluation of even one important aspect of your outreach than to do a haphazard, unsubstantiated overview. Think about what is really measurable, and I don’t mean just statistically measurable. Don’t try to deliver big impressive numbers. If you are using anecdotal information as the basis for your evaluation, make sure you use quotes to substantiate It. As long as you say up front what you’re doing and you’re honest about the results, it’s as valuable (although not "valid") as statistical analysis.

Franco also cautions that "it is usually inappropriate for the actual project team to conduct phone or face-to-face interviews" because the responses will be influenced by the interviewee’s relationship to the interviewer. "So if you’re doing it yourself, mail surveys are probably a better option. Or, try getting someone else to conduct the interviews." Cooper points out, "If your project takes place near a college or university, you may be able to recruit a professor of communications, public relations or sociology who is willing to take on your evaluation as a class project, or recommend it to a graduate student as a subject for a thesis."

CONCLUSION: THE VALUE OF EVALUATION

Evaluation doesn’t have to be a necessary evil. At P.O.V., various internal and commissioned studies have helped us determine which project activities are most successful and should be strengthened and continued, and conversely, which need corrective action. They have been an excellent informational tool as we initiated partnerships with other projects and organizations. And clearly, the existence of an objective assessment has been valuable when asking funders to continue or expand support for a project.

Moreover, the evaluation process has fundamentally helped us reassess our goals and objectives–and therefore strengthen our organization. We haven’t always liked every finding, but collectively the information has contributed to a better institutional understanding of who we are and where we’re going.

Ellen Schneider is executive director of American Documentary, Inc., the company that produces the PBS series, P.O.V.

Melanie Piersol served as audience development coordinator for P.O.V. from 1994 to 1996.