A Common Sense Approach to Evaluation
Publication Date: July 5, 2011
By
Sandra Adams, Program Associate, Evaluation and Planning
To evaluate or not to evaluate…
At the Foundation, we “evaluate” all of the work we support in one way
or another. The question is not “whether to evaluate or not”; rather,
it is “what level of evaluation is appropriate”, which is informed by
answers to the questions “what information do we need” and “how will
that information be used.”
As a starting point, the Foundation tracks whether grantees accomplish what they set out to do. This is done through regular program officer monitoring and annual grantee reporting on grant
outputs and outcomes. Learning that emerges through these evaluative activities is used to inform ongoing work, as well as future work. It is the primary way that we capture and learn from our grantees’ experiences.
Sometimes though, the Foundation needs more in-depth information to understand whether a project or strategy is working (i.e., producing the intended results). At a project level (e.g., a pilot program), a process and outcome evaluation may be commissioned. A process evaluation is used to examine the implementation of a project and provides grantees, the Foundation, and, in some cases, the field with valuable information about whether the project was implemented as intended and how implementation of the project could be improved. This information may also be used to document implementation for the purpose of replication and to judge whether a project is ready for an outcome evaluation. An outcome evaluation is commissioned in order to examine whether a project produces the intended result. Evidence of effectiveness can then be used to make the case for a successful project’s system-wide adoption. The design of an outcome evaluation and the methodology used (i.e., randomized control, quasi-experimental, non-experimental) is determined by the type of project being evaluated and the level of evidence needed to “make the case.”
At the strategy level, tracking of indicators or a strategy review is used to help Foundation leadership and staff understand changes in the fields in which we are working and make appropriate adjustments to our strategies. The tracking of key indicators informs the Foundation about changes in city and regional outcomes (e.g., School District of Philadelphia graduation rate) related to a Foundation-supported body of work. A strategy review provides a more detailed examination of whether a targeted change was achieved through a Foundation-supported body of work by examining how the work contributes to the achievement of a change (or lack thereof). A strategy review is a comprehensive look at all components of a Foundation strategy including our assumptions, choice of work to support, and level of investment made. It may include interviews with key stakeholders, an assessment of the field, and a review of grantee reports and interviews with grantees, although a strategy review is not an assessment of grantees’ accomplishments. Strategy reviews are often conducted after a strategy has been in place for several years.
Both images licensed under Creative Commons
Some rights reserved by Brittany ((Halo))
Some rights reserved by gwen