This link has been bookmarked by 58 people . It was first bookmarked on 23 Jul 2006, by Terri Gilpin.
-
15 Jun 12
-
26 Dec 10
-
06 Dec 10
-
05 Oct 10
-
07 Sep 10
bkmurrayhttp://www.authenticityconsulting.com/pubs/PG_gdes/PG_pubs.htm
On the Free Management Library site, this URL is based on a non-profit field guide to program evaluation, this is a great site that walks the user through the process and methodologies neededanalysis assessment development education evaluation guide learning nonprofit Management Methodology program program_evaluation reference research school work resources career howto program+evaluation program-evaluation programevaluation PPE Planning Ev
-
Brian Murrayhttp://www.authenticityconsulting.com/pubs/PG_gdes/PG_pubs.htm
On the Free Management Library site, this URL is based on a non-profit field guide to program evaluation, this is a great site that walks the user through the process and methodologies neededanalysis assessment development education evaluation guide learning nonprofit Management Methodology program program_evaluation reference research school work resources career howto program+evaluation program-evaluation programevaluation PPE Planning Ev
-
21 Aug 10
DivineChoices AdministratorOther online resources for Program Evaluation -- http://www.new.org/resourceconnect/2010_resources/resources.php?topic_ID=16&type_ID=3&subtopic_ID=15
-
02 Aug 10
-
02 Jul 10
-
25 Apr 10
-
08 Apr 10
E.J. SorensenFree Management Library provides an online resource listing to methods and evaluations.
assessment development education grantwriting resources methodology
-
01 Mar 10
-
12 Feb 10
-
07 Feb 10
-
04 Feb 10
-
08 Jul 09
-
04 May 09
-
The type of evaluation you undertake to improve your programs depends on what you want to learn about the program
-
worry about what you need to know to make the program decisions you need to make, and worry about how you can accurately collect and understand that information.
-
For what purposes is the evaluation being done
-
Who are the audiences for the information from the evaluation
-
What kinds of information are needed to make the decision you need to make and/or enlighten your intended audiences,
-
From what sources should the information be collected,
-
How can that information be collected in a reasonable fashion,
-
When is the information needed
-
What resources are available to collect the information?
-
Outcomes are usually in terms of enhanced learning (knowledge, perceptions/attitudes or skills) or conditions, e.g., increased literacy, self-reliance, etc. Outcomes are often confused with program outputs or units of services, e.g., the number of clients who went through a program.
-
Identify the major outcomes that you want to examine or verify for the program under evaluation
-
"Why are we doing that?"
-
Choose the outcomes that you want to examine, prioritize
-
For each outcome, specify what observable measures, or indicators, will suggest that you're achieving that key outcome
-
Identify what information is needed to show these indicators,
-
Decide how can that information be efficiently and realistically gathered
-
Analyze and report the findings
-
Always start with your evaluation goals:
-
If you are conducting an outcomes-based evaluation, you can categorize data according to the indicators for each outcome.
-
-
03 Feb 09
-
02 Feb 09
-
15 Dec 08
-
13 Dec 08
-
06 Nov 08
-
25 Aug 08
-
19 Jun 08
-
Basic Guide to Program Evaluation
-
More recently (especially as a result of Michael Patton's development of utilization-focused evaluation), evaluation has focused on utility, relevance and practicality
-
Many people believe that evaluation is about proving the success or failure of a program. This myth assumes that success is implementing the perfect program and never having to hear from employees, customers or clients again
-
Success is remaining open to continuing feedback and adjusting the program accordingly. Evaluation gives you this continuing feedback.
-
Program Evaluation
-
Goals-Based Evaluation
-
Goal-based evaluations are evaluating the extent to which programs are meeting predetermined goals or objectives.
-
Process-Based Evaluations
-
Process-based evaluations are geared to fully understanding how a program works -- how does it produce that results that it does. These evaluations are useful if programs are long-standing and have changed over the years, employees or customers report a large number of complaints about the program, there appear to be large inefficiencies in delivering program services and they are also useful for accurately portraying to outside parties how a program truly operates (e.g., for replication elsewhere).
-
Outcomes-Based Evaluation
-
Program evaluation with an outcomes focus is increasingly important for nonprofits and asked for by funders.
-
An outcomes-based evaluation facilitates your asking if your organization is really doing the right program activities to bring about the outcomes you believe (or better yet, you've verified) to be needed by your clients (rather than just engaging in busy activities which seem reasonable to do at the time). Outcomes are benefits to clients from participation in the program. Outcomes are usually in terms of enhanced learning (knowledge, perceptions/attitudes or skills) or conditions, e.g., increased literacy, self-reliance, etc. Outcomes are often confused with program outputs or units of services, e.g., the number of clients who went through a program.
-
To accomplish an outcomes-based evaluation, you should first pilot, or test, this evaluation approach on one or two programs at most (before doing all programs).
-
The general steps to accomplish an outcomes-based evaluation include to:
-
1. Identify the major outcomes that you want to examine or verify for the program under evaluation.
-
2. Choose the outcomes that you want to examine, prioritize the outcomes and, if your time and resources are limited, pick the top two to four most important outcomes to examine for now.
-
3. For each outcome, specify what observable measures, or indicators, will suggest that you're achieving that key outcome with your clients.
-
4. Specify a "target" goal of clients, i.e., what number or percent of clients you commit to achieving specific outcomes with, e.g., "increased self-reliance (an outcome) for 70% of adult, African American women living in the inner city of Minneapolis as evidenced by the following measures (indicators) ..."
-
5. Identify what information is needed to show these indicators, e.g., you'll need to know how many clients in the target group went through the program, how many of them reliably undertook their own transportation to work and stayed off drugs, etc.
-
6. Decide how can that information be efficiently and realistically gathered (see Selecting Which Methods to Use below).
-
Overview of Methods to Collect Information
-
Reporting Evaluation Results
-
-
13 Jun 08
-
05 Apr 08
-
26 Mar 08
-
05 Mar 08
-
08 Feb 08
-
Program evaluation is carefully collecting information about a program or some aspect of a program in order to make necessary decisions about the program. Program evaluation can include any or a variety of at least 35 different types of evaluation, such as for needs assessments, accreditation, cost/benefit analysis, effectiveness, efficiency, formative, summative, goal-based, process, outcomes, etc. The type of evaluation you undertake to improve your programs depends on what you want to learn about the program. Don't worry about what type of evaluation you need or are doing -- worry about what you need to know to make the program decisions you need to make, and worry about how you can accurately collect and understand that information.
-
Understand, verify or increase the impact of products or services on customers or clients - These "outcomes" evaluations are increasingly required by nonprofit funders as verification that the nonprofits are indeed helping their constituents.
-
erify that you're doing what you think you're doing - Typically, plans about how to deliver services, end up changing substantially as those plans are put into place. Evaluations can verify if the program is really running as originally planned.
-
ully examine and describe effective programs for duplication elsewhere
-
The process is how the program is carried out, e.g., customers are served, clients are counseled, children are cared for, art is created, association members are supported, etc. The outputs are the units of service, e.g., number of customers serviced, number of clients counseled, children cared for, artistic pieces produced, or members in the association. Outcomes are the impacts on the customers or on clients receiving services, e.g., increased mental health, safe and secure development, richer artistic appreciation and perspectives in life, increased effectiveness among members, etc.
-
What is the status of the program's progress toward achieving the goals?
3. Will the goals be achieved according to the timelines specified in the program implementation or operations plan? If not, then why?
4. Do personnel have adequate resources (money, equipment, facilities, training, etc.) to achieve the goals?
5. How should priorities be changed to put more focus on achieving the goals? (Depending on the context, this question might be viewed as a program management decision, more than an evaluation question.)
6. How should timelines be changed (be careful about making these changes - know why efforts are behind schedule before timelines are changed)?
7. How should goals be changed (be careful about making these changes - know why efforts are not achieving the goals before changing the goals)? Should any goals be added or removed? Why?
8. How should goals be established in the future? -
they are also useful for accurately portraying to outside parties how a program truly operates (e.g., for replication elsewhere).
-
What is required of employees in order to deliver the product or services?
-
How are employees trained about how to deliver the product or services?
-
What do staff consider to be strengths of the product or program?
-
What typical complaints are heard from employees and/or customers?
11. What do employees and/or customers recommend to improve the product or program? -
Outcomes are benefits to clients from participation in the program. Outcomes are usually in terms of enhanced learning (knowledge, perceptions/attitudes or skills) or conditions, e.g., increased literacy, self-reliance, etc. Outcomes are often confused with program outputs or units of services, e.g., the number of clients who went through a program.
-
There are four levels of evaluation information that can be gathered from clients, including getting their:
1. reactions and feelings (feelings are often poor indicators that your service made lasting impact)
2. learning (enhanced attitudes, perceptions or knowledge)
3. changes in skills (applied the learning to enhance behaviors)
4. effectiveness (improved performance because of enhanced behaviors) -
If you are conducting an outcomes-based evaluation, you can categorize data according to the indicators for each outcome.
-
Basic analysis of "quantitative" information (for information other than commentary, e.g., ratings, rankings, yes's, no's, etc.):
1. Make copies of your data and store the master copy away. Use the copy for making edits, cutting and pasting, etc.
2. Tabulate the information, i.e., add up the number of ratings, rankings, yes's, no's for each question.
3. For ratings and rankings, consider computing a mean, or average, for each question. For example, "For question #1, the average ranking was 2.4". This is more meaningful than indicating, e.g., how many respondents ranked 1, 2, or 3.
4. Consider conveying the range of answers, e.g., 20 people ranked "1", 30 ranked "2", and 20 people ranked "3".Basic analysis of "qualitative" information (respondents' verbal answers in interviews, focus groups, or written commentary on questionnaires):
1. Read through all the data.
2. Organize comments into similar categories, e.g., concerns, suggestions, strengths, weaknesses, similar experiences, program inputs, recommendations, outputs, outcome indicators, etc.
3. Label the categories or themes, e.g., concerns, suggestions, etc.
4. Attempt to identify patterns, or associations and causal relationships in the themes, e.g., all people who attended programs in the evening had similar concerns, most people came from the same geographic area, most people were in the same salary range, what processes or events respondents experience during the program, etc.
4. Keep all commentary for several years after completion in case needed for future reference. -
Attempt to put the information in perspective, e.g., compare results to what you expected, promised results; management or program staff; any common standards for your services; original program goals (especially if you're conducting a program evaluation); indications of accomplishing outcomes (especially if you're conducting an outcomes evaluation); description of the program's experiences, strengths, weaknesses, etc. (especially if you're conducting a process evaluation).
2. Consider recommendations to help program staff improve the program, conclusions about program operations or meeting goals, etc.
3. Record conclusions and recommendations in a report document, and associate interpretations to justify your conclusions or recommendations. -
e sure employees have a chance to carefully review and discuss the report. Translate recommendations to action plans, including who is going to do what about the program and by when.
-
funders will likely require a report that includes an executive summary (this is a summary of conclusions and recommendations, not a listing of what sections of information are in the report -- that's a table of contents); description of theorganization and the program under evaluation; explanation of the evaluation goals, methods, and analysis procedures; listing of conclusions and recommendations; and any relevant attachments, e.g., inclusion of evaluation questionnaires, interview guides, etc. The banker or funder may want the report to be delivered as a presentation, accompanied by an overview of the report. Or, the banker or funder may want to review the report alone.
4. Be sure to record the evaluation plans and activities in an evaluation plan which can be referenced when a similar program evaluation is needed in the future. -
Ideally, management decides what the evaluation goals should be. Then an evaluation expert helps the organization to determine what the evaluation methods should be, and how the resulting data will be analyzed and reported back to the organization.
-
Ensure your evaluation plan is documented so you can regularly and efficiently carry out your evaluation activities. Record enough information in the plan so that someone outside of the organization can understand what you're evaluating and how. Consider the following format for your report:
1. Title Page (name of the organization that is being, or has a product/service/program that is being, evaluated; date)
2. Table of Contents
3. Executive Summary (one-page, concise overview of findings and recommendations)
4. Purpose of the Report (what type of evaluation(s) was conducted, what decisions are being aided by the findings of the evaluation, who is making the decision, etc.)
5. Background About Organization and Product/Service/Program that is being evaluated
a) Organization Description/History
b) Product/Service/Program Description (that is being evaluated)
i) Problem Statement (in the case of nonprofits, description of the community need that is being met by the product/service/program)
ii) Overall Goal(s) of Product/Service/Program
iii) Outcomes (or client/customer impacts) and Performance Measures (that can be measured as indicators toward the outcomes)
iv) Activities/Technologies of the Product/Service/Program (general description of how the product/service/program is developed and delivered)
v) Staffing (description of the number of personnel and roles in the organization that are relevant to developing and delivering the product/service/program) -
6) Overall Evaluation Goals (eg, what questions are being answered by the evaluation)
7) Methodology
a) Types of data/information that were collected
b) How data/information were collected (what instruments were used, etc.)
c) How data/information were analyzed
d) Limitations of the evaluation (eg, cautions about findings/conclusions and how to use the findings/conclusions, etc.)
8) Interpretations and Conclusions (from analysis of the data/information)
9) Recommendations (regarding the decisions that must be made about the product/service/program)
Appendices: content of the appendices depends on the goals of the evaluation report, eg.:
a) Instruments used to collect data/information
b) Data, eg, in tabular format, etc.
c) Testimonials, comments made by users of the product/service/program
d) Case studies of users of the product/service/program
e) Any related literature
-
-
17 Nov 07
-
26 Oct 07
-
13 Sep 07
-
29 May 07
-
22 May 07
-
10 Mar 07
-
20 Jan 07
-
14 Oct 06
-
07 Aug 06
-
30 Mar 06
Would you like to comment?
Join Diigo for a free account, or sign in if you are already a member.