Reflections Paper #3

The two major areas of impact to me from the readings for today are that questions are very important to the planning and process of the evaluation; and that stakeholders and interested audiences must be consulted early in the process, and may need to be consulted often as the evaluation identifies pertinent areas for question in the evaluation project.

Page 200 of the text notes that "an evaluation is only adequate if it collects information from and reports information to all legitimate evaluation audiences."  The process of identifying audiences can be driven towards those audiences that are the primary audiences (sponsor, client) or those that are the "squeaky wheel."  Such narrowing of scope in audience may either create an inadequate evaluation, or lead to discounting the evaluation itself when one of these ignored audiences may be impacted by an evaluation in which their concerns or needs were not consulted.  To help identify all audiences that may need to be considered, there is a helpful matrix on page 202 of the text that may help an evaluation team identify pertinent audiences.

As audiences are identified, the complexity and scope of the evaluation could increase to a point where an economically viable evaluation (in terms of financial, human resources, or time) is not possible.  The object of the evaluation must be carefully aligned with the questions to be addressed in order to have a meaningful evaluation.  The questions on pp. 204-205 seemed extremely helpful to identifying the critical object of the evaluation.  The program theory defined by Chen helps phrase the evaluation question.  Program theory helps the evaluator identify the normative theory (how the program should be) and compare it to the causative theory which identifies the likely outcomes given the inputs of the characteristics of the client and their actions.  This moves the evaluation object from the "black box" of sterile objects and plans, to the reality of the impacts of people, circumstances, organization, etc. that may lead to an evaluation of the program's success within the context of the organizational setting.

After carefully defining the audiences and the scope of the evaluation, the evaluator then moves through a series of questions to guide the evaluation itself.  A two part process of identifying divergent questions will help capture pertinent concerns of the audiences and stakeholders.  By meeting with the different stakeholders and asking them to express all of their concerns through questions that help identify the questions, concerns, and values of the stakeholders the evaluation team can identify possible needs to address in the evaluation.

After casting a wide net in the divergent phase, the evaluation team will then move to the convergent phase which identifies the questions that are truly pertinent to the evaluation and that can be reasonably addressed with resources available.  Primary drivers include:

1. Who needs or wants the information?
2. Would the answer to the question under consideration provide information not now available?
3. Would the answer to the question yield important information?
4. Is the question a matter of passing interest or does it focus on critical dimensions of continuing interest?
5. Would the scope or comprehensiveness be seriously limited if this question were dropped?
6. are resources sufficient to make answering this question feasible?

The purpose of the audience consultation and the development of these questions is to limit the scope and give the evaluation the proper focus.  It also helps identify the criteria that determines if the program is successful or not.  It may be successful at some objectives, but that may not be the objectives that matter to primary stakeholders or audiences.

The impact on our project was demonstrated when we met with the teachers today.  The questions that we asked were an attempt to see what the audience of the teacher would desire from this program evaluation.  We will also consult with the client and ask questions that will focus the data collection and help identify key metrics that are important to the analysis.  These questions should continue as we collect data and prepare the analysis.