Reading Assignment Week One - Chapters 4-7

This course is my first exposure to evaluation as it relates to the social sciences, particularly education.  My previous academic training and professional experiences have focused on evaluation as it relates to auditing and financial strategic decision-making.  While the subject matter related to evaluation in this text differs from the business training I have previously received, the fundamental principles and concerns are the same in this approach to evaluation as they are in the business approaches that have been my career experiences.

Auditing and business strategy analysis share many concerns in common with evaluation in the context of this text.  Among these similarities are the principle concerns regarding:

1. The independence of the evaluator/auditor
2. The role of internal vs external evaluators/auditors and their biases
3. The role of the audit/evaluation as a report to inform, or a tool to influence
4. The standards and rules governing the evaluation and the evaluation purpose
5. The interplay between quantitative and qualitative measurements and their appropriateness for the evaluation need
6. The objectivity or subjectivity of the observations

I feel very comfortable with the concepts I have read this week because it relates so closely to my previous academic and professional training.

So what did I learn this week?  Primarily I learned the classification of the different evaluation approaches.

1. Objectives-Oriented Evaluation - the purpose is to identify the extent to which specific program objectives were obtained.  This can be done by collecting data and analyzing the results to see if program goals are achieved.  The most interesting part of this approach for me was the idea of goal free evaluation.  I see many applications of this approach to identify what was actually accomplished without being anchored to the goals as they were outlined.  I have encountered unintended consequences and unanticipated variances so often in my career that I see real potential in this approach to evaluating the accomplishments of the program.
2. Management-Oriented Evaluation - this is the type of evaluation that I am most familiar with.  Its purpose is to treat the decision maker as the primary stake holder and help them refine and identify the decisions that need to be made and then scope the data collection and analysis to supply the data needed to inform that decision.  This approach reports the facts and its influence on the decision made is largely neutral.  There is danger in trying to anticipate the decision or influence the decision.  This bias may limit the factual credibility of the evaluation.
3. Consumer-Oriented Evaluation - This approach uses checklists or other criteria in hopes of providing useful information to inform consumers.  While a desire to remain objective is important, there are come who would argue that subjective concerns may benefit society as they lead consumers to more effective alternatives that will lead to change in society.  The major growth in this area cam largely from the expansion in federal funding since the 1960s.
4. Expertise-Oriented Evaluation - This approach uses an expert, a panel of experts, a formal, or an informal approach to evaluate programs.  A most common evaluation event using this approach is accreditation of educational, medical, and other institutions.  This is also the idea behind peer-reviewed refereed journals.  The experts render an opinion on the effectiveness or validity of the object or program, particularly related to objectives, standards, or societal expectations.

The implications for our class project indicate that our client hopes to show the benefit of their learning game.  This could mean an analysis of objectives to determine the success of learning improvement goals or even information for decisions about the development of the game in the future.  I am sure there will be other possible implications regarding the type, nature, and scope of the evaluation as I learn more.

Darin