Evaluating Programs & Human Performance
Instructional Design Evaluation Models
The discussion of Instructional Design cannot be complete without including Evaluation Models. There are several models that are used frequently in the Instructional Design industry. Brinkerhoff’s Success Case Method (SCM) and Patton’s Utilization-Focused Evaluation (U-FE) will be my focus today.
Brinkerhoff’s Success Case Method (SCM)
Brinkerhoff’s model takes a “common sense” approach to evaluating instructional programs. (Reiser, 2012, p. 101) In this model the evaluator follows a five-step process that allows them to look at not just data, but also the narrative that is taking place within the program. Looking at both the data and the narrative allows the evaluator to see all aspects of the program and decide what is helping and what is hindering the process.
The five steps in order are, “First, focus and plan the success case (SC) study.” In this step the evaluator works with the stake holders to decide the purpose of the evaluation along with the budget and time frame. The second step is to create a visual impact model. This model outlines what the program should be providing others, the “business goals”, or what the stakeholder should receive from the program. The impact model also includes a section on “behaviors and organizational and environmental conditions.” (Reiser, 2012, p. 101) A research study, in the form of a survey is then conducted as the third step. This study seeks out the most successful cases along with the worst cases. There is a set of specific questions to be asked that outline what parts of the program are being used, who is being successful, and who is struggling with the program. The fourth step is an in-depth interview with the success stories, and even perhaps a few unsuccessful cases for their insight. The last step is to summarize and outline the findings of your case study and report to the stakeholder.
Patton’s Utilization- Focused Evaluation (U-FE)
As the name states, Patton’s evaluation model is focused on utilizing the information that is gathered in the evaluation. The key to utilization is to impart the value of its use to the organization. Reiser outlines the nine steps on page 102 as follows…
1. Conduct a readiness assessment
2. Identify the “primary intended users” and develop a working relationship with them.
3. Conduct a situational analysis.
4. Identify the “primary intended uses”.
5. Focus the evaluation.
6. Design the evaluation.
7. Collect, analyze and interpret the evaluation data
8. Continually facilitate evaluation use.
9. Conduct metaevaluation.
Reflection
As I learned about these two evaluation models, I had a difficult time imagining myself using them to evaluate a lesson in my classroom or library. As I read about them I imagined a corporate training program or a school wide initiative going through these evaluations. It seems impractical to use these formal evaluations for each lesson. I really cannot see myself using these on a personal level. I do however see how useful they would be to evaluate a campus initiative like the Personalized Learning program we are starting at my school this fall. I can see how seeking out those who were successful and those who weren’t and interviewing them would lead to many questions and answers to help going forward. The U-FE model would insure that the information doesn’t stop, that we continue to evaluate and disseminate information throughout the year and years to come.
Evaluation Questions
In addition to questions about learning improvement and student satisfaction I do believe that cost should be factored in. The evaluation should ask if there is enough return on the investment to place value in the program. When I think about investment I don’t just think of monetary value, we should also look at human value and time spent. If a new program is started and learning is taking place and results are seen, but the staff is working 12 hour days and they are stressed and tired, then it might not be the best program. There needs to be balance. Evaluation models need to look at the quality of life for all involved in addition to the learning results.
Human Performance
Dallas ISD has a fantastic automation system for its libraries. Teachers and students can log into the system to request books, place holds, make lists, share lists, view their book status and even download e-books. This is a valuable resource, but very few of the teachers know about it or utilize it. Most teachers at my school don’t even check out books for their classes. As a solution, I could make a video advertising the features of the system. I could also print out cards for the teachers to keep by their computers that give them step by step instructions to access their account and the features that are available to them.
Reiser, R. A., & Dempsey, J. V. (2012). Trends and issues in instructional design and technology. Boston: Pearson.
Kelly,
ReplyDeleteLike you, I have a hard time envisioning myself using these evaluation tools in my classroom. I can see where this would be beneficial for companies or incorporating a program throughout an entire school district. I also feel that cost should be measured in time/effort spent as well as monetary cost. It is very frustrating to spend lots of time learning something new if not everyone is going to have to use it or if they get rid of a program after just one or two years.
I love your idea about printing a card with instructions for teachers to keep beside their desks. This would be especially useful for those who are new to your system so that they feel more comfortable finding information and exploring your system themselves. You never know what you can find until your given the tools to begin the search!
“Evaluation models need to look at the quality of life for all involved in addition to the learning results.”
ReplyDeleteWhen a district fails to consider and address this aspect when evaluating a new product and/or program they do so at their own peril. In my experience, a happy teacher is a productive teacher, less prone to absenteeism and/or departure from a school or district. When a person’s “quality of life” is taken into account stress levels decrease, the work environment improves, the quality of instruction increases, usually equating to higher student performance. This would be an exceptional addition to any evaluation conducted within an educational setting. Good Job!
“Dallas ISD has a fantastic automation system for its libraries. Most teachers at my school don’t even check out books for their classes. As a solution, I could make a video advertising the features of the system. I could also print out cards for the teachers to keep by their computers that give them step by step instructions to access their account and the features that are available to them.”
What an incredibly simple, direct, and effective solution to this problem. Many of your staff members who are not currently taking advantage of this system might want and/or need to use this service but are unsure of how to do so and are too embarrassed or proud to ask for help. Hopefully this video will make this important resource a little less threatening to the staff as they can view it independently and at their own pace, providing the catalyst for an ongoing, professional learning community with the library at its’ nucleus. Great Idea! The quick reference instructional card is also a winner of an idea, the only suggestion I can think of is to laminate the cards before distributing them so that they gain added durability and resilience especially in the face of morning coffee catastrophes, very messy business!
I actually thought the both of the models you mentioned were good ones. :) Our church just used something very similar to Brinkerhoff's to evaluate the needs for a building program. I thought it was a very thorough process that yielded some valuable information to be considered in moving forward.
ReplyDeleteI love your idea about the video and reference card to keep handy! Great problem solving!
I honestly had never thought about the evaluation models for instructional design. I think it would be a bit unrealistic to use this method for every single lesson. As you mentioned, I've been looking at them more with implementing and evaluating programs. I could see myself using these methods if I'm trying to implement a new program into the library program.
ReplyDeleteMy district also has a great automation system and lots of resources for its libraries, and teachers also don't utilize these resources as much as they should. I had also been thinking of creating instructional videos and posting them to the school library webpage for teachers as a part of noninstructional human performance. That way, teachers can access the information at the point of need.
I completely agree with you 100% that the additional work put on teachers and staff needs tobe taken into consideration as well when evaluating programs. Teachers already carry a lot on their shoulders with their regular duties, and the last thing that any teacher wants to hear is that they now have to put in an extra three hours in order to make this new intiative happen.
I feel that it's a recipe for disaster, and it affects the students negatively. If the teachers are burnt out, the quality of instruction is not going to be as effective as it could be. And in this aspect, we have already failed the students.
Great blog post. I appreciate your in depth explanations of the alternate assessments. As I read your post, I was thinking about a new program we are implementing at my school next year. We have seen a heightened amount of stress amongst students. They are forced to do more and more and perform at higher standards than ever before and the result is often that kids just shut down. Or worse, they turn to substance abuse to get relief. Next year, we are implementing a freshman course for learning. Many people in the school will have a hand in the learning so that students get a diverse understanding of many topic areas from how to study, to time management, reflection opportunities and then we are going to give them quiet time. The will be able to do their homework in class. We have an AB schedule and kids have four classes a day, so this would be one of their 8 courses. The hope is to build community and to develop an understanding and a love for learning.
ReplyDeleteYou mentioned your idea about the library cards for the teachers. I see this issue as well and would love to do something like this. Thank you for the great idea!