1. Kirkpatrick's Learning and Training Evaluation Theory by Zhu Min & Zhao Dan www.sinauonline.com Based on Evaluating Training Programs (Third Edition)
2. Agenda Knowledge and Learning for a Better Performance 1 Overview 3 2 Level1: Reaction 4 3 Level 2: Learning 7 4 Level 3: Behavior 9 5 Level 4: Results 11 6 Case and Conclusion 13
3. An Overview of Kirkpatrick’s Four Level Knowledge and Learning for a Better Performance Each level is important and has an impact on the next level. As you move from one level to the next, the process becomes more difficult and time-consuming, but it also provides more valuable information. How those who participate in the program react to it The extent to which participants change attitudes, improve knowledge, and/or increase skill The extent to which change in behavior has occurred The final results that occurred Difficulty & Investment increase
4. Level 1 Reaction Knowledge and Learning for a Better Performance Evaluation on this level measures how those who participate in the program react to it, also called customer satisfaction . Why it is important to measure reaction ? Definition First, it gives us valuable feedback that helps us to evaluate the program. Second, it tells trainees that the trainers are there to help them do their job better and that they need feedback to determine how effective they are. Finally, reaction sheets can provide trainers with quantitative information that can be used to establish standards of performance for future programs.
5. Guidelines for Evaluating Reaction Knowledge and Learning for a Better Performance 3. Encourage written comments and suggestions. 4. Get 100 percent immediate response. 5. Get honest responses. 8. Communicate reactions as appropriate. 1. Determine what you want to find out. In every program, it is imperative to get reactions both to the subject and to the leader. 2. Design a form that will quantify reactions. The ideal form provides the maximum amount of information and requires the minimum amount of time. (See next slide) 6. Develop acceptable standards. Tally the responses in each category for all items. Excellent = 5 Very good = 4 Good = 3 Fair = 2 Poor = 1 7. Measure reactions against standards and take appropriate action. Several approaches are possible if the standard is not met. 1. Make a change—in leaders, facilities, subject, or something else. 2. Modify the situation. If the instructor does not meet the standard, help by providing advice, new audiovisual aids, or something else. 3. Live with an unsatisfactory situation. 4. Change the standard if conditions change.
7. Level 2 Learning Knowledge and Learning for a Better Performance Definition Learning can be defined as the extent to which participants change attitudes, improve knowledge, and/or increase skill as a result of attending the program. Learning is happened when Attitudes are changed. Knowledge is increased. Skill is improved.
8. Guidelines for Evaluating Learning Knowledge and Learning for a Better Performance 2. Evaluate knowledge, skills, and/or attitudes both before and after the program. 3. Use a paper-and-pencil test to measure knowledge and attitudes. 4. Use a performance test to measure skills. 5. Get a 100 percent response. 6. Use the results of the evaluation to take appropriate action. If the learner hasn’t learned, the teacher hasn’t taught. This is a good philosophy for each instructor to have. Summary Evaluating learning is important. Without learning, no change in behavior will occur. If the knowledge/skill is new, there is no need for a pretest. But if we are teaching concepts, principles, and techniques that trainees may already know, a pretest that we can compare with a posttest is necessary. 1. Use a control group if practical. Control group refers to a group that does not receive the training. The group that receives the training is called the experimental group. The control and experimental groups are equal in all significant characteristics.
9. Level 3 Behavior Knowledge and Learning for a Better Performance Behavior can be defined as the extent to which change in behavior has occurred because the participant attended the training program. Four necessary conditions for change to occur Definition 1. The person must have a desire to change. 2. The person must know what to do and how to do it. 3. The person must work in the right climate. 4. The person must be rewarded for changing.
10. Guidelines for Evaluating Behavior Knowledge and Learning for a Better Performance 1. Use a control group if practical. No evaluation should be attempted until trainees have had an opportunity to use the new behavior. 2. Allow time for behavior change to take place. 3. Evaluate both before and after the program if practical. 4. Survey and/or interview: Survey and/or interview one or more of the following: trainees, their immediate supervisor, their subordinates, and others who often observe their behavior. The best approach is to use a patterned interview in which all interviewees are asked the same questions. But interviews are very time-consuming, a survey questionnaire is usually more practical. 5. Get 100 percent response or a sampling. The dictum that something beats nothing can apply when you evaluate change in behavior. Sampling: A few “typical” trainees at random or the persons most likely not to change. 6. Repeat the evaluation at appropriate times. 7. Consider cost versus benefits. Waiting two or three months before conducting the first evaluation, the exact number depending on the opportunity that trainees have to use the new behavior. Cost : In many organizations, much of the cost of evaluation at level 3 is in the staff time that it takes to do. Hiring of an outside expert to guide or even conduct the evaluation. Benefit : changes in behavior and final results. Another important consideration is the number of times the program will be offered. if a program is going to be repeated, the time and money spent evaluating it can be justified by the possible improvements in future programs.
11. Level 4 Results Knowledge and Learning for a Better Performance Results can be defined as the final results that occurred because the participants attended the program. When is result evaluation worthy to do Definition If your programs aim at tangible results rather than teaching management concepts, theories, and principles, then it is desirable to evaluate in terms of results.
12. Guidelines for Evaluating Results Knowledge and Learning for a Better Performance 1. Use a control group if practical. The amount of money that should be spent on level 4 evaluation should be determined by the amount of money that the training program costs, the potential results that can accrue because of the program, and the number of times that the program will be offered. 2. Allow time for results to be achieved. 3. Measure both before and after the program if practical. 4. Repeat the measurement at appropriate times. 5. Consider cost versus benefit. 6. Be satisfied with evidence if proof is not possible.
13. Implementing the Four Levels Knowledge and Learning for a Better Performance start at level 1 and proceed through the other levels as time and opportunity allow. Level 1 "If my customers are unhappy, it is my fault, and my challenge is to please them.” Level 2 Pretest and Posttest. Decide what knowledge, skills, and attitudes you want participants to have at the end of the program. Level 3 Time consuming; Statistics are desired. Benefits versus Cost need to be considered Level 4 Most time consuming. Statistics are desired. Benefits versus Cost is very important. Proof is not easy You can borrow evaluation forms, designs, and procedures from others, but you cannot borrow evaluation results.
