2. Agenda
1. Measuring the Effectiveness of Training
Program
2. Measuring ‘Return on Investment’ of Training
3. Enhancing the Effectiveness and ROI of
Training
If you find this presentation useful, please consider telling others
about our site (www.exploreHR.org)
www.exploreHR.org 2
4. Training Process
Training Training Training
Training
Need Objectives Evaluation
Delivery
Analysis
What are Objective Techniques Measure
the training should be include on- reaction,
needs for measurable the-job- learning,
this person and training, behavior,
and/or job? observable action and results
learning, etc.
www.exploreHR.org 4
5. The Four Levels of Evaluation
Level 1 - Reaction
Level 2 - Learning
Four Levels
of Training
Level 3 – Behavior Effectiveness
Application
Level 4 – Business
Impact
www.exploreHR.org 5
6. The Four Levels of Evaluation
Evaluate trainees’ reactions to Level 1 -
the program. Did they like the Reaction
program? Did they think it
worthwhile?
Test the trainees to determine if Level 2 -
they learned the principles, Learning
skills, and facts they were to
learn.
www.exploreHR.org 6
7. The Four Levels of Evaluation
Ask whether the trainees’ behavior on the job Level 3 –
changed because of the training program. For Behavior
example, are employees in the store’s complaint Application
department more courteous toward disgruntled
customers than previously?
What final results were achieved in terms of the Level 4 –
training objectives previously set? Did the number Business
of customer complaints about employee drop? Did Impact
the reject rate improve? Was turnover reduced,
and so forth.
www.exploreHR.org 7
8. The Four Levels of Evaluation
Value of Frequency of Difficulty of
Information Use Assessment
Level
Least Frequent Easy
valuable
I. Reaction
II. Learning
III. Behavior
Most Infrequent Difficult
IV. Results valuable
www.exploreHR.org 8
9. Level 1 - Reaction
Evaluate trainees’
reactions to the program:
Did they like the program?
Level 1 - Did they like the
Reaction
facilitators?
Did they like the training
accommodation and
facilities?
www.exploreHR.org 9
10. Guidelines for Evaluating Reaction
1. Determine what you want to find out
2. Design a form that will quantify reactions
3. Encourage written comments and suggestions
4. Get 100 percent immediate response
5. Get honest response
6. Develop acceptable standards
7. Measure reactions against standards, and take
appropriate action
8. Communicate reaction as appropriate
www.exploreHR.org 10
11. Please give us your frank reactions and comments. They will help us to evaluate
this program and improve future programs.
Sample of Program :
Reaction Facilitator :
Form 1. How do you rate the subject (interest, benefit, etc)
a. Excellent
b. Very Good
c. Good
d. Fair
e. Poor
2. How do you rate the facilitator? (knowledge, ability to deliver and communicate?)
a. Excellent
b. Very Good
c. Good
d. Fair
e. Poor
3. How do you rate the facilities? (comfort, convinience, etc)
a. Excellent
b. Very Good
c. Good
d. Fair
e. Poor
4. What would have improved the program?
…………………………………………………………………………………
…………………………………………………………………………………
…………………………………………………………………………………
www.exploreHR.org 11
12. Level 2 - Learning
Measuring learning means
determining one or more of
the following :
Level 2 - • What knowledge was
Learning learned?
• What skills were
developed or improved?
• What attitudes were
changed?
www.exploreHR.org 12
13. Guidelines for Evaluating Learning
1. Use a control group if practical
2. Evaluate knowledge, skills and/or attitudes both
before and after the program
3. Use a paper-and-pencil test to measure knowledge
4. Use a performance test to measure skills
5. Get 100 percent immediate response
6. Use the results of evaluation to take appropriate
action
www.exploreHR.org 13
14. Guidelines for Evaluating Learning
Pretest and Posttest Scores on Change Management Training
Example :
Experimental Group Control Group
Pre Test Score 45 46
Post Test Score 55 48
Gain 10 2
www.exploreHR.org 14
15. Level 3 – Behavior Application
• The frequency of
application of new
skills/knowledge/
Level 3 – attitudes (on the job)
Behavior
Application • The effectiveness of the
skills/knowledge/
attitudes (as applied on
the job)
www.exploreHR.org 15
16. Guidelines for Evaluating Learning
1. Use a control group if practical
2. Allow time for behavior change and application to take
place
3. Evaluate both before and after the program if practical
4. Survey and/or interview one or more of the following :
trainees, their immediate supervisor, their subordinates,
and others who often observe their behavior
5. Get 100 percent response or a sampling
6. Repeat the evaluation at appropriate times
7. Consider cost versus benefits
www.exploreHR.org 16
17. Example of Survey to Measure Behavior Application
Instruction:
The objective of this questionnaire is to determine the extent to which those who
attended the recent program on Leadership have applied the principles and techniques
that they learned there to the job.
Circle the answer that you consider appropriate for each question.
5 = Much more 4 = More 3 = Same 2 = Less 1 = Much less
Time and energy spent after the prgram
compared to time and energy spent before
the program
Understanding and Motivating
1. Trying to understand my subordinates 5 4 3 2 1
2. Listening to my subordinates 5 4 3 2 1
3. Praising good work 5 4 3 2 1
4. Talking with subordinates about
5 4 3 2 1
their family and personal interests
5. Asking my subordinates for their ideas 5 4 3 2 1
6. Applying "Management by Walking Around" 5 4 3 2 1
www.exploreHR.org 17
18. Level 4 – Business Results
• Indicate the extent to which you
think this program has
influenced each of these
measures in your work unit,
Level 4 – department, or business unit:
Business • Productivity
Results • Quality
• Customer Response Time
• Cost Control
• Employee Satisfaction
• Customer Satisfaction
• Other
www.exploreHR.org 18
19. Guidelines for Evaluating Learning
1. Use a control group if practical
2. Allow time for results to be achieved
3. Measure both before and after the program if practical
4. Repeat the evaluation at appropriate times
5. Consider cost versus benefits
6. Be satisfied with evidence if proof is not possible
www.exploreHR.org 19
20. Performance Indicators
HARD DATA INDICATORS
• Downtime duration
• Number of defect products
• Sales volume
• Production unit
• Customer satisfaction index
• Response time to orders
• Number of accidents at work
• Others
www.exploreHR.org 20
21. Performance Indicators
SOFT DATA INDICATORS (intangible impacts)
• Job satisfaction
• Conducive working relationship
• Effective communication
• Stress rate
• Quality in decision-making
www.exploreHR.org 21
22. Example : Measuring Training Results
200.00
Program : TQM
Training
Results after 3 150.00
120 units
months of
training, number
100.00 80 units
of defects
dropped to 80
units/day
50.00
0.00
Before training After training
www.exploreHR.org 22
23. Example : Measuring Training Results
Program : 50.00
Sales Training
40.00
Results after 3
months training,
training 30 units
number of sales 30.00
per salesman 20 units
increase to 30 20.00
units/month.
10.00
0.00
Before training After training
www.exploreHR.org 23
24. Measuring
Return on Investment of
Training
www.exploreHR.org 24
25. Level 5 : Return on Investment of Training
Level 1 - Reaction
Level 2 - Learning
Level 3 – Behavior
Application
Level 4 –
Business Impact
Level 5 – Return
on Investment of
Training
www.exploreHR.org 25
26. Criteria for Selecting Programs for Levels
4 and 5 Evaluation
• Importance of the program in meeting the
organization’s goals
• Cost of the program
• Visibility of the program
• Size of the target audience
• Extent of management interest
www.exploreHR.org 26
27. Benefits of ROI of Training
• Measure contribution
• Set priorities
• Focus on results
• Alter management perceptions of training
www.exploreHR.org 27
28. ROI of Training Model
Isolate the Convert Data
Collect
Data Effects of to Monetary
Training Values
Identify
Intangible Calculate ROI
Benefits of Training
Tabulate
Program
Costs
www.exploreHR.org 28
29. Return on Investment Formula
Net Program Benefits
ROI = X 100
Program Costs
Example :
• Costs per program (25 participants) $ 88,500
• Benefits per program (1st year) $230,625
$ 230,625 – 88,500
ROI = X 100
$ 88,500
ROI = 161 %
www.exploreHR.org 29
30. Collecting Data
• Identify appropriate
Collect performance indicators
Data
• Develop a collection plan
www.exploreHR.org 30
31. Example of Performance Indicators
Output Time
• Units produced • Equipment downtime
• Items sold • Overtime
• Work backlog • Time to project
• New accounts opened completion
• Productivity • Processing time
• Inventory turnover • Repair time
• Etc. • Lost time days
• Etc
www.exploreHR.org 31
32. Example of Hard Indicators
Cost Quality
• Unit costs • Scrap
• Variable costs • Waste
• Overhead costs • Rejects
• Operating costs • Error rates
• Number of cost • Rework
reduction • Product defects
• Etc. • Product failure
www.exploreHR.org 32
33. Example of Performance Results
• Some performance results after training
program:
• Scrap was reduced from 11 % to 7.4 %
• Absenteeism was reduce from 7 % to 3.25 %
• The annual turnover rate was reduced from 30
% to 16 %
• Lost time accidents were reduced 95 %
www.exploreHR.org 33
34. Isolating the Effects of Training
Using
Control Group
Trend
Methods to Lines
Isolate the
Effects of Participants
Training Estimate
Supervisors of
Participants
Estimate
www.exploreHR.org 34
35. Isolating the Effects of Training
Using • A control group arrangement can be
Control used to isolate training impact.
Group • With this strategy, one group
receives training, while another,
similar group does not receive
training.
• The difference in the performance of
the two groups is attributed to the
training program.
www.exploreHR.org 35
36. Isolating the Effects of Training
Trend • Trend lines are used to project the
Lines values of specific output variables if
training had not been undertaken.
• The projection is compared to the
actual data after training, and the
difference represents the estimate
impact of training.
www.exploreHR.org 36
37. Trend Lines Analysis
Actual sales performance
At the beginning of
The
May, a Sales training difference
Volume of Sales
Program session was represents
held the estimate
impact of
training.
Trend Projection
Jan Feb Mar Apr May Jun Jul Aug
www.exploreHR.org 37
38. Isolating the Effects of Training
• This method rests on the assumption that
Participants and
participants (and their supervisors) are
Supervisors of
capable of estimating how much a
Participants
Estimate of performance improvement is related to the
Training’s Impact training program.
• Because their actions have produced the
improvement, participants (and their
supervisors) may have very accurate input
on the issue.
• They should know how much of the
change was caused by applying what they
have learned in the program.
www.exploreHR.org 38
39. Isolating the Effects of Training
• Typical Questions to Estimate :
Participants and
Supervisors of • What percent this improvement
Participants can be attributed to the application
Estimate of
of skills/techniques/knowledge
Training’s Impact
gained in the training program?
• What confidence do you have in
this estimate, expresses as a
percent?
• What other factors contributed to
this improvement in performance?
www.exploreHR.org 39
40. Isolating the Effects of Training
Example of a Participant’s Estimation
Percent Confidence
Factors Which Influenced Improvement Expressed as
No. Improvement Caused by a Percent
1 Training Program 50% 70%
2 Change in Procedures 10% 80%
3 Adjustment in Standards 10% 50%
4 Revision to Incentive Plan 20% 90%
5 Increased Management Attention 10% 50%
6 Other - -
The confidence percentage is multiplied by the
estimate (50 % x 70 %) to produce a usable training
factor value of 35 %
www.exploreHR.org 40
41. Isolating the Effects of Training
Example of a Participant’s Estimation
Percent Confidence
Factors Which Influenced Improvement Expressed as
No. Improvement Caused by a Percent
1 Training Program 50% 70%
2 Change in Procedures 10% 80%
3 Adjustment in Standards 10% 50%
4 Revision to Incentive Plan 20% 90%
5 Increased Management Attention 10% 50%
6 Other - -
• The confidence percentage is multiplied by the estimate (50 % x 70 %) to
produce a usable training factor value of 35 %
• This adjusted percentage is then multiplied by the actual amount of
improvement (post-program minus pre-program value) to isolate the portion
attributed to training
• The adjusted improvement is now ready for conversion to monetary values,
and used in the return on investment
www.exploreHR.org 41
42. Converting Data to Monetary Values
• Steps to Convert Data to Monetary
Converting
Values
Data to
Monetary 1. Focus on a unit of improvement
Values 2. Determine a value of each unit
3. Calculate the change in performance
data
4. Determine an annual amount of change
5. Calculate the annual value of
improvement
www.exploreHR.org 42
43. Steps to Convert Data to Monetary Values
An example to illustrate the steps to convert data to monetary values
Steps Illustration
1. Focus on unit • One grievance reaching step two in the
improvement four-step grievance resolution process
2. Determine a • Using internal experts, the cost of an
value of each average grievance was estimated to be $
unit 6,500 when considering time and direct
costs (V = $ 6,500)
www.exploreHR.org 43
44. Steps to Convert Data to Monetary Values
Steps Illustration
3. Calculate the • Six months after the program was completed,
change in total grievances per month reaching step two
performance declined by ten.
data • Seven of the then grievance reductions were
related to the program as determined by
supervisors (isolating the effects of training)
4. Determine an • Using the six month value, seven per month
annual amount (grievance reductions), yields an annual
for the change improvement of 84 (7 x 12 months)
www.exploreHR.org 44
45. Steps to Convert Data to Monetary Values
Steps Illustration
5. Calculate the • Annual value = 84 x $ 6,500 = $ 546,000
annual value of
improvement
www.exploreHR.org 45
46. Tabulating Cost of the Program
Tabulating • Tabulating the costs involves
cost of the monitoring or developing all of the
program related costs of the program
targeted for the ROI calculation.
www.exploreHR.org 46
47. Tabulating Cost of the Program
• Cost components that should be
Tabulating included are :
cost of the
program • The cost to design and develop
the program
• The cost of all program materials
provided to each participant
• The cost for facilitator
• The cost of the facilities of the
training program
www.exploreHR.org 47
48. Tabulating Cost of the Program
• Cost components that should be
Tabulating
included are :
cost of the
program • Travel, lodging, and meal costs
for the participants
• Salaries, plus employee benefits
of the participants who attend
the training
• Administrative and overhead
costs of the training function,
allocated in some convenient
way
www.exploreHR.org 48
49. An Example to Illustrate ROI analysis
Average Weekly Sales
Post Training Data
Weeks after
training Trained Group Control Group
1 US$ 9,723 9,698
2 9,978 9,720
3 10,424 9,812
13 13,690 11,572
14 11,491 9,683
15 11,044 10,092
Average for weeks 12,075 10,449
13, 14, 15
www.exploreHR.org 49
50. An Example to Illustrate ROI analysis
Annualized Program Benefits
Data US$
Average weekly sales
Trained Group 12,075
Control Group 10,449
Increase 1,626
Profit Contribution from Training 2 % 32.50
(Training Impacts)
Total weekly improvement 1,495
(32.5 x 46 participants)
Total annual benefit 71,760
($ 1,495 x 48 weeks)
Note : 46 participants were still in job after 3 months
www.exploreHR.org 50
51. An Example to Illustrate ROI analysis
Cost Summary : An Illustration
Cost US$ ROI (%):
Facilitation fees 11,250
Program materials : $ 35 x 46 1,610 $ 71,760 - $ 29,090
Meals : 3 days x $ 28 x 46 1,288
= x 100
$ 29,090
Participant salaries plus benefit (35 %) 12,442
Coordination and Evaluation 2,500 = 146 %
Total Cost 29,090
www.exploreHR.org 51
53. How Effective is Your Training Program?
Broad and Newstrom (1992) report studies have
shown less than 30% of what is actually taught
transfers to the job in a way that enhances
performance.
performance
Source : Broad, M., & Newstrom, J. W. (1992). Transfer of training: Action packed
strategies to ensure high payoff from training investments. Reading, MA: Addison-
Wesley.
www.exploreHR.org 53
54. Source of Barriers to Training Transfer
• Lack of reinforcement on the job
• Interference from immediate (work) environment
• Nonsupportive organizational culture
• Trainees’ perception of impractical training programs
• Separation from the inspiration or support of the trainer
www.exploreHR.org 54
55. The Transfer Partnership
Trainee Trainee recognizes need for
new skills
Trainer designs and/or
Trainee Trainer
delivers learning experiences
Manager supports learning
Trainee Trainer
and application on the job
Manager
www.exploreHR.org 55
56. The Transfer Matrix
Time Periods
Before During After
Manager
Role Players
Trainer
Trainee
www.exploreHR.org 56
57. Before Training
• Build transfer of training into supervisory
Manager performance standards
• Collect baseline performance data
• Involve supervisors and trainees in needs
analysis process
• Involve trainees in program planning
• Brief trainees on the importance of the training
(course objective, content, process, and
application on the job)
• Review instructional content and materials
• Plan to participate in training sessions
• Encourage trainees attendance at all sessions
www.exploreHR.org 57
58. Before Training
• Align the training plan with the organization's
Trainer strategic plan
• Systematically design instruction
• Provide proactive opportunities
• Design a peer coaching component for the
program and its follow-up activities
www.exploreHR.org 58
59. Before Training
• Provide input into program planning
Trainee • Actively explore training options
• Participate in advance activities
www.exploreHR.org 59
60. During Training
• Prevent interruptions
Manager • Transfer work assignment to others
• Monitor attendance and attention to training
• Recognize trainee participation
• Participate in transfer action planning
• Review information on employee in training
• Plan assessment of transfer of new skills to the
job
www.exploreHR.org 60
61. During Training
• Develop application-oriented objectives
Trainer • Answer the “WIIFM” question
• Manage the unlearning process
• Provide realistic work-related tasks
• Give individualized feedback
• Provide job performance aid
www.exploreHR.org 61
62. During Training
• Maintain an ideas and application notebook
Trainee • Participate actively
• Form support groups
• Plan for applications
• Create behavioral contracts
www.exploreHR.org 62
63. After Training
• Plan trainees’ reentry
Manager • Provide opportunities to practice new skills
• Have trainees participate in transfer-related
decisions
• Reduce job pressures initially
• Give positive reinforcement
• Schedule trainee briefings for co-workers
• Set mutual expectations for improvement
• Arrange proactive (refresher) sessions
• Provide and support the use of job aids
www.exploreHR.org 63
64. After Training
• Provide follow-up support
Trainer • Conduct evaluation surveys and provide
feedback
• Develop and administer recognition system
• Provide refresher/problem-solving sessions
www.exploreHR.org 64
65. After Training
• Practice self-management
Trainee • Review training content and learned skills
• Develop a mentoring relationship
• Maintain contact with training ‘buddies’
www.exploreHR.org 65
66. Recommended Further Readings
1. Donald Kirkpatrick, Evaluating Training Programs : The Four Levels,
Berrett-Koehler Publishers
2. Jack J. Phillips and Patricia Phillips, In Action : Measuring Return On
Investment, American Society for Training & Development
3. Mary Broad and John Newstrom, Transfer of training: Action Packed
Strategies to Ensure High Payoff from Training Investments, Addison-
Wesley.
www.exploreHR.org 66
