This panel addresses the assessment tools that should be used, how agencies can implement an assessment strategy that results in the best hires in a timely and cost efficient manner and how agencies can measure and report on the speed and quality of their hiring processes.

1. The Path to Improving and
Measuring Hiring Results

2. Today’s Presenters
Beverly Dugan, Ph.D.
Vice President, HumRRO Mike Heil, Ph.D.
Assistant Vice
President, Aon Hewitt
Elaine Pulakos, Ph.D.
Chief Operating Officer,
PDRI
2

3. Agenda
• Federal Hiring Reform
• How Can Objective Assessments Help Meet These Requirements?
• Use of Objective Assessments in the Hiring Process
• Selecting or Developing Effective Assessments
• Evaluating Hiring Process and Outcomes
• Use of Assessments in Federal Hiring
• Questions and Answers
3

4. Federal Hiring Reform
4

5. Federal Government Hiring
• Unprecedented numbers of applicants
• Current systems too confusing and time consuming
– Baseline for hiring is 130 days
• Technology helps efficiency but not quality
• Ineffective hiring assessments are “weakest link”
– Do not effectively screen and reduce applicant pools
– Do not identify best qualified
– Unqualified candidates getting through
• Partnership for Public Service, 2010
5

6. President Obama’s Hiring Reform Memorandum
• Improve hiring process and quality of hires
– Eliminate KSA essays
– Reduce time to hire mission-critical and commonly
filled positions
– Measure speed and effectiveness of hiring process
– Measure quality of hires
– Analyze causes of hiring problems
– Implement actions to reduce problems
Quality, Speed, and Measurement are
the Key Requirements of Hiring Reform
6

7. How Can Objective Assessments Help
Meet These Requirements?
7

8. Why Not Use Self-Report Information?
Applicant Prepared Narratives Multiple Choice Questions
Preparing written documents Which response best describes your
experience preparing written documents?
In my previous jobs, I have had a great deal of
experience preparing written documents. I 1. No experience.
served as a research assistant. My primary
job duty was to conduct literature reviews and 2. In training situations but not on a job.
write the results of these. Oftentimes, there
were 100 or more research articles that I had 3. Under supervision, prepared documents in
to review, some of which contained which the content was determined by others.
contradictory findings. I have to look through 4. Independently prepared documents in which
the articles for common findings and offer conclusions were based on my analyses.
explanations base don my analyses as to why
different studies may have obtained what 5. Independently prepared complex documents
appeared to be contradictory results. This that integrated information from many
required collecting a large amount of sources to decide on operations or policy.
information, organizing it in a ….
Narratives no longer permitted as first step in
hiring process
8

9. Why Not Use Multiple Choice Self-Reports?
• Often Inaccurate • Time Consuming
– Some are legends in their – Takes a lot of work to develop
own minds good items
– Others underestimate – Customized for each vacancy
themselves – Often yields unwieldy item banks
• Manager Complaints
• Can Disadvantage
Applicants Who – Do not sufficiently reduce
applicant pools to reasonable
– Lack access to experience number to interview
– Truthfully report lack of – Unqualified applicants on certs
experience
– Qualified applicants not on certs
9

10. Why Not Use Resumes?
• Impractical when there are large numbers of applicants
• Even if resumes could be screened efficiently (e.g.,
automated tool), they are not effective hiring tools
– Also rely on self-report information
– Information not standardized across applicants so difficult
to assess
10

11. Why Use Objective Assessments?
Objective Assessments:
Systematic, accurate measurement of a job-relevant
skill, knowledge area, or other competency
Tell Me vs. Show Me
11

12. Use of Objective Assessments in the
Hiring Process
12

13. Placement of Assessments in Hiring Process
Job Posting
Application
Remote Assessments
Automated and Unproctored
On-site Proctored Verification
Assessments and Interview
Assessment
Great Hiring Decision
13

14. Assessment Administration Options
1. Application-based • Slow, resource-intensive,
recruitments, proctored outdated, poor candidate
paper-and-pencil experience
assessments per job
opening
2. Resume-based • Fast, but still resource-
recruitments, proctored web- intensive, and poor
based assessments on a candidate experience
continuous basis
3. Web resume-based • Very fast, low-cost, low-
recruitments, unproctored overhead, better candidate
web-based assessments on experience
a continuous basis
14

15. Unproctored Testing
Potential Risks Potential Benefits
• Cheating • Reach larger and more
• Applicant identity verification diverse applicant pool
• Test content exposure • Reduced travel cost for
• Unstandardized testing candidates
environments • Reduced staffing cost for
• End-user technical issues Assessment Center
• Reduced recruitment
Cheating, content exposure, and time/cost
applicant identity all mitigated
with security measures
• “Cutting Edge” Image
15

16. Selecting or Developing Effective
Assessments
16

17. Decide What Needs to be Assessed
• Competencies
– The knowledge, skills, abilities, and other attributes required for
effective performance on the job
• Competency Model
– Collection of required competencies in an occupation or
organization
17

18. Example Competencies
Critical Customer Productive
Auditing Interpersonal Communication
Thinking Service Engagement
Reading
Writing
Reasoning Self Management Presenting Information Orally
Math Reasoning Flexibility Persuading and Influencing
Problem Solving Stress Tolerance
Decision Making
Planning/Evaluate
Strategic Thinking
18

19. Identify the Most Important Competencies
Administer
validation survey
to job incumbents
to identify
important
competencies
19

20. Select Assessments to Measure Competencies
Types of Competency Assessments
Most Important
Competencies Situational Work Technical
Cognitive Work Styles Judgment Writing Simulations Proficiency
Reading
Reasoning
Self Management
Interpersonal
Flexibility
Stress Tolerance
Problem Solving
Decision Making
Writing
Auditing
Planning
20

21. Cognitive Assessments
Measure mental
competencies, such as
critical thinking, math,
reading, problem
solving
To protect against
publishing of items,
computer selects
different items for
each applicant
Virtually no two
applicants receive
same items
21 21

22. Work Styles Assessments
Provides assessment of
Applicants choose soft skill competencies, Difficult to cheat
which item is most such as self management, because scoring is
descriptive of them initiative, flexibility, not obvious
teamwork, stress
tolerance, and others
22
22

23. Situational Judgment Assessments
Realistic problems
like encountered on
job
Applicants select
most and least
effective way to
handle situation
Difficult to cheat
because scoring is
not obvious
Positive applicant
reactions
23

24. Writing Assessments
Compose free text response to
question
Please describe why you are a good fit for this role,
including relevant background experiences, and Response automatically
what you feel is your greatest strength. scored on clarity, grammar,
persuasion, content, and
others
24

25. Work Simulation Assessments
Realistic problems like
encountered on job
Can assess many
competencies
Applicant reactions
uniformly positive due to
high job relevance
25

26. Technical Proficiency Assessments
Information
Technology
Assessments
MS Office
Assessments
26

27. Job Interview Assessments
Job interviews are
also assessments
For reliable and valid
assessment:
One set of
competency questions
should be used for
each vacancy and
asked of all
candidates
All candidates should
be evaluated against
standard rating criteria
27

28. Evaluating Hiring Process and
Outcomes
28

29. Getting Started
• Define measures
– Hiring reform requirements
– Agency strategic goals
– Most critical impact areas (reduce costs, reduce turnover, etc.)
• Determine measurement strategy for stable, credible
results
– Collect baseline measures before changing your process
– Collect measures at regular intervals during and after
implementation
29

30. Hiring Process Evaluation
• Process measures answer the question: How well is the new hiring
process working?
• E2E metrics focus on speed and
effectiveness of hiring
– % employees hired within 80 days
– % applicants given information on status of
application
– % new employees reporting satisfaction with
hiring process
– % new employees reporting regular
communication during hiring process
• Other example hiring process metrics
– Total time HR devotes to hiring process
– Total time manager devotes to hiring process
– Manager satisfaction with hiring process
30

31. Improving the Hiring Process
• Technology is key
“Many problems agencies face are information technology-
related,” said Tim McManus, vice president of education
and outreach at the nonprofit Partnership for Public
Service. For example, agencies have in place automated
systems to sort through knowledge, skills and abilities
essays, but have not yet developed assessments based on
shorter questionnaires and specific positions and
qualifications.
“How do you assess in a way that's automated?” he said. “It's hugely
important in this conversation because of the volume of applicants agencies
get -- hundreds and even thousands of applications for a single position.”
Justice, Treasury and Veterans Affairs departments all identified delays
caused by limited or antiquated IT systems.
Excerpt from: Hiring reform deadline creates some headaches for agencies
By Emily Long elong@govexec.com, September 16, 2010
31

32. How Technology Improves the Hiring Process
• Automated tools enable
applicants to apply
anywhere, anytime
• Easy, efficient process
attracts a larger number of
qualified applicants
• Real time, immediate
category scores
– Not Qualified, Qualified, or Best
Qualified
32

33. Caveat from MSPB
• “Plan carefully when
implementing automated tools
designed to support the hiring
process. Do not assume that
technology alone will fix already-
broken processes. For instance,
automating an assessment tool
that is a low predictor of job
success will not improve the
tool’s predictive ability.”
Excerpted from: MSPB report “Reforming federal hiring – beyond faster and cheaper” –
September 1, 2006
33

34. Example Quality of Hire Measures
• Productivity
• Cost savings
• Retention
• Time to promotion
• Training time and costs
• Manager satisfaction
34

35. Example Results from Private Sector
35

36. Increased Efficiency and Quality
34% more reach quality goals Call center agents
$4 million realized from improved
Logistics staff
work efficiency
18 days of additional productivity
At home agents
per employee annually
EFFICIENCY
35% reduced average handle time Call center agents
AND QUALITY
OUTCOMES 43% more likely to pass exam Claims adjustors
8% more calls completed Insurance agents
40% reduction in handle time Call center agents
18% more effective and 28% more
Claims adjustors
likely to hire again
36

37. Reduced Costs
46% reduced turnover Nurses
$2 million saved in reduced repeat
Technicians
service calls
23% reduced turnover Call Center Agents
COST 33% reduced turnover Reservations Agents
REDUCTION
OUTCOMES 83% reduced turnover Call Center Agents
78% million in reduced shrinkage Store Employees
50% reduced turnover Production Workers
63% reduced turnover Call Center Agents
37

38. Improved Leadership
300% higher performance ratings Frontline Managers
57% more likely to be promoted Branch Managers
8% higher sales and profit Territory Managers
LEADERSHIP 59% higher employee engagement Branch Managers
EFFECTIVENESS
OUTCOMES 30% more likely to be promoted Senior Managers
38

39. Effective Hiring Outcomes Recap
Process
Hiring System Effectiveness
and Speed
Technology
Improvements Time to Hire
Applicant
Process Satisfaction
Improvements
Applicant
Communication
Objective
Assessments Quality of Hire
Competencies
Effective Work Productivity
Hires with job-related Behaviors
Knowledge
abilities, skills Turnover
Skills and traits
Fit Cost Savings
Work Styles Engagement
Show Me vs. Manager
Tell Me Satisfaction
39

40. Use of Objective Assessments in
Federal Hiring
40

41. Federal Agencies Using Objective Assessments
• ATF
• DEA
• Department of Commerce
• Department of State
• DoD – uniformed services
• DHS - CBP
• FAA
• FBI
• IRS
• Intelligence Community
• OPM
• Postal Service
• TSA
• Veterans Benefit Administration
41

42. Federal Agencies Using Unproctored
Assessment
• Intelligence Community
– Assessments include writing, foreign language, work styles,
computer skills, English skills, math, proofreading, typing speed
and accuracy, among others
• OPM
– Assessments include reading, reasoning, math reasoning, work
styles, and situational judgment
• U.S. Postal Service
– Assessments including situational judgment and personal
characteristics
42

43. Q&A
Questions

44. Why Objective Assessments “Aren’t for Me”
• I’m only hiring one person for one position
– But …agency may be hiring hundreds or thousands
– Competency assessments (e.g., writing) useful across jobs
– Can get thousands of applicants for one job – how do you find
the ONE?
• Too complex, time consuming, expensive
– Quality assessments available from OPM
– Commercial assessments available
– Usually easier to implement competency assessments than self
report experience items
44

45. How Does Category Rating Work?
• Purpose
– Increases number of qualified applicants from which to select
– Provides flexibility to select without regard to “rule of three”
– Preserves Veterans preference rights
• Process
– Evaluate minimally qualified applicants
– Place all minimally qualified applicants into two or more pre-
determined quality categories
– Combine quality categories if fewer than three in highest category
– Send names of all in highest category to hiring official
– Make selections form highest quality category
45

46. How Does Veterans Preference Work Under
Category Rating?
• Applied after assessment and category placement
• No longer assign points
• Preference eligibles receive absolute preference
– Listed ahead of non-preference eligibles in the category
• Preference eligibles with compensable service-connected
disability of 10% must be placed in highest category
(except for scientific or professional positions GS-9 and
above)
• Agencies must request to pass over a preference eligible
46