Presentation for Council of Chief Librarians, CA Community Colleges

1. BUILDING A
CULTURE OF
ASSESSMENT
WHAT I'VE LEARNED
FROM RESEARCH, TRIAL,
AND (A LOT OF) ERROR
Meredith Farkas
Portland Community College

2. ABOUT ME
➤ Faculty Librarian at Portland Community College (OR)
➤ Previously:
➤ Head of Instructional Services, Portland State University
➤ Head of Instructional Initiatives, Norwich University
➤ Significant research and publishing on what it takes to build
an assessment culture
➤ Former psychotherapist interested in what it takes to really
change culture

7. “A culture of assessment is an
organizational environment in which
decisions are based on facts, research,
and analysis, and where services are
planned and delivered in ways that
maximize positive outcomes and
impacts for customers and stakeholders.
-Amos Lakos, Shelley Phipps, and Betsy Wilson
Lakos, A. & Phipps, S. E. (2004). "Creating a culture of assessment: A catalyst
for organizational change." portal: Libraries and the Academy, 4(3), 345-361.

8. MY DEFINITION
➤ Assessment is the norm - a
regular part of practice
➤ User-focused culture
➤ Learning and curiosity driven
➤ Assessment is done for
improvement
➤ Changes and decisions are
made based on what is learned
➤ Planning for new services
always includes planning for
assessment

9. ASSESSMENT AT MANY INSTITUTIONS
➤ Focused on doing what is
needed to satisfy accreditors or
administrators
➤ Coercive and top-down
➤ Little support
➤ Coalition of the willing
➤ Data is gathered, but rarely
used
➤ Not seen as an integral part of
teaching and learning
➤ Often a lot of cynicism around
assessment

10. MY EXPERIENCE:
ADVENTURES WITH
COORDINATOR SYNDROME

11. “
The word ‘Coordinator’ in a position title. Beware it, especially in
combination with a new or non-traditional job niche…
The problem with these jobs is that as often as not, there’s nothing
actually to coordinate. No budget. No dedicated staff. No
established service. ‘Coordinate’ all too often means ‘try to establish
a beachhead by begging your new colleagues to vouchsafe you a few
minutes of their time now and then, knowing that their supervisors
won’t tell them to and you have no authority whatever to demand
anything of them.’
Oh, but the buck will stop with you, Coordinator, on all matters
regarding your service. Be aware, also, that because what you’re
doing is new, you must hit a home run with it, quickly; slow and
steady gains will not do. If you don’t, two damaging assumptions
arise: that the service is a waste of effort, and that you’re a lousy
librarian. You’re a coordinator now; congratulations!
-Gavia Libraria, “The C-word"

12. YOU SAY YOU NEED SUPPORT? I’VE GOT
YOUR SUPPORT RIGHT HERE.

13. ➤ Farkas, Meredith G. "Building and sustaining a culture of
assessment: best practices for change leadership." Reference services
review 41.1 (2013): 13-31.
➤ Farkas, Meredith G., and Lisa J. Hinchliffe. "Library faculty and
instructional assessment: Creating a culture of assessment
through the high performance programming model of
organizational transformation." Collaborative Librarianship 5.3
(2013): 177-188.
➤ Farkas, Meredith Gorran, Lisa Janicke Hinchliffe, and Amy Harris
Houk. "Bridges and barriers: factors influencing a culture of
assessment in academic libraries." College & Research Libraries
(2014).
➤ Farkas, Meredith. "Accountability vs. Improvement: Seeking
Balance in the Value of Academic Libraries Initiative." OLA
Quarterly 19.1 (2014): 4-7.

15. “For a scholarship of assessment to thrive, we
must align faculty culture, institutional
structures, and leadership for change.
The importance of this point cannot be
overstated. A meaningful assessment program
is more than just a new activity to be
undertaken, it is a change in how we think
about what we do in higher education.
-Don Haviland “Leading Assessment, from Faculty
Resistance to Faculty Engagement.”

16. THESE STUDIES SUGGEST SPECIFIC FACILITATING/HINDERING FACTORS
1. Oakleaf, M. & Hinchliffe, L. J. (2008). Assessment cycle or circular file:
Do academic librarians use information literacy assessment data? In
Proceedings of the 2008 Library Assessment Conference: Building Effective,
Sustainable, Practical Assessment.
2. Ndoye, A. & Parker, M. A. (2010). "Creating and sustaining a culture of
assessment." Planning for Higher Education, 38(2), 28-39.
3. Hiller, S., Kyrillidou, M., & Self, J. (2008). "When the evidence is not
enough: Organizational factors that influence effective and successful library
assessment." Performance Measures and Metrics, 9(3), 223-230.
4. Wright, S. & White, L. (2007). SPEC Kit 303, Library Assessment.
Washington, D.C.: Association of Research Libraries.

17. FACILITATING FACTORS SUGGESTED IN THE LITERATURE
➤ Assessment is a library priority (4)
➤ Commitment from library leadership (1, 3, 4)
➤ Commitment from faculty/staff (2, 3, 4)
➤ Organizational culture (3, 4)
➤ Staff with needed expertise and time (1, 3, 4)
➤ Staff development (4)
➤ Evidence-based decision-making (1, 2, 3, 4)

18. DEMONSTRATING THAT ASSESSMENT IS A
LIBRARY PRIORITY
➤ Shared understanding of
assessment exists within the
organization
➤ Clear expectations or concrete
goals for assessment
➤ Building an assessment plan

19. COMMITMENT FROM ADMINISTRATION
➤ Articulates the value of assessment
➤ Assessment is built into ALL planning
➤ Assessment is built into reward structures
➤ Supports library faculty/staff engaging in assessment
➤ Uses assessment data to inform decisions

20. EXPERTISE AND TIME
➤ Staff need dedicated
time to learn, try, fail,
and learn more
➤ Find ways to capitalize
on expertise outside of
the library

21. STAFF DEVELOPMENT
➤ Don’t just send one person to
a conference
➤ Opportunities for staff to learn
together
➤ Build a culture of reflective
practice and learning
communities

22. WHY DON’T LIBRARIANS USE
ASSESSMENT RESULTS?
➤ Too busy with other work
➤ Won’t be rewarded for assessment
work
➤ Lack of expertise
➤ Lack of centralized support within
the library
➤ Not valued by their library
Oakleaf and Hinchliffe. (2008).
“Assessment Cycle or Circular File: Do
Academic Librarians Use Information
Literacy Assessment Data?” Proceedings
of the 2008 Library Assessment Conference.

23. FACILITATING FACTORS - EXPANDED
1. Clear expectations for assessment
in the library
2. Assessment is a priority of
administration
3. Library has adopted learning
outcomes
4. Library has an assessment plan
5. Library has a shared
understanding of the purpose of
assessment
6. Library faculty/staff are supported
in doing assessment
7. Education and training about
assessment are supported
8. Library has assessment expertise
in-house
9. Library has systems/technologies
to support assessment
10. Library leadership offers explicit
support to get staff doing assessment
11. Data is available to interested
parties
12. Leadership uses assessment data
in decision-making
13. Librarians use results to improve
practice
14. Library culture is user-focused

24. ULTIMATELY, THIS IS
ABOUT CHANGING
CULTURE

26. JOHN KOTTER’S LEADING CHANGE: THE ORGANIZATIONAL CHANGE BIBLE

27. CREATE A SENSE OF URGENCY
➤ Urgency must be driven by
opportunities, not by fear
➤ Appeal to the things that
your audience cares about
(student learning, being
better instructors, etc.)
➤ Sometimes urgency comes
from outside (mandates
from administrators,
outside agencies, etc.)

28. FORM A GUIDING COALITION
(NOT JUST A COORDINATOR)
➤ Form a group to lead and support
assessment efforts
➤ Membership based on skills,
interest, and politics
➤ They will need admin support
➤ Examples:
➤ Northwestern University
libraryassessment.org/
bm~doc/
4macayeallightingtalk.pdf
➤ Portland Community College
www.pcc.edu/edserv/learning-
assessment/LACAboutUs.html

29. MacAyeal & Ryan. (2014). Support System: Establishing, Sustaining,and
Growing a Frameworkfor Assessment. Library Assessment Conference.

30. MacAyeal & Ryan. (2014). Support System: Establishing, Sustaining,and
Growing a Frameworkfor Assessment. Library Assessment Conference.

31. PCC LEARNING ASSESSMENT COUNCIL

32. CREATE AND COMMUNICATE A
VISION
➤ Common understanding of
the purpose of our
assessment program
➤ Develop an assessment
plan
OR
➤ Develop concrete
expectations for what
progress in assessment
looks like

33. EMPOWER OTHERS TO ACT ON
THE VISION
➤ Remove barriers
➤ Time
➤ Lack of knowledge/education
on assessment
➤ Fear of failure
➤ Incentivize assessment
➤ Grants, rewards, recognition
➤ Give staff/faculty the freedom to
determine what and how they
will assess
➤ Demonstrate that assessment
results will be used

34. CREATE SHORT-TERM WINS (OR “THE
PERFECT IS THE ENEMY OF THE GOOD”)
➤ Start small and simple
➤ Assessments that are easy
to do but yield valuable
learning
➤ What are we curious about?
➤ Debrief together and learn
from each experience
➤ Talk about changes made
based on assessment results

35. CONSOLIDATE IMPROVEMENTS AND
INSTITUTIONALIZE NEW APPROACHES
➤ Develop formal mechanisms
for reporting and sharing
assessment results
➤ Build mechanisms for
discussing assessment results
within the library
➤ Demonstrate evidence-based
decision-making
➤ Build assessment into
planning
➤ Build assessment into reward
structures (tenure?)

36. THERE ARE LIMITATIONS WITH
THE LITERATURE WE RELY ON TO
MAKE CLAIMS ABOUT BUILDING
AN ASSESSMENT CULTURE

37. 1. Oakleaf, M. & Hinchliffe, L. J. (2008). Assessment cycle or circular file:
Do academic librarians use information literacy assessment data? In
Proceedings of the 2008 Library Assessment Conference: Building Effective,
Sustainable, Practical Assessment.
2. Ndoye, A. & Parker, M. A. (2010). "Creating and sustaining a culture of
assessment." Planning for Higher Education, 38(2), 28-39.
3. Hiller, S., Kyrillidou, M., & Self, J. (2008). "When the evidence is not
enough: Organizational factors that influence effective and successful library
assessment." Performance Measures and Metrics, 9(3), 223-230.
4. Wright, S. & White, L. (2007). SPEC Kit 303, Library Assessment.
Washington, D.C.: Association of Research Libraries.
Survey sent
to an infolit
listserv
Based on their experience with
research libraries that want to
build an assessment culture
Sent to an assessment
listserv
Survey of ARLs only

38. Our goal:
A systematic survey of culture of assessment and
investigation of factors that facilitate or hinder
libraries.
A survey of libraries, not librarians

39. THE SURVEY
➤ 60 Question Survey
➤ 10 descriptive questions about the institution or the
individual responding for the institution
➤ 47 yes/no questions
➤inquiry about presence/absence of 16 facilitating factors
named in other publications
➤does the presence/absence facilitate or hinder culture of
assessment
➤ 3 open-ended questions about what most facilitates or hinders
creation of a culture of assessment
➤ Respondents presented with different questions depending on
their answers to presence/absence of facilitating factors
(total questions for each respondent = 41 or 42)

40. FACILITATING FACTORS
1. Campus-wide assessment initiative
2. Library involvement in campus-
wide assessment initiative
3. Clear expectations for assessment
in the library
4. Assessment is a priority of
administration
5. Library has adopted learning
outcomes
6. Library has an assessment plan
7. Library has a shared understanding
of the purpose of assessment
8. Library faculty/staff are supported
in doing assessment
9. Education and training about
assessment are supported
10. Library has assessment expertise in-
house
11. Library has systems/technologies to
support assessment
12. Library leadership offers explicit
support to get staff doing assessment
13. Data is available to interested parties
14. Leadership uses assessment data in
decision-making
15. Librarians use results to improve
practice
16. Library culture is user-focused

42. Stage 1: Winter 2013
Libraries at all BA/MA/PhD-granting
institutions
Received a 42% response rate

43. Results presented at ACRL 2013 and
published in College & Research Libraries
Farkas, Meredith Gorran, Lisa Janicke Hinchliffe, and Amy Harris
Houk. "Bridges and Barriers: Factors Influencing a Culture of
Assessment in Academic Libraries." College & Research Libraries
76.2 (2015): 150-169.

44. BUT COMMUNITY
COLLEGES ARE DIFFERENT

45. Stage 2: Spring 2014
Libraries at all U.S. public community
colleges
Received a 32% response rate (n=292)

46. WOULD YOU DESCRIBE YOUR LIBRARY AS HAVING A
CULTURE OF ASSESSMENT, WHERE ASSESSMENT IS A
REGULAR PART OF YOUR INSTITUTIONAL PRACTICE?

47. THIS IS HIGHER THAN AT FOUR YEAR AND ABOVE
INSTITUTIONS?

48. IS THERE A CAMPUS-WIDE ASSESSMENT INITIATIVE AT
YOUR COLLEGE?
Also higher than at 4 year and above institutions (84.9%)
Only 80% of these libraries are involved in the campus-wide
assessment initiative

49. IS ASSESSMENT A PRIORITY OF LIBRARY
ADMINISTRATION?
Higher than at 4 year and above institutions (77.8%)

50. IS THERE A SHARED UNDERSTANDING OF THE PURPOSE OF
ASSESSMENT IN YOUR LIBRARY?
Higher than at 4 year and above institutions (54.3%)

51. ARE THERE CLEAR EXPECTATIONS FOR ASSESSMENT IN
YOUR LIBRARY?
Higher than at 4 year and above institutions (45.7%)

52. DOES YOUR LIBRARY HAVE AN ASSESSMENT PLAN?
Higher than at 4 year and above institutions (40.6%)

53. ARE LIBRARY FACULTY/STAFF ADEQUATELY SUPPORTED IN
THEIR ASSESSMENT WORK?
Higher than at 4 year and above institutions (55.8%)

54. DOES YOUR LIBRARY HAVE THE NECESSARY SKILLS IN-HOUSE
TO DEVELOP AND CONDUCT MEANINGFUL ASSESSMENTS AND
ANALYZE THE RESULTS?
Higher than at 4 year and above institutions (56%)

55. DOES THE LIBRARY LEADERSHIP USE ASSESSMENT DATA
SYSTEMATICALLY IN DECISION-MAKING AND PLANNING?
Higher than at 4 year and above institutions (58.5%)

56. DOES THE LIBRARY STAFF USE ASSESSMENT DATA
SYSTEMATICALLY IN DECISION-MAKING AND PLANNING?
Higher than at 4 year and above institutions (71%)

57. ON WHICH FACILITATING FACTORS DID COMMUNITY
COLLEGES SCORE SIGNIFICANTLY (P<0.05) HIGHER?
➤Is there a shared understanding of the purpose of
assessment in your library?
➤Are library faculty/staff adequately supported in their
assessment work?
➤Does the library leadership use assessment data
systematically in decision-making and planning.

58. WHAT FACTORS ARE
ASSOCIATED WITH HAVING A
CULTURE OF ASSESSMENT?

59. REGIONAL ACCREDITING ASSOCIATION IS ASSOCIATED
WITH A CULTURE OF ASSESSMENT

60. TENURE STATUS AND FACULTY STATUS IS ASSOCIATED
WITH A CULTURE OF ASSESSMENT
p=0.02
This was not the case in the study of four year and above institutions

61. HAVING A CAMPUS-WIDE ASSESSMENT INITIATIVE IS
ASSOCIATED WITH A CULTURE OF ASSESSMENT
P<0.001

62. ALL OF THE FACILITATING
FACTORS ARE SIGNIFICANTLY
ASSOCIATED WITH A CULTURE
OF ASSESSMENT

63. FACTORS WHOSE PRESENCE IS MOST STRONGLY
ASSOCIATED WITH A CULTURE OF ASSESSMENT
➤Clear expectations for assessment in the library (90%
who have it have an assessment culture)
➤Assessment plan (82%)
➤Shared understanding of the purpose of assessment in
the library (81%)
➤Library leadership uses assessment data (78%)
➤Library leadership offers explicit support (77%)

64. FACTORS MOST STRONGLY ASSOCIATED WITH NOT HAVING
A CULTURE OF ASSESSMENT
➤Assessment is not a priority of administration (77%
who report this do not have an assessment culture)
➤Librarians do not use assessment data (72%)
➤Assessment data is not available to interested parties
(74%)
➤Library leadership does not use assessment data
(70%)
➤No shared understanding of the purpose of assessment
(69%)

66. WHAT FACTORS ARE MOST OFTEN PERCEIVED AS
FACILITATING?
➤ Clear expectations for assessment in the library (94%
who perceive that this facilitates their assessment
culture have an assessment culture)
➤ Library has an assessment plan
➤ Librarians use assessment data to improve practice
➤ Library leadership uses assessment data
➤ Existence of learning outcomes

67. OPEN-ENDED RESPONSES: WHAT ARE THE MOST
IMPORTANT FACILITATING FACTORS?
➤Institution-wide emphasis on assessment
➤Administrative priority and/or support
➤Accreditation
➤Organizational culture
➤Information literacy as a campus-wide outcome

68. OPEN-ENDED RESPONSES: WHAT MOST HINDERS
CREATING AN ASSESSMENT CULTURE?
➤Lack of sufficient staffing
➤Lack of time
➤Faculty or staff have not made assessment a priority –
either due to lack of interest or resistance to
assessment
➤Lack of expertise/access to training
➤Institution doesn’t prioritize assessment

69. REFLECTIONS

70. “CULTURE OF ASSESSMENT” IS IN THE EYE OF THE BEHOLDER
http://www.flickr.com/photos/kylemay/1573158163/

71. WITHOUT AN INSTITUTIONAL COMMITMENT TO ASSESSMENT,
A CULTURE OF ASSESSMENT IS FAR LESS LIKELY TO EXIST
Institutions with a
campus-wide
assessment initiative
66% of libraries report
having an assessment
culture
Institutions with no
campus-wide
assessment initiative
36% of libraries report
having an assessment
culture

72. THERE ARE SIGNIFICANT ASSOCIATIONS WITH TENURE AND
FACULTY STATUS THAT ARE NOT PRESENT AT FOUR YEAR AND
ABOVE INSTITUTIONS

73. HAVING A PLAN IS CRITICAL
Fewer than half of libraries...
➤ Have clear expectations for assessment (52%)
➤ Have an assessment plan (45%)
Of libraries reporting clear expectations and an
assessment plan, 92% report having a culture of
assessment (versus 32% of those who report the
absence of both)

74. HOW ADMINISTRATION PROMOTES ASSESSMENT MAKES A
DIFFERENCE

75. Of those indicating assessment is a priority of administration
74% report having a culture of assessment
Of those indicating administration uses assessment data in decision-
making
78% report having a culture of assessment
Of those indicating administration adequately supports faculty/staff in
assessment work
69% report having a culture of assessment
Of those indicating a library has all three factors
88% report having a culture of assessment

76. HAVING INSTRUCTION AND ASSESSMENT LEADERS ALSO
HELPS

77. Of those academic libraries with an instruction coordinator
74% report having a culture of assessment
Of those with an assessment coordinator
87% report having a culture of assessment
Of those with an instruction committee
71% report having a culture of assessment
Of those with an assessment committee
87% report having a culture of assessment
Of those with none of these positions
57% report having a culture of assessment

78. MOST COMMUNITY COLLEGE
LIBRARIES ARE
ASSESSMENT LEADERS

80. CONSIDER
➤ What has helped your library move toward (or achieve!) a
culture of assessment?
➤ What has hindered your library in moving toward a culture of
assessment?
➤ Which of these facilitating factors seem most important for
you to work toward at your institution?

81. FUTURE RESEARCH?

83. Questions? Comments? Challenges?
Meredith Farkas
Portland Community College
Email: meredith.farkas at pcc.edu
Twitter: librarianmer