This presentation outlines some key considerations for researchers working in the fields of open education, OER and MOOC. Key lines of debate in the open education movement will be described and critically assessed. A reflective overview of the award-winning OER Research Hub project will be used to frame several key considerations around the methodology and purpose of OER research (including 'impact' and 'open practices'). These will be compared with results from a 2016 OER Hub consultation with key stakeholders in the open education movement on research priorities for the sector. The presentation will conclude with thoughts on the potential for openness to act as a disruptive force in higher education.

1. Critical issues in contemporary open
education research
Digital University Network Seminar
Society for Research into Higher Education
Nov 2016
Dr. Rob Farrow
@philosopher1978

2. @philosopher1978
@oer_hub

3. • Award-winning research into open education
• Strategies for building worldwide open education research capacity
• Available for research & consultancy (short & long term)
• Current projects include:
oerhub.net

5. Project Co-PILOT

6. Keyword Hypothesis
Performance OER improve student performance/satisfaction
Openness People use OER differently from other online materials
Access OER widen participation in education
Retention OER can help at-risk learners to finish their studies
Reflection OER use leads educators to reflect on their practice
Finance OER adoption brings financial benefits for students/institutions
Indicators Informal learners use a variety of indicators when selecting OER
Support Informal learners develop their own forms of study support
Transition OER support informal learners in moving to formal study
Policy OER use encourages institutions to change their policies
Assessment Informal assessments motivate learners using OER

7. • Research instruments applied
consistently across collaborations:
surveys, interview questions,
focus groups, etc.
• Supplemented by integration of
secondary research
• ‘Agile’ research, sprinting
• Thematic and methodological
cohesion provided by research
hypotheses
Research Process

10. Geographical spread of research participation

11. Some key findings
https://c2.staticflickr.com/4/3523/4006230793_01f3abe22b.jpg

12. • 37.6% of educators (n=268) agree or strongly agree that OER use
increases student satisfaction
• 27.5% of educators (n=196) agree or strongly agree that OER use
improves student grades
• Impact appears to be greater for non-grade related aspects:
- 36.2% (n=254) OER improves student engagement ✓
- 36.2% (n=254) OER promotes new ways of learning ✓
- 35.2% (n=256) OER increases student interest in subject ✓
- 35% (n=249) OER leads to student self-reliance ✓

13. Community College Educators
• Responses from CA, TX, VA, FL, TN
• Most respondents were experienced
teachers with postgraduate degrees
• A majority teach full-time and are
involved in online instruction
• Around half have adapted/used OER
but only around 25% create or upload
OER

14. 6%
13%
14%
14%
16%
18%
19%
21%
22%
23%
3%
5%
1%
3%
4%
2%
2%
2%
2%
2%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
I make use of a wider range of multimedia
I reflect more on the way that I teach
I have broadened my coverage of the curriculum
I now use OER study to develop my teaching
I have improved ICT skills
I more frequently compare my own teaching with others
I have a more up-to-date knowledge of my subject area
I use a broader range of teaching and learning methods
I collaborate more with colleagues
I make more use of culturally diverse resources
strongly agree– agree– neither agree nor disagree– disagree– strongly disagree–
Impact of OER use on teaching practice
(community college educators n=136)

15. • 55.7% (n=370) of formal students agree or strongly agree
that OER increases student satisfaction
• Formal learners reported that increased interest in subject
was the main outcome from using OER (60.1% n=398)
• Others included increased experimentation (49.4% n=398)
and gaining confidence (48.6% n=322)
• For some cohorts (e.g. Saylor Academy) more than half of
learners believed that they grew more confident, became
interested in a wider range of subjects and felt their learning
experiences improved

16. Textbook
access

17. Access to
high quality
content
Community
building

18. 8.5%
8.6%
8.7%
11.5%
11.8%
12.3%
14.3%
15.2%
15.2%
15.4%
15.5%
17.1%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
... increases interest in the subjects taught
... builds confidence
... allows me to better accommodate learners' needs
... increases collaboration and/or peer-support
... increases enthusiasm for future study
... increases participation in class discussions
... increases satisfaction with the learning experience
... leads to improved student grades
... increases engagement with lesson content
... develops independence and self-reliance
... leads to interest in a wider range of subjects
... Increases experimentation with ways of learning
Strongly agree Agree Neutral Disagree Strongly disagree
Perceptions of impact of OER on students
(community college educators n=136)

19. 7.9%
10.8%
12.2%
18 %
23%
25.2%
25.9%
29.5%
32.4%
34.5%
37.4%
38.1%
38.8%
64.7%
69.8%
Data sets
Whole course
Interactive games
Infographics
Audio podcasts
Learning tools, instruments and…
Lesson plans
E-books
Tutorials
Elements of a course (e.g. a…
Open textbooks
Quizzes
Lectures
Images
Videos
Multimedia content is
around twice as popular
as other OER (including
textbooks)
Few reported using a
whole course of OER,
suggesting they cherry
pick resources as
needed
Very few are using
openly available data to
teach
OER Types Used
(community college educators n=136)

20. 1% 2% 4% 5%
9%
11% 13%
17%
20%
26% 26%
32%
35%
43%
71%
0%
10%
20%
30%
40%
50%
60%
70%
80%
OER Repositories Used
(community college educators n=136)

21. https://www.flickr.com/photos/amagill/3366720659
Strong evidence for savings when replacing textbooks with open versions
Evidence for other forms of cost savings is less clear

22. Has your institution saved money
through OER?
yes
44%
no
19%
don't know
37%
“OER resources are old-school, low-
tech modules that are not peer-
reviewed or nationally normed.
There are some interesting tentative
attempts at creativity but much of
OER is cr*p.”
“Students return for
additional classes.”
“They don't like losing the
revenue stream from the
bookstore.”
“I know that some
instructors are only using
OER which provides
substantial savings for our
students.”

23. Have your students saved money
through OER?
yes
62%
no
13%
don't know
25%
“Saving money is a big
incentive for students and
institutions.”
“I still use publishers'
textbooks in my classes. I
use OER as supplements to
the textbook.”
“My students tell me and
enrollment in my classes has
continually increased over
those of my peers.”
“I developed an online
textbook for the personal
health class that I teach. This
saves each of my students
approximately $100.”
“I know that some
instructors are only using
OER which provides
substantial savings for our
students.”

24. How important is open licensing (e.g. CC)?
crucial
20%
very important
34%
somewhat
important
19%
neither important
nor unimportant
22%
not at all important
5%
More than half feel that open
licensing is crucial or very
important, but far fewer actually
practice it.
This could indicate either 1) that
educators are not confident about
licensing their work or 2) they feel
it is an avoidable addition to their
workload.

25. • Most of the respondents have used some sort of OER, though only around a quarter
create OER
• Most report positive effects on their teaching practice as a result of OER use,
particularly around peer collaboration and improved subject knowledge
• A smaller proportion (but still in excess of 40%) feel that OER use directly leads to
improved reflection on pedagogical practice
• Positive effects were also identified for learners, especially around increased self-
reliance, subject interest and experimentation
• There were similar numbers who thought OER wasn’t making much of a difference and
a core of what might be termed ‘anti-OER’ responses
Community college (educators)
Community College Educators: Summary of Results

26. Community College Educators: Summary of Results
• There were mixed views about whether OER was saving institutions
money, but approximately 2/3 felt that students had saved money
• Around 1/3 believe that OER is improving student attrition while around
1/2 believe it is not having an effect
• There is a core of advocates who understand and actively promote
OER; they adopt open educational practices and believe it leads to
benefits
• Only around half of OER creators have used open licensing

27. https://c2.staticflickr.com/6/5082/5282511749_549b203bc2_b.jpg
Farrow, R., de los Arcos, B., Pitt, R., & Weller, M. (2015). Who are the
Open Learners? A Comparative Study Profiling non-Formal Users of Open
Educational Resources. European Journal of Open, Distance and E-
learning, 18(2).

28. 0% 5% 10% 15% 20% 25% 30% 35%
Under 15
15-18
19-24
25-34
35-44
45-54
55-64
65-74
Over 75
Saylor (n=2299) iTunesU (n=103) OpenLearn (n=725)
Age profiles by repository (N=3127)

29. iTunesU channel users were much
more likely to be younger and were
mostly male. They are often in full
time education and use OER on an
informal basis outside of their
formal studies to pursue interest in
a wide range of subjects.

30. Saylor Academy users are more
likely to be in employment and
already in possession of a degree.
They tended to be middle aged
and primarily motivated by
professional development.

31. OpenLearn users were more likely to be older, retired, and
female, and had a higher proportion of users who were
motivated mainly by personal interest (though 40% are in
full time employment).

32. Patterns of OER repository use (N=2460)
0%
20%
40%
60%
80%
100%
Saylor (n=1802) iTunesU (n=104) OpenLearn (n=554)

33. • Data about prior qualification supports idea that OER
mainly used by already educated
• Most learners use a small number of repositories and
indicate little awareness of OER as concept/method
• Very high degree of satisfaction with OER across all
repositories
• However, this doesn’t necessarily equate to increased
likelihood of formal study

34. Impact of OER repository on likelihood of future study
Repository More likely to study formally Less likely to study formally
Saylor
(n=1858)
19.8% 19.9%
OpenLearn
(n=583)
31.4% 13.9%
iTunesU
(n=94)
23.4% 25.5%

35. Adaptation of open resources is high (79.4%) for all roles:
This is higher than found in previous studies (e.g. Wiley,
2009) but more research is needed into adaptation
behaviours (motivations, goals, techniques, evaluation, etc.)
… can one adapt without altering?

36. OER Evidence Report 2014
OER Data Report 2015
http://oerhub.net/reports/

37. Open Research (2016)
https://openresearch.pressbooks.com/

38. Thoughts on research challenges
OER awareness is low, but OER use is high
Evidence about OER impact is mixed
Efficacy studies for open textbooks have become the standard form of study
Lack of triangulation reinforced by nature of openness
Incommensurability & granularity of evidence
Openness also provides opportunities for innovating research practice

39. Thoughts on openness
Openness as constellation
Openness as enabler of critical pedagogies
The ‘return’ of ideology into open education movement?
Ethical, social and political commitments of openness

40. Deimann, M. & Farrow, R. (2013). Rethinking OER
and their use: Open Education as Bildung.
International Review of Online and Distance
Learning 14(3).
http://www.irrodl.org/index.php/irrodl/article/view/13
70/2542
Farrow, R. (2016).“Constellations of Openness” in
Deimann, M. and Peters, M. A. (eds.) The
Philosophy of Open Learning: Peer Learning and
the Intellectual Commons. New York: Peter Lang.
https://www.peterlang.com/view/product/31200
Farrow, R. (2016). A Framework for the Ethics of
Open Education. Open Praxis, 8(2).
http://dx.doi.org/10.5944/openpraxis.8.2.291
Farrow, R. (2015). Open education and critical
pedagogy. Learning, Media and Technology. DOI:
10.1080/17439884.2016.1113991
Some relevant publications

41. The Open Research Agenda

42. What is the Open Research Agenda?
Simple consultation and sharing exercise
Invitation to share thoughts about research priorities in open education
Results are discussed at conferences and workshops
That discussion then becomes part of the following presentation
Promoting an open research culture
The hope is that we will become better able to anticipate research needs
and funding opportunities as well as find potential collaborators

43. What is the Open Research Agenda?

Community consultation exercise to better understand research priorities of
practitioners
Intended to identify patterns across countries and stakeholders
‘ Flipping’ the conference format

Stealing research ideas
Setting the agenda ourselves

44. Survey Methodology
Country (main residence)
Self-perception of OER expertise (adapted from Dreyfus & Dreyfus (1980)
model of skill acquisition)
Role (choose from list or free text)
Context of work/study (based on International Standard Classification of
Education (UNESCO, 2011))
Perception of research priorities (free text)
Most important research questions (free text)

45. As at 31 October 2016:
• 91 survey responses
• Respondents from 24 countries across 5 continents:
Australia, Bangladesh, Brazil, Canada, Chile, Croatia, Ecuador,
Ethiopia, Germany, India, Ireland, Macedonia, Malta, Mexico, New
Zealand, Portugal, South Africa, The Netherlands, United Kingdom
• Approximately a further 90 participants in discussion (3 expert
meetings; 3 conferences; 1 webinar)
Survey Sample Overview

46. Hewlett OER Meeting (February)

47. Action Lab @ OE Global, Krakow (April)

48. Action Lab @ OE Global, Krakow

49. OER16, Edinburgh, UK (April)

50. Global OER Graduate Network (May)

51. Computers and Learning Research Group
Annual Conference (June)

52. European OER Experts Meeting (June)

53. OERu Expert Meeting (October)

54. Open Education 2016 (November)

55. Summary of Results

56. Role

57. • One third of respondents described themselves as having only one role
• More than three quarters (77.3%) of all respondents described themselves
as wholly or in part an educator (n=75); Over 40% (n=42) said they were
both an educator and an advocate
• Three quarters of respondents who describe themselves in whole or part as
a policymaker also describe themselves as an advocate of OER (n=11)
• Some people said they had as many as 7 simultaneous roles
• Most participants who responded with ‘Other’ told us their role was wholly,
or in part, as a librarian (3) or researcher (13)
Role

58. Expertise/Experience

59. What are the most important areas for open
education research over the next year?

60. Most popular themes
0 2 4 6 8 10 12
Assessment
Awareness & Perceptions
Business models
Professional development
Quality
OER for development
Collaboration
Evidence
Impact
Technology
Case studies
Adoption
Pedagogy
OEP

61. ‘Tier 2’ themes
0 0.5 1 1.5 2 2.5 3 3.5
Adult education
Advocacy
Credentialing, Professional development, Reuse
Digital/information literacy
Discoverability
Implementation
Improving access
Policy, Collaboration
Copyright & intellectual property
Discourse analysis/perceptions
Reuse

62. Least popular themes
Accessibility
Ethics
Formal recognition of OER
Global North/South Innovation (marginalised voices)
Lifelong learning
MOOC
Non-formal learning
OER Creation Open Data
Social mission of open education
+ all the others not mentioned…

63. Open practices and pedagogies
• OER for lifelong learning and professional
development
• Policy development
• Benefits of ‘non-disposable assignments’
• Local/National platforms for teachers
• Networked learning
• OER for language development
• “Open pedagogy”
• Citation and cataloguing protocols
• Integration of OER into curriculum
• Creating assessment tools
• Micro-credentialling
• Recognition of prior learning
• Sustainable business models

64. OER impact research • Measuring the influence of
the use of OER and OEP on
learning outcomes and
institutions
• More empirical studies with
wider focus than purely on
open textbook efficacy
• A wider range of research
methods (including
qualitative approaches)
• Quantifying the return on
investment from going open
• Scant mention of student
cost savings

65. Promoting adoption and adaptation
• Factors affecting adoption of individual OER
• Qualitative descriptions of adaptation
• Barriers to remix and sharing (& implications for design)
• Barriers to institutional adoption
• Strategies for easing implementation by faculty

66. Collaboration and community
• Peer review
• Co-operation between institutions
• Transnational co-operation
• Collaborative instructional design
• Promoting equality of opportunity
• Partnerships that provide added value
• Quality

67. Technology and infrastructure
• Sustainable OER ecosystem
• Ways of cataloguing and sharing OER
(e.g. shared database)
• Linked open data for education
• Open digital badging
• Geographical hubs for sharing OER
• Using OER with VLEs
• Beyond repositories
• Interoperability

68. Ethical issues
• Privacy, security and trust in online
learning
• Ownership of (student) data and ‘safe
spaces’
• Access & accessibility vs. remixable
resources
• Appropriate literacies
• Being ethical ‘in the open’
• Some conflation with ethical issues in
e-learning more gnenerally

69. Openness
• Tension between differing
interpretations of openness
• Social and political aspects of open
education
• Articulating the benefits of open
• Building open communities and
supporting cross-community exchange

70. What are the most pressing questions that
need to be answered?

71. Reflections on this approach
It’s quite hard to get people to do anything!
Envisaged breakdowns of role, level of expertise have been problematic
because of multiple roles and most identifying as expert
http://oerhub.net/collaboration-2/the-open-research-agenda-2/

73. “Messes are complex, multi-dimensional, intractable,
dynamic problems that can only be partially addressed
and partially resolved.”
(Brydon-Miller et al., 2003:21)

74. Action Research
Originally conceived by Luwin (1946)
“a cycle of posing questions, gathering data,
reflection, & deciding on a course of action”
(Ferrance, 2000)
Several approaches (traditional, contextual,
radical, educational)
Simple Action Research Model
(MacIsaac, 1995)

75. Participatory Action Research (Freire)
Blackall & Hegarty (2012:69) identify four key aspects to Participatory Action
Research (PAR) which are relevant to the worldwide OER community:
The explicit aim to engage all stakeholders […] in describing the problems
Asking those stakeholders to research the problem and propose solutions
Empowering those stakeholders to carry out their plans
Repeating the cycle, reflecting on lessons learned and publishing the
research

76. Suggestions for future research
Holistic case studies which include a focus on efficacy and cost savings but also provide
(balanced) qualitative data regarding the impact on persons and practices
Detailed case studies of open pedagogy (including theoretical perspective)
Studies of the narrative on ‘open’ (e.g. discourse analysis) – there are ever more voices in
this space with differing interpretations
Business analysis -> generate sustainable business models
How can a more holistic OER ecosystem be achieved? OER World Map?
Sociology/psychology of conversion, persuasion, and culture change
Sharing best/effective practices: institutions, educators, learners

77. Tensions in the overall picture
Desire for control and predictability Desire for freedom, exploration and
innovation
Advocacy ‘Pure’ research
A well defined community of practice An ‘open’ community
‘I have this problem and I need a
solution…’
‘I think research should be done in…’
Local context Global context
Pragmatic Ideological

78. To conclude:
The evidence base about the impact of OER continues to grow
Much of the debate in the USA is about the relative efficacy of OER alternatives
Openness presents challenges to researchers along several axes: lack of clarity; validity;
data collection; dissemination.
Openness is often contextual and situated: it can be hard to generalise
Certain tensions can be determined in the open education movement (conservative/radical;
idealist/pragmatic; &c.)
Openness can be seen to have a close connection with radical, critical connections through
the association with enhanced freedom/autonomy

79. Additional image credits
https://upload.wikimedia.org/wikipedia/commons/5/59/American_Queen_Eads_Bridge.jpg
"Life is Sharing" (CC-BY 2.0 Generic, Alan Levine)
“Open” Chuck Coker
https://upload.wikimedia.org/wikipedia/commons/4/41/Richmond,_Virginia_downtown.jpg
https://www.flickr.com/photos/caveman_92223/4016133284/
https://commons.wikimedia.org/wiki/File:Water_drop_impact_on_a_water-surface_-_(5).jpg
http://www.thebluediamondgallery.com/pictures/webinar.jpg
http://www.thebluediamondgallery.com/highlighted/a/advocacy.html CC-BY-SA 3.0
http://creativecommons.org/licenses/by-sa/3.0/
https://pixabay.com/static/uploads/photo/2012/11/07/10/14/quality-65060_960_720.jpg
https://pixabay.com/static/uploads/photo/2014/04/02/10/16/key-303320_960_720.png
https://static.pexels.com/photos/4316/technology-computer-chips-gigabyte.jpg
https://upload.wikimedia.org/wikipedia/commons/thumb/b/b8/Openness.jpg/377px-Openness.jpg
https://upload.wikimedia.org/wikipedia/commons/8/84/Openness_and_Collaboration.png

80. • Award-winning research into open education
• Strategies for building worldwide open education research capacity
• Available for research & consultancy (short & long term)
• Current projects include:
oerhub.net