Presentation from ALT-C conference, 2014 on the value of mapping and curation as an approach to impact research. The presentation includes some discussion of results from OER Research Hub.

1. Understanding patterns in
OER Impact: the role of
mapping and curation
Dr. Rob Farrow
The Open University

2. Structure
OER Research Hub
Collaborative Research
Evidence Model
Mapping and Curation
Preview of Project Results
OER Impact Map

3. #oerrhub
@philosopher1978

4. OER Research Hub
• Research project at The Open University (UK)
• Funded by William & Flora Hewlett Foundation for two years
• Tasked with building the most comprehensive picture of OER impact
• Organised by eleven research hypotheses
• Collaboration model works across different educational sectors
• Global reach but with a USA focus
• Openness in practice: methods, data, dissemination
oerresearchhub.org
#oerrhub

5. Rationale
• Consolidatory phase for OER movement after years of investment and piloting
• The need for an evidence base: advocacy, strategy, policymaking
• Lack of robust evidence for OER impact
• The evidence ‘problem’ in OER impact research
• Isolated ‘pockets of innovation’
• No agreement over best methods for impact comparison
• Compare diverse experiences to assess the influence of ‘open’
• Mixed methods data from experimental + non-experimental research

6. Collaboration
Developing an ecology of sharing

8. Research Process
• Research instruments applied
consistently across collaborations:
surveys, interview questions,
focus groups, etc.
• Supplemented by integration of
secondary research
• ‘Agile’ research, sprinting
• Thematic and methodological
cohesion provided by research
hypotheses

9. Practical Advantages /
Methodological Disadvantages
Inclusive
Holistic
Validation of experiences at the ‘cutting edge’
Open to diverse perspectives
Incommensurability?
Granularity?
Synthesis?
How will comparison
of experiences be
made rigorous?

10. Hypotheses
Key claims about open education

11. ‘Evidence’ is only evidence in relation to a claim or hypothesis:
the project hypotheses form the core of the metadata model.
Keyword Research Hypothesis
Performance OER improve student performance/satisfaction
Openness People use OER differently from other online materials
Access OER widen participation in education
Retention OER can help at-risk learners to finish their studies
Reflection OER use leads educators to reflect on their practice
Finance OER adoption brings financial benefits for students/institutions
Indicators Informal learners use a variety of indicators when selecting OER
Support Informal learners develop their own forms of study support
Transition OER support informal learners in moving to formal study
Policy OER use encourages institutions to change their policies
Assessment Informal assessments motivate learners using OER

12. Evidence Model
Design for information architecture

13. Key Questions for Evidence Model
What evidence exists to support key claims about OER?
How do patterns of OER impact differ across countries?
How do patterns of OER impact differ according to sector?
How can the veracity of evidence be checked?
How can diverse evidence forms be meaningfully collected together?

14. Evidence: Information Architecture
Title • Text
Copy • Text / HTML - Supports embedding of multimedia
Hypothesis • Association of evidence with hypothesis
Polarity • Evidence is either +ve/-ve in relation to a hypothesis
Location • Geolocation
Sector • School (K12) / College / Higher Education / Informal
Citation • Academic citation or Hyperlink / URL

15. Sample Records
Multimedia embedding

16. • Popular OS
platform
• Easy to customize
• Over 27,000
existing plugins
Wordpress
Customization
• CSV importer
• Location plugin
• JSON
• Project code
available on
GitHub
• Easy to link, share
and engage
Openness
Technical Development

17. Synthesis
Mapping and curation

18. Synthesis
Synthesis Methods
• Isolating data by hypothesis, sector, country, or any combination
• Collaborative curation of research data
• Data visualization, reporting
• Editorial quality control exercised centrally
Validation
• Iteration through current and future patterns of evidence
• Open citation trails allow public auditing of evidence
• Community voting

19. in service of The Open University

20. OER Impact Map
oermap.org

21. OER Impact Map: Maps
oermap.org
• Country Summary Map - evidence nodes organized by country
• OER Project Map - OER initiatives and projects around the world
• OER Evidence Map - all impact evidence is categorized according to the OER
Research Hub hypotheses
• OER Policy Map is the single largest curated collection of OER policies
• OER Impact Map aggregates the other maps
• Tweetmaps show a geographical summary of tweets for a particular Twitter
hashtag. E.g. #oermap for outreach; #oerrhub for tracking project activity.
• Collection of maps by others

22. Global Impact Summary
Select country to zoom

23. Flowmap changes according
to country selected…
in service of The Open University

24. Impact Summary (USA)
Node colour = sector
Node border = polarity

25. Positive Evidence
Negative Evidence
“Uncategorized Evidence”
OER Evidence Map

26. OER Policy Map
L
R
N
I
Local/Institutional policies
Regional policies
National policies
International policies
CC Open Policy Network
POERUP
SPARC

27. in service of The Open University
OER Projects
OER Project Map

28. in service of The Open University
OER Impact Map

29. in service of The Open University
OER Impact Map
Combines other maps
Supports filtering

30. tweetmap
tiny.cc/oermap
#oermap

31. in service of The Open University

32. Hypothesis Reporting

33. Hypothesis Reporting

34. in service of The Open University

35. in service of The Open University

36. Mapping & Curation: Summary
• A collaborative approach to OER impact research is recommended
• Mapping and curation are effective tools for building a collective understanding of
a problem and building an evidence base
• Open methods for researching openness: openly licensed instruments; remix
and reuse data; working openly; sharing
• Takeaway message: the whole is greater than the sum of the parts

37. OER Research Hub
Preview of consolidated survey
results

38. OER impact on Student Performance:
• More than 1/3 of educators agree (or strongly agree) that OER increase
student satisfaction with learning
• a higher percentage of formal students (55.7%) agree (or strongly
agree)
• OER benefits for formal learners: increased interest in subjects taught
(60.1%, n=398); increased experimentation with new ways of learning’
(49.4%, n=327); gaining confidence (48.6%, n=322).

39. The importance of open licensing:
• Comparatively high level of adaptation amongst all types of users
• Two thirds of educators (67.5%) consider open licensing important and
are familiar with Creative Commons logo (55.7%)
• But only 12.4% of educators create resources and publish them on a
Creative Commons license
• This is, however, consistent with the fact that only 26.8% (n=215) of
educators are concerned with knowing whether they have permission to
use or change a resource.

40. OER and access to education:
• A majority of informal OER users already educated to degree
level
• However, some learners are using OER as a replacement
for formal education to which they might not otherwise have
access.
• 88.4% of all learner respondents (n= 3761) indicated that the
opportunity to study at no cost was significant, and many
formal students use OER as replacement for higher education
• Data from OpenLearn suggests that that OER increase
access to education for informal learners with disabilities

41. OER impact on educators & reflective practice
• Broader range of teaching practices
• More reflection on method
• More comparison with other teachers
• More collaboration
• Use OER for inspiration
• Broadening coverage of curriculum
• Bringing in multiple perspectives on topics

42. Financial savings:
• Strong evidence that open textbooks save money
• Almost 75% of educators believe that OER saves students money
• Just under 80% of students believe they are saving money
Indirect benefits for institutions:
• Making college affordable encourages students to continue to study
• Savings made through digitization and reuse

43. OER Impact Map: Next Steps
• Adding qualitative data analysis to evidence base
• Increasing the evidence base through desk research
• Using the map as a storytelling tool
• Refinement of user interface
• A badging system to reward contributors and commenters
• New hypotheses, new contributors, (hopefully) community uptake
• School of Open course on open research; Open Data; CC-BY research tools
• Survey data ‘exploration tool’

44. Survey Data Explorer (prototype)
in service of The Open University

45. Survey Data Explorer (prototype)
in service of The Open University

46. Survey Data Explorer (prototype)
in service of The Open University

47. Mapping & Curation: Reflection
• Would mapping & curation be useful in your own practice?
• What would you map & why?
• How would you categorize your data to make it meaningful for others?
• What would you omit from the map to make it more effective?
• What would your map look like? Even a very basic attempt to produce a map
can help identify the most relevant data
• Dissemination: mapping also helps focus on visual communication and
conveying message
• Cartographic pedagogies: we use this for research, but it could also be used for
teaching as an artifact shared between learners

48. Join us in building understanding
of open education
School of Open
course on
#openresearch
(Sep 2014)
Take our general
OER impact survey
Contribute to OER
Impact Map

49. Thanks for listening!
oerresearchhub.org
oermap.org
rob.farrow@open.ac.uk
@philosopher1978