As part of the Learning@Work Exploratorium Lab that is run by the European Learning Industry Group (ELIG) within the FP7 funded HoTEL research project we have been running a joint ELIG / Pearson interactive Learnshop that aimed at critically reflecting on how to innovate in a profoundly changing education ecosystem. A key focus on this Learnshop had been to further advance on the question of how the Pearson efficacy framework might be best scoped, structured and contextualized [e.g. as a part of a larger model] so as to foster its usefulness and applicability as a tool to support TEL for individuals / institutions, through the practical application of the framework with real-life ‘cases’ This document provides an overview on the Learnshop structure and might be of use to those that intend to organize similar activities.

1. How to Guide Innovation in a
Changing Education Ecosystem
Learnshop Structure
v.1.1 - 6 February 2014
Registered Address: Pearson plc, 80 Strand, London, WC2R 0RL
53723
Registered in England

2. Online Educa Berlin Learnshop – An Overview
The conference itself (Wednesday 4th – Friday 6th December 2013) will take place at the Hotel
InterContinental Berlin:
Budapester Str. 2
10787 Berlin
Germany
Tel.: +49 (0) 30 26 02-0
Fax: +49 (0) 30 26 02 26 0
Session runs: Friday 6th December 2013 - 11:45 - 13:30
(http://www.online-educa.com/ap/programme_detail.php?id=LWL56)
***
How to Guide Innovation in a Changing Education Ecosystem
This interactive Learnshop aims critically to reflect how to innovate in a profoundly changing
education ecosystem. What are the opportunities for innovation within emerging lifelong and
life-wide multi-stakeholder and multi-sided ecosystems?
Join the panel of experts addressing this challenge in this Learnshop, organised as part of the
EU FP7 funded HoTEL project „HOlistic approach to Technology-Enhanced Learning‟. The overall
objective of the project is to design, develop and test a support model for innovation in the area
of technology-enhanced learning.
Facilitators
Andreas Meiszner
European Learning Industry Group (ELIG), The Netherlands
Elmar Husmann
European Learning Industry Group (ELIG), Germany
Kelwyn Looi
Analyst, Office of the Chief Education Advisor, Pearson, UK
Vaithegi Vasanthakumar
Associate, Office of the Chief Education Advisor, Pearson, UK
Fadi Khalek
VP-Higher Ed & Voc Learning Solutions, Pearson Education EMA
Adam Black
Chief Learning Technologies Office, Pearson ELT;
SVP Efficacy and Global Scale of English Products, Pearson English, UK
Background
The HOTEL (Holistic Approach to Technology Enhanced Learning) project originates from the
observation that most of the TEL research so far has concentrated mainly on the development
of ad hoc technologies for learning, failing to capture both the potential adoption of emerging
technologies not originally designed for learning in education and training environments and the
innovative use that is made of technologies in non-formal and informal ways of learning and the
extent to which this could be transferred / adapted to formal learning environments.
This is believed by the Consortium to provide a misleading and fragmented picture of the extent
to which new forms of using technologies (already mainstreamed and/or successfully piloted
and/or emerging) support learning. Innovative practices in the use of technologies for learning
(especially in non-formal and informal learning environments) are often not sufficiently
considered by research whereas bottom-up innovation is playing an increasingly important role
in the field of TEL, which might lead to new theories for learning. On the other hand, there is a
need to verify the impact of existing learning theories on TEL practices to determine whether
Registered Address: Pearson plc, 80 Strand, London, WC2R 0RL
England 53723
Registered in
2

3. this has led/is leading to innovation. Furthermore, the lack of a holistic approach in TEL as
described above puts at risk the effectiveness and mainstreaming of new ways of using ICT for
learning purposes: too often the timespan between the identification of technologies that have a
potential for learning, the theoretical analysis of pedagogical implications, the piloting of such
technologies and their adoption (first at small scale and then mainstreamed) is so long that the
technology itself becomes out-dated compared to the changing environment and learning
needs.
Pearson‟s Efficacy Framework would be tested as a [e.g. stand-alone] means to support the
different stakeholders to innovate in TEL / education. ELIG and Pearson have been working
together to design a learnshop that would foster the hands-on experience of using the efficacy
framework with test cases of technological innovationsin order to examine:
i.
Whether new innovations necessarily support learning enhancement
ii.
The impact of existing learning theories on TEL practices to determine whether this has
led / is leading to innovation
Existing knowledge through the Alive in the Swamp study can be leveraged to provide
(http://www.nesta.org.uk/library/documents/Alive_in_the_Swamp.pdf). Though the study had
a K-12 focus, it might still be well suited to provide some general guidance and transversal
(high-level) criteria/aspects that would apply to any type of education, and particularly in
regards to digital innovations.
Our recommendation is that the “Alive in the Swamp” document is summarised in an outline
form, with the facilitators to use some of the key questions of the innovation index as stimuli for
discussion when facilitating the use of the efficacy framework in practice and to ensure a more
specific and targeted steer for discussions.
Objectives
1. We want to further advance on the question of how the efficacy framework might be
best scoped, structured and contextualized [e.g. as a part of a larger model] so as to
foster its usefulness and applicability as a tool to support TEL for individuals /
institutions, through the practical application of the framework with real-life „cases‟
2. For attendees the learnshop would provide the opportunity:
a. To acquaint themselves with the efficacy framework as a tool to engender
learning-focused discussions when assessing and evaluating prospective
innovations
b. To be involved at the input phase of the project, allowing for feedback provided
on evaluating the efficacy framework as a support model to be incorporated into
future iterative decisions of the holistic model
c. To examine their own „case‟ (or a sample case) through the lens of a rigorous and
structured framework, providing the participant key takeaways at both the
transversal and individual case level
Registered Address: Pearson plc, 80 Strand, London, WC2R 0RL
England 53723
Registered in
3

4. Introduction to Efficacy
In line with the recent public commitment to efficacy(http://efficacy.pearson.com/)the
learnshop is seen to be an appropriate showcase for the application of the efficacy framework to
a wide variety of interested parties. Inside and outside Pearson “efficacy” has different
meanings. At Pearson we have agreed on a definition of efficacy. Efficacy is defined as:
“A measurable impact on improving someone’s life through learning.”
We need to be able to identify the specific impact for a learner.Efficacy has direct and obvious
applications for those who are designing and delivering products, services and solutions to
learners. The Efficacy Framework was developed by Sir Michael Barber (Chief Education
Advisor) and his team. It draws on best practices about delivery from Pearson, and the public
and private sectors.
The Efficacy Framework has two purposes: to understand whether we are delivering efficacy,
and to identify a path to improve efficacy. This is outlined below, with the four key questions
asked as part of the framework and a set of ratings for identification.
Registered Address: Pearson plc, 80 Strand, London, WC2R 0RL
England 53723
Registered in
4

5. Pre-workshop actions
Participants are provided with a pre-determined template for the learnshop (see Appendix 1) to
enable the workshop facilitators to deliver a bespoke and tailored session to ensure that mutual
benefit for all parties is achieved.Cases brought to the workshop to be evaluated by the Efficacy
Framework will be demonstrated in this template.
Guiding Principles
1. A clearly defined link between the scope of the workshop and the title outlined in the
Educa Berlin documentation: “How to Guide Innovation in a Changing Education
Ecosystem” and this is to be outlined in the initial introduction
2. A specific joint value proposition from Pearson and the HoTEL Project which outlines the
aim to achieve both the intended outcomes for each party, as well as to deliver an
engaging and thought-provoking workshop for the participant
Basic format(for11:45 - 13:30 slot)
1. Introduction to the Learnshop and outlining the objectives (5mins)
2. Context: HoTEL Project and Pearson (10mins)
3. Introduction to efficacy (15mins)
4. The Efficacy Framework in Practice: (60mins)
5. Summary, reflections and Q&A, feedback (10mins)
Inventoryin order to conduct the learnshop:
Beamer
Printed copies of the Efficacy Framework and evaluation forms
Brown paper, sticky dots, pens, post-it notes
Hard copies of Asking Moreand The Incomplete Guide
Registered Address: Pearson plc, 80 Strand, London, WC2R 0RL
England 53723
Registered in
5

6. Appendix 1: Case Study template
Joint ELIG Pearson Online Educa Berlin Learnshop
How to Guide Innovation in a Changing Education
Ecosystem
Friday 6th December 2013 - 11:45 - 13:30
Case Study Template &
Background Information
This interactive Learnshop aims critically to reflect how to innovate in a profoundly changing
education ecosystem. What are the opportunities for innovation within emerging lifelong and
life-wide multi-stakeholder and multi-sided ecosystems?
The subsequent case information will form the base within the Learnshop to examine the case
through the lens of a rigorous and structured framework, the Pearson Efficacy Framework, as a
tool to engender learning-focused behaviours when assessing and evaluating prospective
innovations.
Facilitators
Andreas Meiszner
Elmar Husmann
Kelwyn Looi
Vaithegi Vasanthakumar
European Learning Industry Group (ELIG), The Netherlands
European Learning Industry Group (ELIG), Germany
Analyst, Office of the Chief Education Advisor, Pearson, UK
Associate, Office of the Chief Education Advisor, Pearson, UK
Fadi Khalek
VP-Higher Ed &Voc Learning Solutions, Pearson Education EMA
Adam Black
Chief Learning Technologies Office, Pearson ELT;
SVP Efficacy and Global Scale of English Products, Pearson
English, UK
Contacts:
Kelwyn Looi
(kelwyn.looi@pearson.com)
Registered Address: Pearson plc, 80 Strand, London, WC2R 0RL
England 53723
Andreas Meiszner
(andreas.meiszner@elig.org)
Registered in
6

7. Case Study Template
General Information
Name of Case :
Case Website:
Contact information
Name & Surname:
Institution or
Company:
Email:
Stage of
Development:
Initial – Idea / Research Project / Pilot / Prototype / Commercialised
Case Information
1. Background & introduction
Briefly describe the nature of the case
2. Conception and progress to date
How did the case come about, what progress has it made to date, and what are plans for the
future?
3. Outcomes & value proposition
What are the intended outcomes & value proposition of the case with regards to enhancing
learning and education?
4. Measures of achievements and success
What are the intended (or already implemented) measures of achievements and success
through the case life cycle? This could include the data that you will collect, examples of
research activity etc.
5. Impact
What is the envisioned impact of the case, in particular on the wider education community and
the changing education ecosystem?
Registered Address: Pearson plc, 80 Strand, London, WC2R 0RL
England 53723
Registered in
7

8. Appendix 2a): Alive in the Swamp – A primer
(http://www.nesta.org.uk/library/documents/Alive_in_the_Swamp.pdf).
This report provides practical advice on how to navigate through digital innovations in
education, and suggests where more innovation effort is needed. The report, from Michael
Fullan and Pearson's Katelyn Donnelly, provides an actionable guide to learning technology that
will allow founders, funders, and teachers to make better decisions. It identifies persistent gaps
in innovation activity and points to what needs to be done if we are to finally make good on the
promise of technology to transform learning.
Key applications to the workshop:
Objective
To leverage the Alive in the Swamp “Innovation Index” toprovide moretechnology-focused
questions aligned to the core efficacy framework.
The questions normally asked during efficacy reviews at Pearson have been used for the full
suite of products, services, solutions, acquisitions and investments, including digital products,
and adapted as necessary, so as to verify the diversity of use of the Efficacy framework.
Registered Address: Pearson plc, 80 Strand, London, WC2R 0RL
England 53723
Registered in
8

9. Appendix 2b): Alive in the Swamp – Supporting document
Rationale for questions:
1. To make transformative system improvements we need to know, with precision and
clarity, what the learning goals are
2. Digital technologies that do not align with what is to be learned will likely not translate
into learning enhancement
Outcomes
How clearly are the learning outcomes of the innovation defined?
Are the learning outcomes explicit and defined for learner and the organisation?
What is the quality of case model design?
Does the technology incorporate latest design principles for user experience?
Does the product accelerate learning?
Does this innovation have the ability to scale system–wide?
How expensive is the product or design change itself?
Is the product of sufficient value, demonstrated by learning outcomes, to justify change?
Are there hidden costs such as infrastructure upgrades?
Are there overall cost savings realised by the innovation?
Evidence
What is the quality of the assessment platform? Is it adaptive and does it include an
optimal amount of detail?
Is it clear how the outcomes will be measured?
Is the technology integrated and seamless?
How does the learner use the assessment system to monitor and motivate his or her
own learning?
How refined is the pedagogical underpinning?
Is 24/7 access and learning enabled?
Does the pedagogy reflect the latest global research, including the emphasis
on constructivism and real–world examples?
Planning
Is there a mechanism to ensure the pedagogy is updated?
Is the assessment system integrated into the pedagogy and learning curriculum?
Is the technology adaptable and highly connective?
How does the innovation implement in the whole system?
Is there a plan for scale based on world–leading change knowledge?
Registered Address: Pearson plc, 80 Strand, London, WC2R 0RL
England 53723
Registered in
9

10. Capacity
Is (Are) the clarity of case outcome(s) shared by all stakeholders?
What is the quality of the user experience? Is it engaging, efficient and intuitive?
is capacity building a central component of the strategy?
What support is provided to ensure the technology functions (for all parts including
software, hardware, maintenance, electricity and connectivity?)
What is the nature of the implementation support provided?
How long is the implementation support or servicing in place for?
Is the support based on a culture of learning, risk-taking and learning from mistakes?
Does the innovation include user training and professional development? Are user
development goals explicit?
Registered Address: Pearson plc, 80 Strand, London, WC2R 0RL
England 53723
Registered in
10

11. Appendix 3: Pearson event feedback and evaluation questionnaire
Key: (1 = not useful at all, 5 = useful to some extent, 10 = extremely useful; please fill in the
appropriate number into the box)
1. How useful was the Efficacy Framework as a support model to assess the impact of
technological innovations on learning outcomes?
2. How useful was the workshop in aligning to your objectives? If you could suggest
some improvements to the workshop, what would they be and how would they
better align with your objectives?
3. What takeaways do you have from the workshop at the (i) (if relevant) Individual
case level and (ii) In a general sense?
4. How will your learnings from the workshop influence how you approach
Overall Satisfaction
Registered Address: Pearson plc, 80 Strand, London, WC2R 0RL
England 53723
Registered in
11

12. (1 = not at all satisfied, 5 = satisfied to some extent, 10 = extremely satisfied; please fill in the
appropriate number into the box)
Additional comments
If you would be interested in further collaboration and exploring efficacy in more
detail, then please do not hesitate to contact
efficacy.global@pearson.com
In addition, please fill in your personal details below, if you would like to be contacted about
future opportunities for collaboration. Note that your information will remain private and
confidential:
Name: …………………………………………………………..…………………………………………………………..……………….
Country: …………………………………………………………..………………………………………………………………………..
Company: …………………………………………………………..………………………………………………………………………
Job Title: …………………………………………………………..…………………………………………………………..…………..
Email: …………………………………………………………..…………………………………………………………..………………..
Registered Address: Pearson plc, 80 Strand, London, WC2R 0RL
England 53723
Registered in
12

13. Appendix 4: HoTEL event feedback and evaluation questionnaire
Did the event meet your expectations? Why?
Please assess in the following table the overall aspects of the HOTEL
event
Event
Evaluation
Overall usefulness of
the event
Relevance of the
event for your own
work/the work of your
institution/ company
Overall rating
1 2 3 456 7
low
high
1 2 3 456 7
dissatisfactory
Overall quality of the
interventions
dissatisfactory
Balance of content
presentation and your
participation
possibilities
high
1 2 3 456 7
low
Organisation of the
event
Judgment of the
moderated/interactive
sessions if any
Comments
excellent
1 2 3 456 7
excellent
1 2 3 456 7
dissatisfactory
excellent
1 2 3 456 7
dissatisfactory
Registered Address: Pearson plc, 80 Strand, London, WC2R 0RL
England 53723
excellent
Registered in
13

14. Please assess in the following table the outcomes/results of the event
(A)
Overall rating
Outcomes Evaluation
Transferability of the
outcomes/results to
your working context
(B)
Comments
1 2 3 456 7
Low
National added value
of the
outcomes/results
Low
Relevance of the
outcomes/results to
local context needs
Low
High
1 2 3 456 7
High
1 2 3 456 7
High
What are the key messages of the project/event you will take back
home?
Would you be interested in being informed on the developments of
the project and on opportunities for joint work/collaboration?
Registered Address: Pearson plc, 80 Strand, London, WC2R 0RL
England 53723
Registered in
14

15. Evaluation Questionnaire HoTEL events
Please use this section if you wish to add any other comments:
If you wish, you may leave your personal details here. All your
answers will be treated with the highest confidentiality.
Personal Details
Name & Surname:
Institution or Company:
Country:
Email:
Thank You very much!
Registered Address: Pearson plc, 80 Strand, London, WC2R 0RL
England 53723
Registered in
HoTEL | page
15
1