Apoyo al profesorado con analíticas de comunidad, diseño y aprendizaje Supporting teachers with community, design and learning analytics Seminario eMadrid, UAM 05/2019 http://www.emadridnet.org/index.php/es/eventos2/1100-seminario-emadrid-sobre-tecnologias-dentro-y-fuera-del-aula http://www.emadridnet.org/index.php/es/28-eventos-y-seminarios/1102-apoyo-al-profesorado-con-analiticas-de-comunidad-diseno-y-aprendizaje Abstract I will present an overview of the educational technologies research conducted by the TIDE research group of the Information and Communication Technologies Department at Universitat Pompeu Fabra in Barcelona (http://www.upf.edu/web/tide @TIDE_UPF). The overview will be articulated around the perspective of supporting teachers and teacher communities (e.g., a school) in the design of the best possible (technology-enhanced) learning activities considering their students and their contexts. Main contributions that will be presented include a community platform for integrated learning design (ILDE), including multiple authoring tools (e.g. PyramidApp for collaborative learning, edCrumble for blended learning) and the use of data analytics at different levels (learning, design, community) to support community awareness and teacher reflection when designing for learning. The presentation will include results of several research projects (METIS, CoT, MdM-EDS, RESET, SmartLET, Illuminated). En esta ponencia presentaré un resumen de la investigación en tecnologías educativas llevada a cabo por el grupo TIDE del Departamento de Tecnologías de la Información y las Comunicaciones en la Universidad Pompeu Fabra en Barcelona (http://www.upf.edu/web/tide @TIDE_UPF). El resumen se presenta desde la perspectiva del apoyo al profesorado y a comunidades de profesores (como, por ejemplo, una escuela) en el diseño de buenas actividades de aprendizaje considerando los estudiantes y sus contextos. Las contribuciones principales incluyen una plataforma de comunidad para el diseño integrado de actividades de aprendizaje (ILDE), incluidas herramientas de autoría (como PyramidApp para aprendizaje colaborativo apoyado por ordenador, edCrumble para aprendizaje híbrido) y el uso de analíticas de datos a diferentes niveles (aprendizaje, diseño, comunidad) ara facilitar la conciencia de comunidad y la reflexión por los profesores cuando diseñan para generar aprendizajes. La presentación incluirá resultados de varios proyectos de investigación (METIS, CoT, MdM-EDS, RESET, SmartLET, Illuminated). http://www.upf.edu/web/tide

1. Apoyo al profesorado con analíticas de comunidad, diseño y aprendizaje
Supporting teachers with community, design and learning analytics
eMadrid, UAM, 10 Mayo 2019
Davinia Hernández-Leo
TIDE, Universitat Pompeu Fabra, Barcelona
twitter: @TIDE_UPF
website : http://www.upf.edu/web/tide

2. https://www.upf.edu/web/tide/
TIDE research group
2

3. Support teachers in designing the “best possible”
conditions for their students to learn, documenting them,
making their design ideas explicit and sharable.
(Conole, 2012; Agostinho, 2011; Laurillard, 2013;
Mor, Craft & Hernández-Leo, 2013)
Learning Design
3
Teaching as a design science

4. Learning Design
ACAD framew ork: Design -> Runtime
(Carvahlo & Goodyear, 2015)
ACAD framework
4

5. The case of CSCL macro-scripting
Free collaboration does not necessarily lead to fruitful
learning. An option is to structure the collaborative learning
flow (activity sequence, roles, group formation, resource
allocation) to trigger potentially effective social interactions.
(Dillenbourg & Tchounikine, 2007; Kobbe et al., 2007)
Learning Design in Collaborative Learning
5

6.  Learning design authoring and enactment
— Pyramid collaborative learning flow pattern. Scalable approach. Flow
creation and control mechanisms for elasticity and dynamism
6
Learning Design in CSCL: example of PyramidApp
https://www.upf.edu/web/tide/tools
Manathunga, K., & Hernández‐Leo, D. (2018). Authoring and enactment of mobile pyramid‐based
collaborative learning activities. British Journal of Educational Technology, 49(2), 262-275.

7.  Design layers metaphor
(by spaces)
 Systematically co-designed
with teachers
 Emphasis in design for
blended learning
(PBL, flipped classroom, …)
7
Learning Design in blended learning: edCrumble
https://www.upf.edu/web/tide/tools
Albo L, Hernández-Leo D. edCrumble: designing for learning with data analytics. EC-TEL 2018. Best Demo Award.

8. 8
Learning design tools
Tools for conceptualization, authoring, implementation
Generic vs. specific tools wrt. pedagogical approach
Sharing, co-creating learning designs

9. 9
Integrated Learning Design Environment
 Integrated solution to support the full lifecycle of (co-)designing,
sharing, and deploying ICT-based learning activities
Hernández-Leo, D., et al. (2018). An Integrated Environment for Learning Design. Frontiers in ICT, 5, 9

10. Integrated Learning Design Environment (ILDE)
10
Individual Space Social Space

11. ILDE, previous results
Used in several different educational contexts
(higher education, vocational education, adult education)
— Well accepted in all its functions
— Fits the needs of different types of communities
— No difference in acceptance
— Flexible design processes
— Limited social awareness and inquiry process
Data used to evaluate and understand processes…
11

12. Learning design and data
 Data used to evaluate and understand processes…
 How can data be used to support learning design and learning
design communities?
12

13. Data Analytics in teaching & learning
 Learning Analytics (LA): “the measurement, collection, analysis, reporting of
data about learners and their contexts, for porpuses of understanding and
optimizing learning and the environments in which it occurs” SoLAR (Ferguson,
2012)
 Social Learning Analytics (Buckingham-Shum & Ferguson 2012) & Community
analytics and visualizations (Klamma, 2013; Vasileva and Sun, 2007), including Social
Network Analysis (SNA) (de Laat & Schreurs, 2013)
13

14. ANALYTICS LAYERS FOR LEARNING DESIGN FRAMEWORK
Supporting learning design with analytics
14
Community of practitioners
and related stakeholders
Learning Design (LD) tools
Learners experiencing
learning designs
Community Analytics
metrics and patterns of design activity
Design Analytics
metrics of pedagogical decisions
Learning Analytics
metrics of learners’ engagement,
achievement... aligned with design intent
Hernández-Leo, D., et al. (2019) Analytics for learning design: A layered framework and tools, British Journal of Educational
Technology. https://doi.org/10.1111/bjet.12645

15. Supporting learning design with analytics
15

16.  Available data
 ILDE2, extension with data analytics features
— https://www.upf.edu/web/tide/tools/ilde2
 CoT project, 2 schools
 RESET, Smartlet, Illuminated:
— Master program for teacher training
— MOOC for teachers
— Teacher training workshops
Supporting learning design with analytics
16

17. Analytics for learning design
Community of practitioners
and related stakeholders
Community Analytics
metrics and patterns of design activity
 Understanding community behaviour
 Triggering orientation and inspiration
(e.g. awareness, provides a context for your own activity)
 Data includes:
— what tools are being used,
— what types of designs are created, tags,
— who are the (co-)authors, edits, views,
— versioning of designs,
— social ratings …
Michos, K., Hernández-Leo, D., (2016) Understanding
Collective Behavior of Learning Design Communities.
EC-TEL 2016.
Garreta-Domingo, M., Sloep, P., Hernández-Leo, D., Mor, Y.,
(2018) Design for collective intelligence: Pop-up communities in
MOOCs. Journal AI & Society 33(1), 91-100.

18. Analytics for learning design
Community of practitioners
and related stakeholders
Community Analytics
metrics and patterns of design activity
 Understanding community behaviour
 Triggering orientation and inspiration
 Data includes:
— what tools are being used,
— what types of designs are created, tags,
— who are the (co-)authors, edits, views,
— versioning of designs,
— social ratings …
Duplicates ….

19. Analytics for learning design
Community of practitioners
and related stakeholders
Community Analytics
metrics and patterns of design activity
 Understanding community behaviour
 Triggering orientation and inspiration
 Data includes:
— what tools are being used,
— what types of designs are created, tags,
— who are the (co-)authors, edits, views,
— versioning of designs,
— social ratings …
Michos, K., Hernández-Leo, D., (2018) Supporting awareness in communities of
learning design practice, Computers in Human Behavior, 85, 255-270.

20. A community analytics dashboard
 Designs tab, Tools tab, Members tab
20

21.  Blending the dashboard with learning design tasks
 With/Without dashboard use
 Differences in community members´ participation and perceptions
Community dashboard evaluation
21

22.  Dashboard use increased social interactions by means of comments and profile views
 Positive user experience (UMUX questionnaire) for understanding community participation
 Alignment with tasks for learning design
Community dashboard evaluation
22
Variables Group N Ranks U Z p r
profiles views High dashboard use 25 31.26 138.50 -3.238 .001 -0.46
Low dashboard use 24 18.27
comments High dashboard use 25 30.70 157.50 -2.877 .004 -0.41
Low dashboard use 24 19.06
created designs High dashboard use 25 30.12 172.00 -2.570 .010 -0.36
Low dashboard use 24 19.67
re-used designs High dashboard use 25 30.08 173.00 -2.547 .011 -0.36
Low dashboard use 24 19.71
edits High dashboard use 25 29.60 185.00 -2.335 .020 -0.33
Low dashboard use 24 20.21

23. Learning Design (LD) tools
Design Analytics
metrics of pedagogical decisions
 Categorise learning designs
 Support awareness and reflection
about design decisions
 Data includes:
— learning objectives,
— type of tasks tasks,
— the social planes (individual, collaborative),
— resources and tools associated with the task,
— expected length of time for tasks, …
Analytics for learning design
Albo L, Hernández-Leo D. edCrumble: designing for learning with data
analytics. EC-TEL 2018. Best Demo Award.

24. Learning Design (LD) tools
Design Analytics
metrics of pedagogical decisions
 Categorise learning designs
 Support awareness and reflection
about design decisions
 Interaction with the
Community Analytics layer
Analytics for learning design
Albo L, Hernández-Leo D. edCrumble: designing for learning with
data analytics. EC-TEL 2018. Best Demo Award.

25. Learners experiencing
learning designs
Learning Analytics
metrics of learners’ engagement,
achievement... aligned with design intent
 Accumulated evidence of design’s impact
 Support awareness and reflection
about effects of designs, redesign
 Data includes:
— snapshot data capturing relevant milestones
(e.g., access to resources or completion of tasks)
— data about the process (e.g., participation in
discussions)
— performance (e.g., grades)
— students’ and teachers’ opinions…
Analytics for learning design
Michos, K., Manathunga, K., Hernández-Leo, D. (2016). Connecting pattern-based
learning designs with analytics: The case of the PyramidApp., CLAD - EC-TEL 2016

26. Learners experiencing
learning designs
Learning Analytics
metrics of learners’ engagement,
achievement... aligned with design intent
 Accumulated evidence of design’s impact
 Support awareness and reflection
about effects of designs, redesign
 Data includes:
— snapshot data capturing relevant milestones
(e.g., access to resources or completion of tasks)
— data about the process (e.g., participation in
discussions)
— performance (e.g., grades)
— students’ and teachers’ opinions…
Analytics for learning design
Michos, K., Fernández, A., Hernández-Leo, D., & Calvo, R.
(2017). Ld-Feedback App: Connecting Learning Designs with
Students’ and Teachers’ Perceived Experiences. EC-TEL.

27. Learners experiencing
learning designs
Learning Analytics
metrics of learners’ engagement,
achievement... aligned with design intent
 Accumulated evidence of design’s impact
 Support awareness and reflection
about effects of designs, redesign
 Interaction with Design Analytics
and Community Analytics
— Teacher inquiry, Community inquiry
Analytics for learning design
Michos, K., & Hernández-Leo, D., Albó, L. (2018). Teacher-led
inquiry in technology-supported school communities. British
Journal of Educational Technology 49(6), 1077-1095.

28. Teacher-led Inquiry for Learning dEsigns (TILE) tool
28

29. Teacher produced artifacts & Learning Analytics
29
 Learning Analytics reports
 Teacher produced artifacts
 Reflections on the enactment

30. Some results
 Awareness and regulation of design activity (before, during, and after enactment)
 Connection between initial design and data-informed reflections
 Documentation for later review
 Sharing learning design problems with solutions, with contextual information
 Teacher roles as designers is guided by practicality and time constraints
 In-depth discussions about teaching and learning strategies with the use of student generated data
and TILE
30

31. Conclusions
31
 Data driven learning design support in teacher communities
 Support to teachers
 Three analytics layers: Community, Design and Learning analytics
 And interplays!
 Awareness, inspiration, reuse, teacher (and community) inquiry, learning redesign
 Future work:
— New communities (e.g., Maker activities)
— Motivations, incentives
— Socio-emotional aspects and learning design (Spotlighters project)
— Actionable dashboard for PyramidApp
 Concerned about ethical aspects, e.g.
— Beardsley, M., Santos, P., Hernández-Leo, D., Michos, K. (2019). Ethics in educational technology research:
informing participants in data sharing risks. British Journal of Educational Technology. 10.1111/bjet.12781

32. Thank you!
twitter: @daviniahl @TIDE_UPF
website : http://www.upf.edu/web/tide
METIS, RESET, CoT, MdM, SmartLET, Illuminated, Spotlighers