2022_03_28 EDUCON 2022 “Replication of an Evaluation of Teacher Training in the Classification of Programming Exercises Using Bloom’s Taxonomy” - Ángel Velázquez Iturbide

1. Replication of an Evaluation of Teacher Training
in the Classification of Programming Exercises
Using Bloom’s Taxonomy
. Susana Masapanta-Carrión
Pontificia Universidad Católica del Ecuador, Quito, Ecuador
J. Ángel Velázquez-Iturbide
LITE research group, Universidad Rey Juan Carlos, Madrid, Spain

2. • Repositories are a key resource to access high-
quality exercises:
• One central issue in their design and use is classification of
exercises
• According to our systematic review (Masapanta & Velázquez,
2018), Bloom’s taxonomy is used in CSE:
• for programming courses
• to classify assessment tests
Introduction

3. • Difficulties using Bloom’s taxonomy:
• Notable divergent classifications (Johnson & Fuller 2006, Lewis et al.
2013, Oliver et al. 2004, Shuhidan et al. 2009, Thompson et al. 2008, Whalley et
al. 2006)
• Higher confidence than actual accuracy! (Gluga et al. 2012)
• Deficiencies of Bloom’s taxonomy from a formal point of view
(Velázquez, 2021)
• Despite its deficiencies… high appeal and
acceptance of Bloom’s taxonomy in CSE (Masapanta &
Velázquez, 2018)
• Our goal:
• Reducing instructors’ difficulties when using the revised Bloom’s
taxonomy
Introduction

4. • Bidimensional nature of revised Bloom’s taxonomy:
• Cognitive dimension:
• 6 categories of cognitive processes
• 19 cognitive processes
• Type of knowledge dimension:
• 4 types of knowledge
• 11 subtypes of knowledge
• Emphasis on subcategories rather than
on categories
Introduction

5. • Previous proposal and evaluation of three
clarifications on the revised Bloom’s taxonomy
(Masapanta & Velázquez, 2018):
• Classify into the two dimensions of the taxonomy
• Classify an exercise into several subcategories
• Provide CS (programming) examples
• The aim of the paper is to replicate the evaluation:
• With similar experimental settings (and minor enhancements)
• With a different sample of participants
Background

6. • Classification of an exercise into several cognitive
processes:
• Categories in the revised taxonomy do not have a strict hierarchy
• Solving a problem may involve several cognitive processes
• Classifying an exercise implies guessing a sequence of cognitive
processes to solve the exercise:
• Previous identification of educational context
• A most relevant cognitive process can be identified?
Background

7. • Example (Shuhidan et al., 2009, Q2):
• Remember (Recognizing), Factual (Terminology)
• Analyze (Organizing), Conceptual (Classifications)
• Apply (Executing), Procedural (Algorithms)
Background

8. • Intended use of measures:
• Instructors’ outcomes on classifying exercises
• Impact of clarifications on instructors’ outcomes
• Operationalization of outcomes:
• Decomposition
• Variation
• Accuracy
• Confidence
Experimental setting
Objective
Subjective

9. • Research questions:
• Instructors’ outcomes on classifying exercises:
• RQ1: What is the degree of variation of the main cognitive process in
participants’ classification?
• RQ2: What is the percentage of accuracy in participants’ classifications
(compared to ours)?
• RQ3: What is participants’ degree of confidence on their classifications?
• Impact of clarifications on instructors’ outcomes:
• RQ4: What is the degree of decomposition into different cognitive processes
in participants’ classifications?
• RQ5: Do clarifications influence participants’ degree of variation?
• RQ6: Do clarifications influence participants’ percentage of accuracy?
• RQ7: Do clarifications influence participants’ degree of confidence?
Experimental setting

10. • Participants:
• 7 instructors with low to medium familiarity with Bloom’s taxonomy
• Procedure:
Experimental setting
− Video general presentation of
revised Bloom’s taxonomy
− Classify 10 programming
exercises:
• 2 dimensions
• Confidence
− Video with programming
classified examples
− Decompose and classify 9
programming exercises:
• 2 dimensions
• Confidence
20 min.
break

11. • Materials:
• 2 videos
• Diagram with categories and subcategories
• Written summary of the taxonomy
• Nine programming exercises from:
• Fitzgerald et al., 2005
• Shuhidan et al., 2009
• Thompson et al., 2008
• Whalley et al., 2006
Experimental setting

12. • Exercises:
Experimental setting
E1 Answer a question about Java elements
E2 Identify the value of an expression
E3 Identify the generic contents of a variable after a sequence of operations
E4 Identify the specific contents of a variable after a sequence of operations
and conditionals
E5 Identify the string printed after a sequence of statements containing loops
E6 Identify a false statement about methods
E7 Write a method that provides a given functionality
E8 Identify a correct Java implementation of a diagram
E9 Fill a given piece of code to provide a given functionality

13. • Post-session semi-structured interview, with three
questions on:
• What did you like the most from the experience?
• What was the most difficult part of the experience?
• May you suggest ways of improving the instruction given on
Bloom’s taxonomy?
Experimental setting

14. • Variation:
• 1st session:
• 2nd session:
Results
min max med.
Categories of cogn. proc. 2 4 3
Cognitive processes 3 6 4
Types of knowledge 1 2 1
Subtypes of knowledge 2 5 3
min max med.
Categories of cogn. proc. 2 4 4
Cognitive processes 3 6 4
Types of knowledge 2 3 2
Subtypes of knowledge 3 5 4

15. • Decomposition:
Results
E1 E2 E3 E4 E5 E6 E7 E8 E9
minimum 1 2 1 1 2 1 1 2 1
maximum 1 3 3 4 3 2 2 3 4
median 1 3 2 2 2 2 3 2 3
ours 1 3 2 2 3 2 3 2 4

16. • Accuracy:
• 1st session:
• 2nd session:
Results
min max med.
Categories of cogn. proc. 0 6 3
Cognitive processes 0 4 0
Types of knowledge 0 7 4
Subtypes of knowledge 0 6 2
min max med.
Categories of cogn. proc. 1 6 3
Cognitive processes 0 5 3
Types of knowledge 0 6 4
Subtypes of knowledge 0 5 1

17. • Confidence:
Results
1st
session
2nd
session
difference
E1 69% 80% 11%
E2 66% 71% 6%
E3 63% 69% 6%
E4 66% 69% 3%
E5 66% 71% 6%
E6 71% 71% 0%
E7 69% 77% 9%
E8 71% 69% -3%
E9 63% 69% 6%

18. • Summary:
• RQ1: high degree of variation
• RQ2: low degree of accuracy
• RQ3: overconfidence
• RQ4: decomposition similar to ours
• RQ5: moderate increase of variation
• RQ6: small increase of accuracy for cognitive processes, and
decrease for types of knowledge
• RQ7: small increase of confidence
Results

19. • Comparison with the previous evaluation:
• Similar results for all RQs but RQ6 (e.g., impact on accuracy) and
RQ7 (e.g. impact on confidence):
• Minor differences between familiarized and non-familiarized participants
with Bloom’s taxonomy
• Interviews:
• Satisfied with enhanced understanding of the taxonomy
• Many concepts to learn in a limited lapse of time
• More time working with more examples
Results

20. • Similar results in both evaluations
• Semi-structured interviews have provided
additional insights into instructors’ difficulties
• In progress: qualitative analysis of instructors’
discrepancies
Conclusions

21. • Assessment of the proposals:
• Difficulty for classifying also at the type of knowledge dimension
• Decomposition can be successfully adopted
• Need for more programming examples
• Longer training is necessary
• Development by the CSE community of shared classification
criteria
Conclusions

22. Thank you very much!
¡Muchas gracias!