Hironori Washizaki, "Rubric-based Assessment of Programming Thinking Skills and Comparative Evaluation of Introductory Programming Environments," 4th International Annual Meeting on STEM Education (IAMSTEM 2021), Keynote, August 12-14, 2021, Keelung, Taiwan and Online

1. 4th International Annual Meeting on STEM Education (IAMSTEM 2021)
August 12-14, 2021, Keelung, Taiwan and Online
Rubric-based Assessment of Programming Thinking
Skills and Comparative Evaluation of Introductory
Programming Environments
Hironori Washizaki
Professor at Waseda University, Tokyo, Japan
washizaki@waseda.jp
http://www.washi.cs.waseda.ac.jp/
Special thanks to Lecturer Dr. Daisuke Saito

2. Prof. Dr. Hironori Washizaki
• Professor and the Associate Dean of the Research
Promotion Division at Waseda University in Tokyo
• Visiting Professor at the National Institute of Informatics
• Outside Directors of SYSTEM INFORMATION and eXmotion
• Research and education projects
• (with Dr. Saito) Leading projects on STEM education with
a particular focus on introductory programming
environments
• Leading a large-scale grant at MEXT enPiT-Pro Smart SE
• Leading framework team of JST MIRAI eAI project
• Professional contributions
• IEEE Computer Society Vice President for Professional
and Educational Activities
• Editorial Board Member of MDPI Education Sciences
• Steering Committee Member of the IEEE Conference on
Software Engineering Education and Training (CSEE&T)
• Associate Editor of IEEE Transactions on Emerging Topics in
Computing
• Advisory Committee Member of the IEEE-CS COMPSAC
• Steering Committee Member of Asia-Pacific Software
Engineering Conference (APSEC)
• Convener of ISO/IEC/JTC1 SC7/WG20

3. Agenda
• Introductory programming and learning tools
• Rubric for programming education
• Case studies and validation of rubric
3

4. Introductory programming for kids
• 4th Industrial revolution: AI, IoT, BigData
– Providing the next generation of children with the
skills to grasp the essence, think logically, and redefine
the world in a procedural manner
– Mandatory education for elementary school students
to learn "programming thinking” in Japan:
Logical/computational thinking, goal-oriented
problem-solving, trial and error process
• How to motivate children to learn programming
and improve their retention?
– Many digital and unplugged (i.e., analog) tools for
introductory programming
4

5. 5
100+

6. Research method
• Target tools
– Searched Web using Google API
– Identified 43 tools in 2017
• Qualitative evaluation of features
– Created a taxonomy
• Quantitative evaluation pre and post learning
workshop
– Assessment of understanding of programming
concepts
– Self-assessment of attitudes toward programming,
computers, and creativity
6

7. Taxonomy for classifying tools
1. Style of Programming
2. Programming Construct
3. Representation of Code
4. Construction of Programs
5. Support to Understand
Programs
6. Designing Accessible
Language
7. Game Elements
8. Supporting Language
9. Operating Environment
10. Interface
11. Experience
7
Daisuke Saito, Ayana Sasaki, Hironori Washizaki, Yoshiaki Fukazawa, Yusuke Muto, “Program Learning for Beginners: Survey
and Taxonomy of Programming Learning Tools,” IEEE 9th International Conference on Engineering Education (ICEED 2017)

8. Classification results (excerpt)
8
Style of Programming
Procedural Object-based Object-oriented Event-based
Game Elements
Scratch
Rule Goal Rewards Cooperation
Beta the Robot https://betatherobot-blog.tumblr.com/
https://scratch.mit.edu/
Daisuke Saito, Ayana Sasaki, Hironori Washizaki, Yoshiaki Fukazawa, Yusuke Muto, “Program Learning for Beginners: Survey
and Taxonomy of Programming Learning Tools,” IEEE 9th International Conference on Engineering Education (ICEED 2017)

9. 9
Workshops
Nikkei News Paper (Oct 6th 2016) Nikkei News Paper (Nov 12th 2016)

10. Assessment quizzes
• Q1. Repeat
• Q2. Conditional
branch
• Q3. Finding a rule
from a given sequence
• Q4. Thinking of
algorithms using the
law in Q3
• Q5. Drawing a free
line through all
squares in one stroke
• Q6. “How did you
draw that line?” in Q5
10
Daisuke Saito, Shota Kaieda, Hironori Washizaki, Yoshiaki Fukazawa, “Rubric for Measuring and Visualizing The
Effects of Programming Learning for Elementary School Students,” Journal of Information Technology Education:
Innovations in Practice (JITE:IIP), Vol. 19, pp.203-227, 2020.
Example of Q3
Example of Q1

11. Assessment score improvement:
pre and post workshop
11
Workshop No.1 (N=45) 第2回 調査報告より
理解度確認テスト得点
学習前 学習後
Workshop No.2 (N=26)
Pre Post Pre Post

12. 12
Q5. Drawing a free line through all
squares in one stroke

13. Scratch
(object-based and event-based)
13
https://news.mit.edu/2013/scratch-two-released-0514

14. Common answer
14
Repeat until all squares are reached:
Move forward while the square ahead can be reached
Turn right
Move forward 1 step
Turn right
Move forward while the square ahead can be reached
Turn left
Move forward 1 step
Turn left

15. Viscuit
(object-based, fuzzy rewriting)
15
DIGITAL POCKET, Dr. Yasunori Harada, https://www.viscuit.com/
Before After
Example: Moving forward
Before After
Rewriting rule

16. Common answer
16
Repeat until all squares are reached:
Move forward while the square ahead is unreached
Turn right

17. Comparison of tools
17

18. Assessment result summary (excerpt)
18
Tool Classification Understanding
Comprehensive
skill
Attitude
Scratch
Visual
language
Overall improvement
Problem-solving
improved
“Programming is easy”
Viscuit
Visual
language
Overall improvement
Problem-solving ,
abstraction ability
improved
Interest in programming
improved
Code
Monkey
Game
Overall improvement
(especially
conditional branch)
Problem-solving,
ability to explain
improved
Motivation for
programming improved
Lightbot Game Overall improvement
Problem-solving
improved
Interest in programming
improved
OSMO
Coding
Tangible Overall improvement
Problem-solving
improved
Interest in programming
improved
Robot
Turtles
Unplugged
Overall improvement
(especially
conditional branch)
Problem-solving
improved
Interest in programming
improved
NOTE: The results are based on the instructor's implementation of specific
content in a limited amount of time and do not capture all the characteristics
of each tool or environment.

19. Summary
• We identified 43 programming learning tools (in
2017)
• We can classify tools based on taxonomy
• Different tools have different learning effects:
Understanding, comprehensive skills, attitudes
• Limitation: Small number of subjects, subjects’
different backgrounds
19
Daisuke Saito, Ayana Sasaki, Hironori Washizaki, Yoshiaki Fukazawa, Yusuke Muto, “Program Learning for Beginners: Survey
and Taxonomy of Programming Learning Tools,” IEEE 9th International Conference on Engineering Education (ICEED 2017)
Daisuke Saito, Hironori Washizaki, Yoshiaki Fukazawa, “Comparison of Text-Based and Visual-Based Programming Input
Methods for First-Time Learners,” Journal of Information Technology Education: Research (JITE: Research), Vol.16, pp.209-226,
2017.

20. Agenda
• Introductory programming and learning tools
• Rubric for programming education
• Case studies and validation of rubric
20

21. Rubric design
• Need to have comprehensive assessment framework
• Levels and related learning outcomes
– SOLO: Pre-structural, Uni-structural, Multi-structural, Relational,
Extended abstract
– Revised Bloom (Krathwohl, 2002): Remember, Understand, Apply,
Analyze, Evaluate, Create
• Learning outcome and observation
– Understanding: Examination (i.e., quizzes)
– Comprehensive skills: Lecturers’ observation on learner’s behavior
– Attitude: Questionnaire-based self assessment
21

22. Rubric design (cont.)
• Evaluation criteria for learning programming
– Computer Science Teachers Association (CSTA) and Association
for Computing Machinery (ACM): K-12 Computer Standards
– International Society for Technology in Education (ISTE):
Standards for Computer Science Educators
22
Programming
and Computing
concepts
Program
design
Program
construction
Read, edit,
evaluate
program
Attitude Self-independent
autonomous
Cooperation
with others

23. Developed rubric (excerpt)
23
Designing
programs

24. Application of the rubric
• Examination (i.e., quizzes)
• Lecturers’ observation on learner’s behavior
• Questionnaire-based self assessment
24
Excerpt of correspondence mapping between quizzes and evaluation criteria

25. Summary
• We developed a comprehensive rubric based
on well-accepted taxonomies (SOLO, Bloom)
and evaluation criteria (CSTA/ACM, ISTE)
• 8 evaluation categories
– Attitude, Programming concept, Computing
concept, Designing programs, Creating programs,
Read-Edit-Evaluate programs, Self-independence,
Cooperation
• 4 levels
• Example mapping to quizzes is provided.
25
Daisuke Saito, Shota Kaieda, Hironori Washizaki, Yoshiaki Fukazawa, “Rubric for Measuring and Visualizing The
Effects of Programming Learning for Elementary School Students,” Journal of Information Technology Education:
Innovations in Practice (JITE:IIP), Vol. 19, pp.203-227, 2020.

26. Agenda
• Introductory programming and learning tools
• Rubric for programming education
• Case studies and validation of rubric
26

27. Rubric as basis for course design and
learning outcome assessment
Course
design
Course (with
observation)
Exam,
questionnaire
Rubric-based
assessment
Feedback and
refinement
Goal and
content design
based on
evaluation
criteria
Visualize
learning
outcome
Individual
feedback and
course
refinement
27

28. Case studies
28
Course Content
Workshop 90min
Programming
Community
ICT club
activity
120min * 5
Programming
Circuit design
Crafting
Teamwork
Elementary
school course
45min * 8
Robot
programming

29. Team presentation and demonstration
29

30. Case of elementary school course (N=19)
30
Concepts of sequence and loop
have been pre-understood.
Good improvement in problem
analysis and extraction while
deepening programming concepts.
Pre
Sequence
Repeat
Conditional
Subdivision
Analysis Extraction
Construction
and
functionalization
Design
document
Programming
language
Post
Sequence
Repeat
Conditional
Subdivision
Analysis Extraction
Construction
and
functionalization
Design
document
Programming
language

31. Summary of cases
• Confirmed different learning effects by courses
• Need to have more data and further validation
31
Course Content Particularly learned Overall
Workshop 90min
Programming
• Sequence
• Loop
• Conditional
Basic programming
concepts learned
Community
ICT club
activity
120min * 5
Programming
Circuit design
Crafting
Teamwork
• Loop
• Conditional
• Subdivision
• Design document
Some learning in
abstraction and
problem-solving
Elementary
school
course
45min * 8
Robot
programming
• Subdivision
• Analysis
• Construction
• Design document
Significant learning in
abstraction and
problem-solving

32. Summary
• Rubric can be used as basis for course design and learning
outcome assessment
• Different course types can be compared based on the
same rubric: Workshop, ICT club activity, School course
• Different course types have different learning effects
– Case of school course: Concepts of sequence and loop have
been pre-understood. There were good improvement in
problem analysis and extraction while deepening programming
concepts.
• Further validation (undergoing):
– Sampling Adequacy: E.g., Kaiser-Meyer-Olkin (KMO) test
– Reliability and consistency: E.g., Cronbach’s coefficient alpha
32
Daisuke Saito, Shota Kaieda, Hironori Washizaki, Yoshiaki Fukazawa, “Rubric for Measuring and Visualizing The
Effects of Programming Learning for Elementary School Students,” Journal of Information Technology Education:
Innovations in Practice (JITE:IIP), Vol. 19, pp.203-227, 2020.
Daisuke Saito, Shota Kaieda, Risei Yajima, Hironori Washizaki, Yoshiaki Fukazawa, Hidetoshi Omiya, Misaki
Onodera, Idumi Sato, “Assessing Elementary School Students’ Programming Thinking Skills using Rubrics,”
IEEE International Conference on Teaching, Assessment, and Learning for Engineering (IEEE TALE 2020)

33. Summary of summaries
• There are 40+ available programming learning tools.
– Tools can be classified based on taxonomy.
– Different tools have different learning effects.
– Tools can be adopted based on their characteristics and
fitness to learning goals.
• Rubric can be used as basis for course design and learning
outcome assessment.
– Different course types can be compared based on the same
rubric.
– Different course types have different learning effects.
– Courses can be designed based on a rubric (or framework)
and refined by monitoring assessment results.
• Future: More data and further validation
33