Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Word Coach - Pitt Sept 08


Published on

Design of, results from, an empirical study of vocabulary learning from a video game

Published in: Education, Technology
  • Login to see the comments

Word Coach - Pitt Sept 08

  1. 1. <ul><li>Tom Cobb </li></ul><ul><ul><ul><li>Université du Québec à Montréal </li></ul></ul></ul><ul><li>“ Does My Word Coach </li></ul><ul><li>coach words?” </li></ul> This PPT at
  2. 2. Video Game use – exponential increase <ul><li>A great deal of some kind of learning no doubt happens </li></ul><ul><li>But can targeted learning be made to happen? </li></ul><ul><li>Can we exploit this trend rather than fight it? </li></ul>
  3. 3. Many learning claims get made
  4. 4. <ul><li>Few games have specific learning goals </li></ul><ul><li>Claims are about “new ways of learning,” not about any particular content </li></ul><ul><li>Few (any?) claims are based on empirical investigation </li></ul>But more in glossy books than dull research papers
  5. 5. <ul><li>To design a game with concrete learning goals </li></ul><ul><li>Based on learning principles </li></ul><ul><li>In one’s own research area </li></ul><ul><li>And investigates its effects empirically </li></ul>An interesting opportunity therefore…
  6. 7. <ul><li>Game contains all the non-specialist words of English </li></ul><ul><ul><li>Sequenced by frequency/range in the BNC </li></ul></ul><ul><li>Yes-no test determines start-point in sequence </li></ul><ul><li>Words are introduced in six coordinated word games </li></ul><ul><li>Games focus on </li></ul><ul><ul><li>Form </li></ul></ul><ul><ul><li>Meaning </li></ul></ul><ul><ul><li>Verbal efficiency (speed of access) </li></ul></ul><ul><li>Words are dropped only after six correct uses </li></ul><ul><ul><li>Principled recycling </li></ul></ul><ul><li>Games are integrated within a tutor-model </li></ul><ul><ul><li>Choice of tutor </li></ul></ul><ul><ul><li>Goal setting, record keeping, error review </li></ul></ul><ul><li>Game incorporates a level-appropriate dictionary </li></ul><ul><ul><li>CALD </li></ul></ul>My Word Coach - description
  7. 8. <ul><li>Belief: Discovery is not a sufficient strategy to grow an L2 lexicon </li></ul><ul><li>Some type of vocab focus is demonstrably necessary </li></ul><ul><li>Computer /video game is an ideal start to this job </li></ul><ul><ul><ul><li>Although maybe not finish </li></ul></ul></ul>My Word Coach - philosophy
  8. 9. Predecessors 1
  9. 10. Predecessors 2
  10. 11. Predecessors 3
  11. 13. Word Coach’s Pedagogy
  12. 14. Behind-the-scenes pedagogy
  13. 15. Word Coach’s Coaches
  14. 16. Content – 3 strands 1. Form of words
  15. 17. 2. Meaning of words
  16. 19. 3. Processing speed
  17. 20. All packaged in a small box … and announced to the world in a huge publicity campaign - <ul><li> </li></ul>
  18. 21. <ul><li>Are more word meanings known </li></ul><ul><ul><ul><li>both immediately + after delay </li></ul></ul></ul><ul><li>Are more words used </li></ul><ul><ul><ul><li>in oral production </li></ul></ul></ul><ul><li>Are words accessed more quickly? </li></ul>Research Questions Following substantial use of the game…
  19. 22. <ul><li>Grade 6 primary school in suburban Montreal </li></ul><ul><ul><ul><li>Feb-June 2007 </li></ul></ul></ul><ul><li>Two intact classes, same teacher, n=50 </li></ul><ul><ul><ul><li>Condition 1: both groups get to use game </li></ul></ul></ul><ul><ul><ul><li>Condition 2: testing sessions not to exceed 1 hr </li></ul></ul></ul><ul><li>60%+ immigrant children, some quasi-bi/tri-lingual </li></ul><ul><ul><ul><li>French, English, another </li></ul></ul></ul><ul><li>Instruction is in French, </li></ul><ul><ul><ul><li>2 hrs/wk ESL </li></ul></ul></ul><ul><li>Game is integrated into ESL classwork </li></ul><ul><ul><ul><li>Parents solidly behind experiment </li></ul></ul></ul><ul><li>DS players + game disks from Ubisoft </li></ul>Subjects and setting
  20. 23. <ul><li>1. For meanings known </li></ul><ul><ul><ul><li>Paul Nation’s BNC-based 14k Levels test </li></ul></ul></ul><ul><ul><ul><li>10 levels used, then 5 </li></ul></ul></ul><ul><li>2. For words used </li></ul><ul><ul><ul><li>Boy, dog and frog story (Mercer) </li></ul></ul></ul><ul><ul><ul><li>Analyzed with VP-kids </li></ul></ul></ul><ul><li>3. For word access </li></ul><ul><ul><ul><li>UNESCO method </li></ul></ul></ul><ul><ul><ul><ul><ul><li>Read words aloud from 60-word list in fixed time </li></ul></ul></ul></ul></ul>Instruments
  21. 24. 1
  22. 25. 2
  23. 29. (60 words) 3
  24. 30. Research Design
  25. 31. Results To note first: Equal game use between groups (despite considerable variance)
  26. 32. Results 1 1a. Levels Test (pretest, both groups, no diffs) About 2350 word meanings known from first 5000 families but only just over half at key 1k level (600 words, SD 150)
  27. 33. Results 1b. Levels post-test <ul><ul><li>Significant gains for first 5 k-levels combined at T3 </li></ul></ul>
  28. 34. Results 1c. Levels Test (3 times, both groups)
  29. 35. Results – Levels Test - Summary Initial state: No significant difference Baseline (602, T1-T2): No significant difference Following game use 601: 459 words average gain, **sig – but at T3, not T2 602: 178 words average gain, * sig Note: These are average gains Incorporating wide range of variance
  30. 36. Results 2 2a. Frog stories – new words in use
  31. 37. Results 2b. Frog stories – where are these gains?
  32. 38. Results 2c. Frog stories – story size
  33. 39. Results 2c. Frog stories – the main difference
  34. 40. <ul><li>Listen & compare </li></ul><ul><ul><li>Which Frog story is T1, and which is T3? </li></ul></ul><ul><ul><li>From same user </li></ul></ul><ul><ul><ul><li>Just above average use-record </li></ul></ul></ul><ul><ul><ul><li>Francophone </li></ul></ul></ul><ul><ul><ul><ul><li>but no code-switching issue </li></ul></ul></ul></ul><ul><ul><li>Press on to see VP-Kids profiles  </li></ul></ul>
  35. 41. Results - 2e. Example
  36. 42. Results – 2f. Example
  37. 43. Results – Frog stories - Summary Initial state: No significant difference (n.s.d.) - In # families used - In # tokens (=story size) Baseline (602, T1-T2): n.s.d. Following game use Despite variance: - Average 3 new word families - Average 35 more word tokens - Average 45% reduction in # of L1 words used
  38. 44. Results 3 3. Access – number of words read in 1 min.
  39. 45. Results 3. Access – number of words read in 1 min.
  40. 46. Results – Lexical access - Summary Initial state: No significant difference (n.s.d.) Baseline (602, T1-T2): Small but sig gains * Practice effect Following game use Despite variance: - Av. 23 more words read/minute - Statistically significant **
  41. 47. Results 4 – qualitative + post-hoc 4a. Beyond our means
  42. 48. Results – qualitative + post-hoc 4b. Winning profiles
  43. 49. Results – qualitative + post-hoc 4b. Winning conditions
  44. 50. <ul><li>Are more word meanings known </li></ul><ul><ul><ul><li>both immediately + after delay </li></ul></ul></ul><ul><li>Are more words used </li></ul><ul><ul><ul><li>in oral production </li></ul></ul></ul><ul><li>Are words accessed more quickly? </li></ul>Research Questions Following substantial use of the game…
  45. 51. <ul><li>For this population, results are significantly and meaningfully positive on all three measures </li></ul><ul><ul><li>Esp. compared to baselines </li></ul></ul><ul><ul><ul><li>e.g., Milton & Meara (1995): 550 wds/yr in classroom on recognition measure </li></ul></ul></ul><ul><li>But paradoxical </li></ul><ul><ul><li>Lexical development is at opposite ends of the spectrum </li></ul></ul><ul><ul><ul><li>Recognition – gains throughout first 5k </li></ul></ul></ul><ul><ul><ul><li>Production – gains in first 250 words </li></ul></ul></ul>Conclusions
  46. 52. <ul><li>Source of productive gains unclear </li></ul><ul><ul><ul><li>Learning or activation of existing knowledge? </li></ul></ul></ul><ul><li>Apparent confirmation that lexical access is trainable </li></ul><ul><ul><ul><li>Small effect for practice with format; big effect for time-pressure processing </li></ul></ul></ul><ul><ul><ul><ul><li>As known since SkiJump days but never widely exploited </li></ul></ul></ul></ul><ul><li>First successful use of Levels Test as pre-post measure </li></ul><ul><ul><ul><li>Unsurprising since game and test derive from same wordlists </li></ul></ul></ul>Conclusions 2
  47. 53. <ul><ul><li>Significant prior vocab knowledge from somewhere </li></ul></ul><ul><ul><ul><li>2350 known words at pretest </li></ul></ul></ul><ul><ul><li>… but not in use Frog stories of 100 toks / 40 fams </li></ul></ul><ul><ul><ul><ul><li>(cf. age 6 NS: 350 toks / 91 fams,) </li></ul></ul></ul></ul><ul><ul><li>… and distributed so that half the words are unknown at key 1k + 2k levels (# 32) </li></ul></ul><ul><ul><ul><ul><li>Which explains weak reading levels </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Which reflects neglect of lexis in MEQ curriculum </li></ul></ul></ul></ul>Conclusions 3 – Interesting portrait of Quebec ESL learner
  48. 54. <ul><ul><li>Can Word Coach compensate for any of this? </li></ul></ul><ul><ul><li>Effects of moderate-to-heavy users of the game </li></ul></ul><ul><ul><li>Word knowledge : 1k and 2k levels rise to equivalent age native speaker (NS) levels (80% plus) </li></ul></ul><ul><ul><li>Word use : Story size of 250 word, pushing age= NS average of 350 </li></ul></ul><ul><ul><ul><li>Access : 85 words read aloud/minute is approaching NS average of 110 </li></ul></ul></ul><ul><ul><ul><li>A lot of compensation in a short time! </li></ul></ul></ul>Conclusions 3 – Interesting portrait of Quebec ESL learner
  49. 55. <ul><li>Two possible directions </li></ul><ul><li>More of the same </li></ul><ul><ul><li>Game 2 which extends Word Coach approach into collocation, context, idioms, specialized domains </li></ul></ul><ul><li>Integration of Word Coach principles (whole lexicon, sequence, recycle, review) into a full adventure narrative </li></ul><ul><ul><ul><li>“Look out ! They’re shooting from the balcony !!” </li></ul></ul></ul>Next steps for Word Coach
  50. 56. “ Look out – the other one’s in a garbage can!”
  51. 57. Further reading 1 <ul><li>Frequency lists </li></ul><ul><ul><ul><ul><ul><li>Leech et al, Word Frequencies in Written & Spoken English </li></ul></ul></ul></ul></ul><ul><li>Lemmatization procedures </li></ul><ul><ul><ul><ul><ul><li>Nation, unpublished </li></ul></ul></ul></ul></ul><ul><li>20k as size of adult educated lexicon </li></ul><ul><ul><ul><ul><ul><li>Goulding, Nation & Read (1990) </li></ul></ul></ul></ul></ul><ul><li>Yes-No Test </li></ul><ul><ul><ul><ul><ul><li>Buxton & Meara (1987) </li></ul></ul></ul></ul></ul><ul><li>Spaced recycling </li></ul><ul><ul><ul><ul><ul><li>Mondria & Mondria-Wit de Boer (1993) </li></ul></ul></ul></ul></ul><ul><li>Reaction-time & practice </li></ul><ul><ul><ul><ul><ul><li>Snellings, van Geldeen, & de Glopper (2002) </li></ul></ul></ul></ul></ul><ul><li>Easy and hard spelling </li></ul><ul><ul><ul><ul><ul><li>Connor (c.1986), N. Ellis (c.1996), Cognitive processes in spelling </li></ul></ul></ul></ul></ul>
  52. 58. Further reading 2 <ul><li>Baseline 550 words/year classroom </li></ul><ul><ul><ul><ul><ul><li>Milton & Meara </li></ul></ul></ul></ul></ul><ul><li>Pet-200 study </li></ul><ul><ul><ul><ul><ul><li>Cobb, 1997 </li></ul></ul></ul></ul></ul><ul><li>PET-2000 study </li></ul><ul><ul><ul><ul><ul><li>Cobb, 1999a+b </li></ul></ul></ul></ul></ul><ul><li>Francophones and cognates in Levels test </li></ul><ul><ul><ul><ul><ul><li>Cobb 2000 </li></ul></ul></ul></ul></ul><ul><li>Notion of a fixed order of lex growth </li></ul><ul><ul><ul><ul><ul><li>Biemiller and Slonim, 200x </li></ul></ul></ul></ul></ul><ul><li>Need for delayed post in vocab studies </li></ul><ul><ul><ul><ul><ul><li>Cobb, 1999b </li></ul></ul></ul></ul></ul>
  53. 59. <ul><li>Goulden, R., Nation, P., & Read, J. (1990). How large can a receptive vocabulary be? Applied Linguistics 11 , 341-358. </li></ul><ul><li>Meara, P., & Buxton, B. (1987). An alternative to multiple choice vocabulary tests. Language Testing 4 , 142-154. </li></ul><ul><li>Nation, P. (2007). How large a vocabulary is needed for reading and listening? Canadian Modern Language Review 63 (1), 1-12. </li></ul><ul><li>Mondria, J.-A. & Mondria-De Vries, S. (1993). Efficiently memorizing words with the help of word cards and 'hand computer': Theory and applications. System 22, 47-57. </li></ul><ul><li>Snellings, P., van Gelderen, A,, & de Glopper, K. (2002). Lexical retrieval: An aspect of fluent second language production that can be enhanced. Language Learning 52 (4), 723-754. </li></ul>
  54. 60. <ul><li>Cobb, T. (2000). One size fits all? Francophone learners and English vocabulary tests. Canadian Modern Language Review, 57 (2), 295-324. </li></ul><ul><li>Cobb, T. (1999a). Applying constructivism: A test for the learner-as-scientist . Educational Technology Research & Development, 47 (3), 15-33 </li></ul><ul><li>Cobb, T. (1999b). Breadth and depth of vocabulary acquisition with hands-on concordancing. Computer Assisted Language Learning 12 , p. 345 - 360. </li></ul><ul><li>Cobb, T. (1997). Is there any measurable learning from hands-on concordancing? System 25 (3), 301-315. </li></ul>This PPT at
  55. 61. [email_address] This PPT at