The Human-Computer Interaction perspective on Information Retrieval, Keynote Speech presented at the 15th Dutch-Belgian Information Retrieval Workshop - Nov 25th 2016

1. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
The HCI Perspective on IR
Max L.Wilson
DIR2016 Keynote

5. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
MyTalk
• A brief view of how our fields differ
• A brief history of IR in HCI
• What CHI wants from IR
• The Challenges of giving CHI what it wants from IR
• Examples of getting there

6. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
I admire the fundamentals of
batch-processing IR
Queries
Right
Answers

7. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Interesting Differences
between HCI & IR
• IR is build on top of prior work
- HCI is explore new ideas (not necessarily on top)
• IR is identify a rule to improve system performance
- HCI is to extract a principle of Human Behaviour
- or User Interface Design rule
• IR Research is temporally close to IR implementation
- HCI is 10-15y gap
All the papers in between wrestle with these tensions

8. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Microsoft Surface Dial
image: mashable.com
Fitzmaurice, Ishii, and Buxton.
"Bricks: laying the foundations
for graspable user interfaces."
CHI’95.ACM

9. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
A History of IR in HCI
(not comprehensive)
(or rigorous)
(actually: IR papers in CHI)
(based on the ACM DL’s ranking)
http://bit.ly/IR-HCI-Papers

10. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

11. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
IR papers at CHI
• “Information Retrieval” in CHI in ACM DL
- 5000+ papers sorted* by ‘relevance’
- of 450,000+ IR papers in whole DL
• Export gives me most relevant 1018
- 373 Full Papers, 65 Short Papers, 580 Extended Abstracts
- of Full Papers, only first 25 tag themselves with IR
• Looked at Peer-Reviewed papers in 200 most relevant (N=84)
- Rejected a further 12 as not really being about IR
*Order keeps changing when press ‘next page’ :-/
http://bit.ly/IR-HCI-Papers

12. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
author 'tle year
Jo W. Tombaugh and Sco1 A.
McEwen
Comparison of Two Informa>on Retrieval Methods on Videotex:
Tree-structure Versus Alphabe>cal Directory
1982
G. Fischer and H. Nieper-Lemke Helgon: Extending the Retrieval by Reformula>on Paradigm 1989
Stuart K. Card and George G.
Robertson and Jock D. Mackinlay
The Informa>on Visualizer, an Informa>on Workspace 1991
Michael Mills and Jonathan Cohen
and Yin Yin Wong
A Magnifier Tool for Video Data 1992
H. Ulrich Hoppe and Franz Schiele Towards Task Models for Embedded Informa>on Retrieval 1992
Ben Shneiderman and Christopher
Williamson and Christopher
Ahlberg
Dynamic Queries: Database Searching by Direct Manipula>on 1992
Daniel M. Russell and Mark J. Stefik
and Peter Pirolli and Stuart K. Card
The Cost Structure of Sensemaking 1993
Vicki L. O'Day and Robin Jeffries
Orienteering in an Informa>on Landscape: How Informa>on
Seekers Get from Here to There
1993
Gene Golovchinsky and Mark
Chignell
Queries-R-Links: Graphical Markup for Text Naviga>on 1993
The oldest IR papers in CHI
http://bit.ly/IR-HCI-Papers

13. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

14. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
In general, little evidence for the
superiority of one method can be
found.There were no significant
differences in search times, or in the
number of pages accessed for the
two methods.
[BUT]
Users switched twice as often from
the tree to the directory than from
the directory to the tree.Thus, while
users started with the tree and
directory an equal number of times,
the answer was more often
obtained with the directory
method.

15. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

16. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

17. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

18. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
The concept goes beyond the usual notion of an information
retrieval system to encompass the cost structure of information
from secondary storage to immediate use.

19. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

20. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

21. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

22. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

23. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

24. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
The direct manipulative qualities of incrementability,
reversibility, and smooth graphical feedback encourage the user
to explore the database, freely manipulating the properties
without fear of syntax errors or of getting lost.

25. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

26. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

27. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

28. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

29. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

30. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
author 'tle year Cita'
ons
Kerry Rodden and Kenneth R. Wood How Do People Manage Their Digital Photographs? 2003 4261
Kevin Larson and Mary Czerwinski
Web Page Design: Implica>ons of Memory,
Structure and Scent for Informa>on Retrieval
1998 3884
Elizabeth Sillence and Pam Briggs and
Lesley Fishwick and Peter Harris
Trust and Mistrust of Online Health Sites 2004 3572
Ed H. Chi and Peter Pirolli and Kim Chen
and James Pitkow
Using Informa>on Scent to Model User
Informa>on Needs and Ac>ons and the Web
2001 3188
Stuart K. Card and George G. Robertson
and Jock D. Mackinlay
The Informa>on Visualizer, an Informa>on
Workspace
1991 2789
Bill N. Schilit and Gene Golovchinsky and
Morgan N. Price
Beyond Paper: Suppor>ng Ac>ve Reading with
Free Form Digital Ink Annota>ons
1998 2570
Jurgen Koenemann and Nicholas J. Belkin
A Case for Interac>on: A Study of Interac>ve
Informa>on Retrieval Behavior and Effec>veness
1996 2537
Joseph 'Jofish' Kaye and Janet Vertesi and
Shari Avery and Allan Dafoe and Shay
To Have and to Hold: Exploring the Personal
Archive
2006 2262
Daniel M. Russell and Mark J. Stefik and
Peter Pirolli and Stuart K. Card
The Cost Structure of Sensemaking 1993 2188
The Most Cited IR Papers in CHI
http://bit.ly/IR-HCI-Papers

31. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

32. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Subjects really "liked" the
penetrable version that
allowed them to
manipulate the list of
suggested terms.

33. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Most Recent IR Papers at CHI
author 'tle year
Marta E. Cecchinato and Abigail Sellen and
Milad Shokouhi and Gavin Smyth
Finding Email in a Mul>-Account, Mul>-Device World 2016
Deokgun Park and Simranjit Sachar and
Nicholas Diakopoulos and Niklas Elmqvist
Suppor>ng Comment Moderators in Iden>fying High
Quality Online News Comments
2016
Haizi Yu and Biplab Deka and Jerry O. Talton
and Ranjitha Kumar
Accoun>ng for Taste: Ranking Curators and Content in
Social Networks
2016
Saraschandra Karanam and Herre van
Oostendorp
Age-related Differences in the Content of Search
Queries when Reformula>ng
2016
Benedikt Loepp and Katja Herrmanny and
Jürgen Ziegler
Blended Recommending: Integra>ng Interac>ve
Informa>on Filtering and Algorithmic Recommender
2015
Khalil Klouche and Tuukka Ruotsalo and
Diogo Cabral and Salvatore Andolina and
Designing for Exploratory Search on Touch Devices 2015
Jaimie Y. Park and Neil O'Hare and Rossano
Schifanella and Alejandro Jaimes and Chin-
A Large-Scale Study of User Image Search Behavior on
the Web
2015
Yanir Kleiman and Joel Lanir and Dov Danon
and Yasmin Felberbaum and Daniel Cohen-Or
DynamicMaps: Similarity-based Browsing Through a
Massive Set of Images
2015
Ben Steichen and Luanne Freund
Suppor>ng the Modern Polyglot: A Comparison of
Mul>lingual Search Interfaces
2015
http://bit.ly/IR-HCI-Papers

34. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
A few more IR papers at CHI
http://bit.ly/IR-HCI-Papers

35. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

36. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

37. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

38. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

39. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

40. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

41. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Not all these ideas work
all the time
we know humans do not ‘do’ relevance feedback explicitly
- even though it would help them

42. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Image:Yahoo Research Blog

43. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
add donato paper?

44. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Google’s Wonder Wheel
image: blogoscoped.com

45. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
What does CHI want from IR?
A principle of human behaviour and when it holds true
or
A user interface idea and when it will work
A clear generalisable problem and context being studied
and

46. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Challenges in HCIR
• Effect of Document Collection on Recommendations
- WWW, Collections, Personal Data, Social Media, Multimedia
• Effect ofTask Design - Exploratory is extremely vague
- subjective, comprehensive, comparative, investigative, etc
• >> When should we provide support
• What to Measure about it

47. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
https://ils.unc.edu/searchtasks

48. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
TaskTaxonomy
• Goal: Learn
• Topic: Conceptually General
• Target: Open ended
• Target: Multiple items
• Target: Uncertainty
• Goal:Ambiguous
• Process: dynamic
• Process: Long
• Goal: multi-faceted
• Goal: not too easy

49. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Challenges in HCIR
• Focus of Searching - Options
- WWW, Collections, Personal Data, Social Media, Multimedia
• Task Design - Exploratory is extremely vague
- subjective, comprehensive, comparative, investigative, etc
• >> When should we provide support
• What to Measure about it

50. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
IIiX 2010

51. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Our findings provide further empirical validation toVakkari’s task-
based theory of IR [26], [..] when search interfaces provide
support to the search activities associated with the search
task, effective interaction ensues helping the user address it
efficiently. Otherwise, excessive and unrelated search support
can hamper progress and distract the searcher.

52. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
TeevanThumbnails
only when refinding

53. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

54. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Challenges in HCIR
• Focus of Searching - Options
- WWW, Collections, Personal Data, Social Media, Multimedia
• Task Design - Exploratory is extremely vague
- subjective, comprehensive, comparative, investigative, etc
• >> When should we provide support
• What to Measure about it

55. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Challenges in HCIR
• Focus of Searching - Options
- WWW, Collections, Personal Data, Social Media, Multimedia
• Task Design - Exploratory is extremely vague
- subjective, comprehensive, comparative, investigative, etc
• When should we provide support
• >> What to Measure about it

56. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

57. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
tag cloud papersCHI2011

58. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
CHI2012

59. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Wilson & Wilson

60. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Challenges in HCIR
• Focus of Searching - Options
- WWW, Collections, Personal Data, Social Media, Multimedia
• Task Design - Exploratory is extremely vague
- subjective, comprehensive, comparative, investigative, etc
• When should we provide support
• >> What to Measure about it

61. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

62. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
What is evident is that there
is a cost-benefit in
Search User Interface Design

63. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
BCI Evaluation for IR
• 2009 - Concluded that we needed to understand the
Cognitive Load created by Search User Interfaces
• 2011 - EuroHCIR paper on it
• 2012 - EuroHCIR paper using EEG
• 2012 onwards - focusing on fNIRS

64. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Neuro IR

65. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

66. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
SUI Design + Brain Response
Cognitive LoadTheory
Total Mental Capacity
Simple UI
EasyTask

67. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
SUI Design + Brain Response
Cognitive LoadTheory
Total Mental Capacity
Simple UI
HardTask

68. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
SUI Design + Brain Response
Cognitive LoadTheory
Total Mental Capacity
Complex UI
HardTask

69. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Three RelatedTheories
• Semantic vs Syntactic Load (HCI / CompSci)
• Cognitive LoadTheory (Education)
• Mental Workload (Human Factors & Psychology)

70. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Mental Workload
the limited resource model [22] describing the relat
between the demands of a task, the resources alloca
e task and the impact on performance.
re 3: Resources available vs task demands
act on performance [22]
Megaw,T. (2005)The definition and measurement of mental workload. Evaluation
of human work, 525-551.

71. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Erin’s paper

72. Dr Max L.Wilson http://cs.nott.ac.uk/~mlw/
Measuring Mental Workload
Nasa-TLX
fMRI
EEG
fNIRS

73. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

74. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

75. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw

76. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Challenges in HCIR
• Focus of Searching - Options
- WWW, Collections, Personal Data, Social Media, Multimedia
• Task Design - Exploratory is extremely vague
- subjective, comprehensive, comparative, investigative, etc
• When should we provide support
• >> What to Measure about it

77. Dr Max L.Wilson http://cs.nott.ac.uk/~pszmw
Questions?
http://bit.ly/IR-HCI-Papers