Slides for lecture 4 of the course Introduction to Legal Technology at the University of Turku Law School, presented Feb 3 2015. This lecture combines different perspectives on the role of human factors in legal technology: legal reasoning as cognition and how to model it, and software usability as it relates to legal technology.

1. TLS0070 Introduction to
Legal Technology
Lecture 4
Human factors
University of Turku Law School 2015-02-03
Anna Ronkainen @ronkaine
anna.ronkainen@onomatics.com

2. ‘Preliminary try-outs of decision machines
built according to various formal specifications
can be made in relation to selected
administrative or judicial tribunals. The
Supreme Court might be chosen for the
purpose.’
(Harold Lasswell 1955)

3. ‘Can we “feed” into the computer that the judge’s
ulcer is getting worse, that he had fought earlier
in the morning with his wife, that the coffee was
cold, that the defence counsel is an apparent
moron, that the temporarily assigned associate
judge is unfamiliar with the law and besides
smokes obnoxious cigars, that the tailor’s bill was
outrageous etc. etc.?’
(Kaarle Makkonen 1968, translation ar)

4. Say hi to System 1 (1/3)
A bat and a ball cost $1.10 in total. The bat
costs $1.00 more than the ball. How much
does the ball cost? __ cents

5. Say hi to System 1 (2/3)
If it takes 5 machines 5 minutes to make 5
widgets, how long would it take 100 machines
to make 100 widgets? __ minutes

6. Say hi to System 1 (3/3)
In a lake, there is a patch of lily pads. Every
day, the patch doubles in size. If it takes 48
days for the patch to cover the entire lake,
how long would it take for the patch to cover
half of the lake? __ days

7. If you got any of them wrong,
you’re not alone...
Table 1 CRT Scores, by Location
Percentage scoring 0, 1, 2 or 3
Mean "Low” "High”
Locations at which data were collected CRT score 0 1 2 3 N =
Massachusetts Institute of Technology 2.18 7% 16% 30% 48% 61
Princeton University 1.63 18% 27% 28% 26% 121
Boston fireworks display* 1.53 24% 24% 26% 26% 195
Carnegie Mellon University 1.51 25% 25% 25% 25% 746
Harvard University* 1.43 20% 37% 24% 20% 51
University of Michigan: Ann Arbor 1.18 31% 33% 23% 14% 1267
Web-based studies* 1.10 39% 25% 22% 13% 525
Bowling Green University 0.87 50% 25% 13% 12% 52
University of Michigan: Dearborn 0.83 51% 22% 21% 6% 154
Michigan State University 0.79 49% 29% 16% 6% 118
University of Toledo 0.57 64% 21% 10% 5% 138
Overall 1.24 33% 28% 23% 17% 3428
(Frederick 2005)

8. ”As we know, there are known knowns. There
are things we know we know. We also know
there are known unknowns, that is to say, we
know there are some things we do not know.
But there are also unknown unknowns, the
ones we don’t know we don’t know.”
– Donald Rumsfeld (2002)

9. (Un)known (un)knowns
known
unknowns
known
knowns
unknown
unknowns
??

10. (Un)known (un)knowns
known
unknowns
known
knowns
unknown
unknowns
unknown
knowns

11. (Un)known (un)knowns
conscious
ignorance
conscious
knowledge
unconscious
ignorance
unconscious
knowledge

12. Dual-process cognition
System 1
• evolutionarily old
• unconscious, preconscious
• shared with animals
• implicit knowledge
• automatic
• fast
• parallel
• high capacity
• intuitive
• contextualized
• pragmatic
• associative
• independent of general
intelligence
System 2
• evolutionarily recent
• conscious
• distinctively human
• explicit knowledge
• controlled
• slow
• sequential
• low capacity
• reflective
• abstract
• logical
• rule-based
• linked to general intelligence
(Frankish & Evans 2009)

13. Systems 1 and 2 in legal reasoning:
interaction
System 1:
making the
decision
System 2:
validation and
justification
(Ronkainen 2011)

14. What’s that got to do with AI?
- MOSONG, my 1st (and so far only) system
prototype
- built for studying the use of fuzzy logic in
modelling various issues in legal theory
- specifically, the use of Type-2 fuzzy logic for
modelling vagueness and uncertainty
- trademarks initially just a random example
domain
- but the knowledge acquired through this
research also proved useful for TrademarkNow...

15. Classical (crisp) logic
0 1
no yes

16. Fuzzy logic
0 0.5 1
no meh yes

17. Fuzzy logic
0 0.1 0.5 0.9 1
hell no no meh yes hell yes

18. Second-order/Type-2 fuzzy logic
0.1±0.1 0.5±0.2 0.9±0.1
no meh yes

19. Open texture
‘Whichever device, precedent or legislation, is
chosen for the communication of standards of
behaviour, these, however smoothly they work
over the great mass of ordinary cases, will, at
some point where their application is in
question, prove indeterminate; they will have
what has been termed an open texture.’
(Hart 1961)

20. Example of open texture :
No vehicles in a park
‘When we are bold enough to frame some
general rule of conduct (e.g. a rule that no
vehicle may be taken into the park), the
language used in this context fixes necessary
conditions which anything must satisfy if it is to
be within its scope, and certain clear examples
of what is certainly within its scope may be
present to our minds.’ (Hart 1961)

21. A park (and vehicles in it?)
1
2

22. A park...

23. And vehicles in it?

24. Inherent open texture:
No boozing in a park
Section 4
Intake of intoxicating substances
The intake of intoxicating substances is prohibited in public
places in built-up areas [...].
The provisions of paragraph 1 do not concern [...] the intake of
alcoholic beverages in a park or in a comparable public
place in a manner such that the intake or the presence
associated with it does not obstruct unreasonably encumber
other persons’ right to use the place for its intended purpose.
(Public Order Act (612/2003))

25. Mosong: the domain
Article 8
Relative grounds for refusal
1. Upon opposition by the proprietor of an earlier trade mark,
the trade mark applied for shall not be registered:
(a) if it is identical with the earlier trade mark and the goods or
services for which registration is applied for are identical with
the goods or services for which the earlier trade mark is
protected;
(b) if because of its identity with or similarity to the earlier trade
mark and the identity or similarity of the goods or services
covered by the trade marks there exists a likelihood of
confusion on the part of the public in the territory in which the
earlier trade mark is protected; the likelihood of confusion
includes the likelihood of association with the earlier trade
mark.
[...]
(CTM Regulation (40/94/EC))

26. Mosong: the domain
Tentative rule
Article 8
Relative grounds for refusal
1. Upon opposition by the proprietor of an earlier trade mark,
the trade mark applied for shall not be registered:
(a) if it is identical with the earlier trade mark and the goods or
services for which registration is applied for are identical with
the goods or services for which the earlier trade mark is
protected;
(b) if because of its identity with or similarity to the earlier trade
mark and the identity or similarity of the goods or services
covered by the trade marks there exists a likelihood of
confusion on the part of the public in the territory in which the
earlier trade mark is protected; the likelihood of confusion
includes the likelihood of association with the earlier trade
mark.
REFUSAL = MARKS-SIMILAR and GOODS-SIMILAR

27. ‘Training’ set: 119 cases
(Salmi et al 2001)

28. Training set
119 cases from 1997–2000, of which
107 from the Opposition Division (1st instance)
and
12 from the Boards of Appeal (2nd instance)

29. Results for the training set
0
0.2
0.4
0.6
0.8
1

30. Validation set
30 most recent (2002) relevant cases:
20 from the Opposition Division and
10 from the Boards of Appeal
Result*: all cases predicted correctly
* when coded into the system by a domain expert

31. Results for the validation set
0
0.2
0.4
0.6
0.8
1
Opposition Division Boards of Appeal

32. Non-expert validation
• done by non-law students taking a course on
intellectual property law (n=75)
• original validation set in two parts (15+15 cases)
at the beginning and the end of the course
• completed non-interactively through a web form
• correct answer: 54.6±6.5%
• incorrect answer: 25.9±7.5%
• no answer: 19.5±5.2% (± = σ)

33. Non-expert validation
% ±stderr before after own total
group 1 (n=15) 41.3±1.7 65.8±2.8 53.5±1.7
group 2 (n=12) 46.1±2.0 65.0±3.0 55.6±1.9
group 3 (n=48) 43.3±1.3 65.9±1.3 54.7±0.9
total (n=75) 43.4±1.0 65.8±1.1 54.6±0.8

34. Initial conclusions from this work
- it (sort of) works; using fuzzy logic makes
sense in this context
- poses more questions than it answers...
- ...and that’s how I ended up tryin to reverse-
engineer human lawyers rather than just
trying to build systems based on existing
legal theory literature

35. Implications for legal AI
- using rule-based methods has its advantages
- human-readable
- comparatively quick to develop
- modifiable (esp. relevant wrt legislative
changes)
- but they can’t do the work alone
- can’t make sense about situations which they
weren’t specifically built to handle
- real-world complexity needs (sometimes)
statistical/machine-learning approaches

36. Design thinking

37. Law and design
?

38. Design in law
- not (just) about the esthetics of physical
object (wrong faculty for that)
- not about the legal protection of designs
(wrong course for that)
- design as a way to rethink business
processes in law...
- ...and as a way to think about the use of
information in legal applications (UI/UX
design)

39. Design thinking
- Peter Drucker: the job of designers is
“converting need into demand” – figuring
out what people want and giving it to them
(i.e., innovating)
- Tim Brown of IDEO: The challenge for
design thinkers is to “help people to
articulate the latent needs they may not
even know they have”
- desirable, viable, feasible

41. Nudging
- design thinking in (eg) governmental services
- manipulating the choice architecture to help
people make better choices (unconsciously)
- example: organ donation opt-in vs. opt-out,
consent rate ~10% vs. >99%

42. Business model innovation
through service design: Wevorce

44. Wevorce
- “turning every divorce amicable”
- started in 2012, Y Combinator W13 alumn,
founded in Boise, ID, but moved to Silicon
Valley, “divorce architects” operating in ~30
markets across the US
- $2M in venture capital funding

47. Usability in legal informatics

48. A tale of two electric kettles

49. A tale of two electric kettles

50. What is usability?
“Usability is the extent to which a system can
be used by specific users to achieve specified
goals with effectiveness, efficiency and
satisfaction in a specified context of use.”
ISO 9241-210

51. What is usability?
“It is important to realize that usability is not a single, one-dimensional property of
the user interface. Usability has multiple components and is traditionally
associated with these five usability attributes:
- Learnability: The system should be easy to learn so that the user can rapidly
start getting some work done with the system.
- Efficiency: The system should be efficient to use, so that once the user has
learned the system, a high level of productivity is possible.
- Memorability: The system should be easy to remember, so that the casual user
is able to return to the system after some period of not having used it, without
having to learn everything all over again.
- Errors: The system should have a low error rate, so that users make few errors
during the use of the system, and so that if they do make errors they can easily
recover from them. Further, catastrophic errors must not occur.
- Satisfaction: The system should be pleasant to use, so that users are
subjectively satisfied when using it; they like it.”
Nielsen 1993

53. Levels of usability
mental
model
high-level
represented
model
low-level
represented
model
implementation
model

54. Levels of usability: law
mental
model
high-level
represented
model
low-level
represented
model
implementation
model
§§

55. How to implement usability
- evaluation of current systems and processes
- field studies
- mock-ups, paper prototypes
- iterative development
- heuristic evaluation by an expert
- end-user usability testing

56. How to implement usability
- evaluation of current systems and processes
- field studies
- mock-ups, paper prototypes
- iterative development
- heuristic evaluation by an expert
- end-user usability testing

57. Software is not always the answer!
Our project management solution:
(... until a month ago)

58. Legal regulation of usability in Finland...
For example:
- EN 62366:2008 Medical devices - Application of
usability engineering to medical devices
authorized by Chapter 2 of the Medical Supplies
and Equipment Act (629/2010)
- CLC/TS 50459:2005 Railway applications.
Communication, signalling and processing
systems. European rail traffic management
system. Driver-machine interface. Data entry for
the ERTMS/ETCS/GSM-R systems, authorized by
28 § 2 of the Railways Act (555/2006)

59. ...and why it might be a good idea...

60. ...seriously
(Viitanen et al 2011)

61. Levels of usability
mental
model
high-level
represented
model
low-level
represented
model
implementation
model

67. most
important
results at
the top

68. line break technology
hyperlinks to docs

70. bullet
point
technology

72. codes explained
in context

73. Good usability is good for
- increased productivity
- reducing training and support costs
- speeding up development
- speeding up legal processes
- quality of legal decisions
- occupational well-being
So why isn’t there more of it in the legal field?
And why (almost) no research?

74. Questions?