More Related Content Similar to Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution- NYC Similar to Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution- NYC (20) More from Daniel Katz (13) Legal Analytics, Machine Learning and Some Comments on the Status of Innovation in the Legal Industry - Professors Daniel Martin Katz & Michael J Bommarito II - Presentation @ The Forum on Legal Evolution- NYC1. daniel martin katz
michael j bommarito
adjunct professor @ university of michigan
associate professor of law @ illinois tech - chicago kent
co-founder @ LexPredict
co-founder @ LexPredict
Legal Analytics, Machine Learning and
Some Comments on the Status of
Innovation in the Legal Industry
4. "We always overestimate the change that will
occur in the next two years and underestimate
the change that will occur in the next ten”
- bill gates
6. today’s focus is primarily
legal analytics + process engineering
© daniel martin katz michael j bommarito
9. three faces of innovation in legal
© daniel martin katz michael j bommarito
10. (1) lawyers for innovators / entrepreneurs
© daniel martin katz michael j bommarito
11. © daniel martin katz michael j bommarito
what most lawyers and law schools think of as “Law+Entrepreneurship"
(1) lawyers for innovators / entrepreneurs
12. (2) lawyers as innovators - substance
© daniel martin katz michael j bommarito
13. poison pill - “the most important innovation in corporate law
since Samuel Calvin Tate Dodd invented the trust
for John D. Rockefeller and Standard Oil in 1879”
© daniel martin katz michael j bommarito
(2) lawyers as innovators - substance
14. emerging areas - 3D Printing, Driverless Cars, Augmented Reality,
Data Breach, Big Data+Privacy, etc.
Drones, Internet of Things, CyberSecurity,
© daniel martin katz michael j bommarito
(2) lawyers as innovators - substance
15. (3) lawyers as innovators - business/process
© daniel martin katz michael j bommarito
16. innovation directed toward transforming the practice of law
© daniel martin katz michael j bommarito
(3) lawyers as innovators - business/process
18. © daniel martin katz michael j bommarito
there are different ways that
organizations are innovating
on the third face
21. © daniel martin katz michael j bommarito
some traditional law firms
have been very aggressive
23. © daniel martin katz michael j bommarito
but most of the innovation
is Lex.Startup
24. © daniel martin katz michael j bommarito
Lex.Startup
is beginning to take hold
28. 15 425+
2009 2014
Law or Legal Related Companies*
as highlighted by Josh Kubicki @ ReInventLaw London 2013
Lex.Startup
42. © daniel martin katz michael j bommarito
So what are these folks doing?
43. R + D Function in the
Legal Industry
© daniel martin katz michael j bommarito
44. We Could Imagine a World
Where Law Firms Did the
R+D for the Industry
© daniel martin katz michael j bommarito
46. © daniel martin katz michael j bommarito
Lex.Startup
is undertaking that function
47. © daniel martin katz michael j bommarito
Here are the specific
approaches that are
being undertaken
48. © daniel martin katz michael j bommarito
Some organizations are
doing more than one
50. © daniel martin katz michael j bommarito
labor
arbitrage
process/tech
arbitrage
51. © daniel martin katz michael j bommarito
labor
arbitrage
process/tech
arbitrage
regulatory
arbitrage
52. © daniel martin katz michael j bommarito
labor
arbitrage
process/tech
arbitrage
regulatory
arbitrage
design as the
ultimate bespoke
53. © daniel martin katz michael j bommarito
labor
arbitrage
process/tech
arbitrage
regulatory
arbitrage
design as the
ultimate bespoke
predictive
analytics
54. © daniel martin katz michael j bommarito
could do an
individual talk on
any of these topics...
55. © daniel martin katz michael j bommarito
labor
arbitrage
process/tech
arbitrage
regulatory
arbitrage
design as the
ultimate bespoke
predictive
analytics
56. © daniel martin katz michael j bommarito
labor
arbitrage
process/tech
arbitrage
regulatory
arbitrage
design as the
ultimate bespoke
predictive
analytics
59. © daniel martin katz michael j bommarito
The Data Driven
Future of the
Legal Industry
72. © daniel martin katz michael j bommarito
Quantitative Legal Prediction
- or -
How I Learned to Stop Worrying and
Start Preparing for the Data Driven
Future of the Legal Services Industry
Daniel Martin Katz
Associate Professor of Law
Michigan State University
62 Emory L. J. 909 (2013)
73. © daniel martin katz michael j bommarito
Cause
and
Effect
Quantitative
Legal
Prediction
vs.
74. © daniel martin katz michael j bommarito
Cause
and
Effect
Quantitative
Legal
Prediction
vs.
75. © daniel martin katz michael j bommarito
Machine Learning
is the heart of
predictive analytics
77. Supervised
Statistical models
Bayesian, e.g., Naïve Bayes Classification
Frequentist, e.g., Ordinary Least Squares
Neural Networks (NN)
Support Vector Machines (SVM)
Random Forests (RF)
Genetic Algorithms (GA)
Semi/Unsupervised
Neural Networks (NN)
Clustering
K-means
Hierarchical
Radial Basis (RBF)
Graph
Some Machine Learning Methods
© daniel martin katz michael j bommarito
81. © daniel martin katz michael j bommarito
Adapted from Slides By
Victor Lavrenko and Nigel Goddard
@ University of Edinburgh
Take A LookThese 12
82. © daniel martin katz michael j bommarito
72
Female
Human
3
Female
Horse
36
Male
Human
21
Male
Human
67
Male
Human
29
Female
Human
54
Male
Human
44
Male
Human
50
Male
Human
42
Female
Human
6
Male
Dog
7
Female
Human
83. © daniel martin katz michael j bommarito
Classification
(Supervised Learning)
decision
boundary
female
male
f( )
Gender?
84. © daniel martin katz michael j bommarito
Classification
(Supervised Learning)
decision
boundary
female
male
f( )
Gender?
Regression
(Supervised Learning)
#f( )
Age?
723
2
3
67
54
29
42
44 50
7
6
27 44 53 3
68
2
48
10
6
743
4
4
85. © daniel martin katz michael j bommarito
Classification
(Supervised Learning)
decision
boundary
female
male
f( )
Gender?
f( )
Loan
Application?
Yes
Multi Class Classification
(Supervised Learning)
No
Maybe
Yes
Perhaps
No
Multiclass =
Boundary
Hyperplane
Regression
(Supervised Learning)
#f( )
Age?
723
2
3
67
54
29
42
44 50
7
6
27 44 53 3
68
2
48
10
6
743
4
4
86. © daniel martin katz michael j bommarito
Classification
(Supervised Learning)
decision
boundary
female
male
f( )
Gender?
f( )
Loan
Application?
Yes
Multi Class Classification
(Supervised Learning)
No
Maybe
Yes
Perhaps
No
Multiclass =
Boundary
Hyperplane
Regression
(Supervised Learning)
#f( )
Age?
723
2
3
67
54
29
42
44 50
7
6
27 44 53 3
68
2
48
10
6
743
4
4
Clustering
(Unsupervised
Learning)
Clusterf( )
Group?
87. © daniel martin katz michael j bommarito
Regression as a Prediction Tool
88. © daniel martin katz michael j bommarito
Regression as a Prediction Tool
89. © daniel martin katz michael j bommarito
Standard Linear Regression
Can Be Used to
Predict a Probability
(using LPM, Logit, etc.)
90. © daniel martin katz michael j bommarito
Standard Linear Regression
Can Be Used to
Predict a Quantity
91. © daniel martin katz michael j bommarito
Task = Predict the Expected Cost of
a Given Legal Service
f( )
Cost?
#
and/or
010
101
001
Regression (Supervised Learning)
92. © daniel martin katz michael j bommarito
http://reinventlawchannel.com/ron-gruner-were-on-a-mission/
93. © daniel martin katz michael j bommarito
Y = βo +/- β1 ( X1 ) +/- β2 ( X2 ) +/- β3 ( X3 ) +/- β4 ( X3 ) +/- β5 ( X3 ) + ε
Y = $151 + $15 ( ) + 161 ( ) + 95 ( ) + 34 ( ) +/- β5 ( ) + ε
Per
100
Lawyers
If Tier 1
Market
is True
Partner
Status
is True
Per
10
Years
Practice
Area
94. © daniel martin katz michael j bommarito
Turn Around and
Use This Model
To Predict Other Lawyers
(also Matters, etc.)
95. © daniel martin katz michael j bommarito
This Requires a Method to Deal
With Changes in Dynamics, etc.
96. © daniel martin katz michael j bommarito
This Requires a Method to Update
the Model as Time Moves Forward
97. © daniel martin katz michael j bommarito
Must Deal With Overfitting
to the Existing Data
100. imagine your client is served
with a request for production
© daniel martin katz michael j bommarito
104. © daniel martin katz michael j bommarito
classification
clustering
regression
dimension reduction
108. © daniel martin katz michael j bommarito
LearningTask = Determine Whether a Given
Document is Relevant?
Relevant
Not Relevant
f( )
relevance?
Binary Classification (Supervised Learning)
and/or
010
101
001
109. take the sample set as
a training set and
use human experts
© daniel martin katz michael j bommarito
110. the use of the human
experts is called
“supervised learning”
© daniel martin katz michael j bommarito
111. in the simple binary case,
ask humans to assign
objects to two piles
© daniel martin katz michael j bommarito
116. What Allows A
Human To Separate
These Two Classes of
Documents?
© daniel martin katz michael j bommarito
118. most vendors are selling a
largely undifferentiated product
© daniel martin katz michael j bommarito
120. to place those
documents in their
respective bins
(i.e. relevant, non-relevant)
© daniel martin katz michael j bommarito
122. machine learning task is
trying to recover (learn)
what separates the
relevant from the
non-relevant documents
© daniel martin katz michael j bommarito
123. once we learn the
rule / boundary
we can apply it to separate
the remain documents into
the two classes
© daniel martin katz michael j bommarito
124. © daniel martin katz michael j bommarito
we want to take what we learn here
125. © daniel martin katz michael j bommarito
we want to take what we learn here
126. © daniel martin katz michael j bommarito
we want to take what we learn here
and apply it here
128. the future of e-discovery
will follow the arc of
machine learning
© daniel martin katz michael j bommarito
131. There Is Learning
Within a Matter
(i.e. learning from a
specific training set)
© daniel martin katz michael j bommarito
132. In other words, it is
possible for the
machine to learn from
the experience of
having processed
documents in the past
© daniel martin katz michael j bommarito
133. both inside a given
company but also
across companies ...
© daniel martin katz michael j bommarito
134. this is how
data aggregation / reusing data
becomes very powerful
© daniel martin katz michael j bommarito
135. data aggregation / reusing data
make the naive into the informed
© daniel martin katz michael j bommarito
136. data aggregation / reusing data
help move from the supervised
to the semi/unsupervised
© daniel martin katz michael j bommarito
141. © daniel martin katz michael j bommarito
The system comes pre-trained
for provisions including:
Title, Parties, Date, Term, Change of
Control, Assignment, Indemnity,
Confidentiality, Governing Law,
License Grant, Bankruptcy, Notice,
Amendment, Non-Solicit, and more.
142. Based on testing, we know our system finds
90% or more of the instances of nearly
every substantive provision it covers.
This 90% number is our system’s recall;
its precision differs by provision by
provision but is consistently very
manageable.
© daniel martin katz michael j bommarito
143. We are able to build custom provisions on
request. Thanks to our highly customized
training algorithms, this process is easy and
relatively automated. We are also engaged
in adding more provisions.
© daniel martin katz michael j bommarito
151. © daniel martin katz michael j bommarito
Model Leverages
Classification Tree
(Tool from Machine Learning)
153. Need a more complex approach
© daniel martin katz michael j bommarito
154. Predicting the Behavior of the
United States Supreme Court:
A General Approach
© daniel martin katz michael j bommarito
Black
Reed
Frankfurter
Douglas
Jackson
Burton
Clark
Minton
Warren
Harlan
Brennan
Whittaker
Stewart
White
Goldberg
Fortas
Marshall
Burger
Blackmun
Powell
Rehnquist
Stevens
OConnor
Scalia
Kennedy
Souter
Thomas
Ginsburg
Breyer
Roberts
Alito
Sotomayor
Kagan
1953 1963 1973 1983 1993 2003 2013
9-0 Reverse
8-1, 7-2, 6-3
19 19 19 19 19 20 20
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
- Reverse
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
0.00
0.25
0.50
0.75
1.00
-
8-1, 7-2, 6-3
9-0
19 19 19 19 19 20 20
155. feature engineering
© daniel martin katz michael j bommarito
The real world gives us raw material, at best.
Typically, you even have to dig the stuff raw
material out of your own unstructured data
158. © daniel martin katz michael j bommarito
Case Prediction
and
Litigation Data
162. © daniel martin katz michael j bommarito
“John Dragseth, a principal at
Fish & Richardson (the most active
IP litigation firm in the United
States, according to Corporate
Counsel magazine), credits Lex
Machina’s database with helping
him spot meaningful but
otherwise hidden trends in IP
litigation—and he won’t give
details. “If you published it, then
people on the other side would
know,” he says.
163. © daniel martin katz michael j bommarito
Notice there is an
offloading of data but
it is up to the end user
to derive meaning
164. © daniel martin katz michael j bommarito
In general, the relevant
consumer market is not
yet mature when it
comes to data science
165. © daniel martin katz michael j bommarito
Difficult to sell machine
learning technology in
instances where the
end user does not have
the right assets in place
167. © daniel martin katz michael j bommarito
Many other examples ...
just starting to come online
168. © daniel martin katz michael j bommarito
Attorney Quality
and Performance
170. © daniel martin katz michael j bommarito
Leveraging Public Data
for Legal Insight
174. © daniel martin katz michael j bommarito
Bulls and Bears
~1984 - 2009 ~2009 - 2014
175. © daniel martin katz michael j bommarito
53 in 2009
58 in 2014
If you were
28 in 1984
than you were
176. © daniel martin katz michael j bommarito
before 2009 most of
the individuals in the
profession have only
known the bull market
177. © daniel martin katz michael j bommarito
it is a bear market now ...
and in a bear market you
need a serious strategy
178. © daniel martin katz michael j bommarito
analytics/data should be
part of that strategy
179. © daniel martin katz michael j bommarito
“data is the oil of the 21st Century”
181. © daniel martin katz michael j bommarito
law < > finance
many elements in law look
like finance did 25 years ago
183. © daniel martin katz michael j bommarito
When it comes to
innovation at the
level that is going
to be needed ...
184. © daniel martin katz michael j bommarito
Assigning a innovation partner
or an innovation committee is
probably not enough
186. © daniel martin katz michael j bommarito
how many
organizations have a
full time data scientist
(data science team)?
187. © daniel martin katz michael j bommarito
need a full scale
and empowered
R+D team
(data science, tech, etc.)
190. daniel martin katz
michael j bommarito ii
adjunct professor of Law @ michigan state university
associate professor of law @ illinois tech - chicago kent
co-founder @ LexPredict
director of research @ reInventLaw laboratory
co-founder @ LexPredict
Forum on Legal Evolution
NYC