This document summarizes Daniel Katz and Michael Bommarito's presentation on their research at the University of Michigan Center for the Study of Complex Systems. They discuss three areas of their research: using network science to explore the evolution of law and legal institutions; predicting legal outcomes using experts, crowdsourcing, and algorithms; and measuring legal complexity in complex systems using a mean-field approach. They provide examples of projects they conducted in each of these areas while at UM and describe how their current work has built upon this research foundation.
Similar to Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems - Professors Daniel Martin Katz + Michael J Bommarito
INSIDER'S PERSPECTIVE: Three Trends That Will Define the Next Horizon in Lega...LexisNexis
Similar to Law + Complexity & Prediction: Toward a Characterization of Legal Systems as Complex Systems - Professors Daniel Martin Katz + Michael J Bommarito (20)
2. Complexity &
Prediction
daniel martin katz
blog | ComputationalLegalStudies.com
corp | LexPredict.com
page | DanielMartinKatz.com
michael j bommarito
blog | ComputationalLegalStudies.com
corp | LexPredict.com
page | bommaritollc.com
edu | illinois tech - chicago kent law
edu | illinois tech - chicago kent law
Toward a Characterization of Legal Systems as Complex Systems
Law +
lab | theLawLab.comlab | theLawLab.com
6. Here Are a Few
Things from Our Time
@ UM CSCS
3D HD Visualization of Supreme
Court Citation Network
Campaign Contributions and
Legislative Ecosystems
Six Degrees
of
Marbury
v.
Madison
Electronic
World
Treaty
Index
Radial
SCOTUS
Citation
Network
8. Mathematical Formalization of
the United States Code
Congressional Bills
Law as a Seamless Web?
SCOTUS + TAX
Here Are a Few
Things from Our Time
@ UM CSCS
27. Network Science + Law
Exploring the ‘Evolution’ of the Law and its Institutions
28. Network Science + Law
Exploring the ‘Evolution’ of the Law and its Institutions
Experts, Crowds + Algorithms
Three Forms of (Legal) Prediction
29. Network Science + Law
Exploring the ‘Evolution’ of the Law and its Institutions
Three Forms of (Legal) Prediction
Experts, Crowds + Algorithms
Mean-Field Measurement of Risk + Legal Complexity
Measuring Complex Systems
36. Hiring and Placement of
Political Science PhD’s
Co-Sponsorship in Congress
(Fowler et al)
Spread of Obesity
(christakis and fowler)
High School Friendship
(Moody)
Roll Call Votes in Congress
(Mucha, et al)
Social Science
The Political Blogosphere
(Adamic & Glance)
37. Among others, we
undertook some of the early
work (exploring merely a
small subset) of possible
applications for law
42. D. Katz, D. Stafford, Hustle and Flow: A Social
Network Analysis of the American Federal
Judiciary. Ohio St. L. J. 71, 457 (2010)
available at https://papers.ssrn.com/sol3/papers2.cfm?abstract_id=1103573
48. Katz, D. Gubler J. Zelner J. Bommarito M.
Provins E. & Ingall E. (2011). Reproduction
of Hierarchy? A Social Network Analysis of
the American Law Professoriate. Journal of
Legal Education, 61(1), 76-103.
https://papers.ssrn.com/sol3/papers2.cfm?abstract_id=1352656available at
54. There has
Cases Decided by
the Supreme Court
Citations in the
Current Year
Citations from
prior years
watch full video at https://vimeo.com/9427420
58. we observe is the
highly skewed distribution
of authority in legal systems
one of the common features —
59. This is some evidence
supporting the thesis that
Complex SystemsLAW =
60. Katz, et al (2011)
American Legal Academy
Katz & Stafford (2010)
American Federal Judges
Geist (2009)
Austrian Supreme Court
Smith (2007)
U.S. Supreme Court
Smith (2007)
U.S. Law Reviews
Post & Eisen (2000)
NY Ct of Appeals
62. In 2017, 2018 and beyond -
we hope to revisit and expand
upon our earlier work in a
variety of important ways …
(we are open to collaborating
in appropriate instances with
those who are interested)
75. #Predict Relevant Documents
Data Driven EDiscovery/Due Diligence
(Predictive Coding)
#Predict Contract Terms/Outcomes
Data Driven Transactional Work
76. #Predict Relevant Documents
Data Driven EDiscovery/Due Diligence
(Predictive Coding)
#Predict Rouge Behavior
Data Driven Compliance
#Predict Contract Terms/Outcomes
Data Driven Transactional Work
77. #Predict Relevant Documents
#Predict Case Outcomes
Data Driven Legal Underwriting
Data Driven EDiscovery/Due Diligence
(Predictive Coding)
#Predict Rouge Behavior
Data Driven Compliance
#Predict Contract Terms/Outcomes
Data Driven Transactional Work
78. #Predict Relevant Documents
#Predict Case Outcomes
Data Driven Legal Underwriting
Data Driven EDiscovery/Due Diligence
(Predictive Coding)
#Predict Rouge Behavior
Data Driven Compliance
#Predict Contract Terms/Outcomes
Data Driven Transactional Work
#Predict Regulatory Outcomes
Data Driven Lobbying, etc.
79.
80. In so much as prediction is
the task in question …
#LegalTech #FinTech
#Fin(Legal)Tech
81. “The real roll-up of all this isn’t robot lawyers,
it’s financialization, with law becoming an
applied branch of finance and insurance.”
Daniel Martin Katz, professor, Illinois Tech’s Chicago Kent College of Law
http://www.ozy.com/fast-forward/why-artificial-intelligence-might-replace-your-lawyer/75435
89. Columbia Law Review
October, 2004
Theodore W. Ruger, Pauline T. Kim,
Andrew D. Martin, Kevin M. Quinn
Legal and Political
Science Approaches to
Predicting Supreme
Court Decision Making
The Supreme Court
Forecasting Project:
117. “Software developers were asked on
two separate days to estimate the
completion time for a given task, the
hours they projected differed by 71%,
on average.
W h e n p a t h o l o g i s t s m a d e t wo
assessments of the severity of biopsy
results, the correlation between their
ratings was only .61 (out of a perfect
1.0), indicating that they made
inconsistent diagnoses quite frequently.
Judgments made by different people
are even more likely to diverge.”
119. (most pundits did not
identify as a serious
candidate him until
mid-January 2017)
Neil Gorsuch was #1
o n o u r F a n t a s y
Platform 12 Days after
Donald Trump was
elected President
(i.e Nov 20)
123. Columbia Law Review
October, 2004
Theodore W. Ruger, Pauline T. Kim,
Andrew D. Martin, Kevin M. Quinn
Legal and Political Science
Approaches to Predicting
Supreme Court Decision
Making
The Supreme Court
Forecasting Project:
124. Ruger, et al (2004)
relied upon
Brieman(1984)
(as partially shown below)
130. Random forest is an approach to
aggregate weak learners into
collective strong learners
(using a combo of bagging and random substrates)
(think of it as crowd sourcing of models)
131. Our algorithm is a special version
of random forest (time evolving)
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2463244
available at
Revise
+
Resubmit
Version 2.02 -
January 16, 2017
132.
133. We call this a ‘general’ model
of #SCOTUS Prediction
available at
https://arxiv.org/pdf/1612.03473
134. Not just interested in accuracy
over a short time window
available at
https://arxiv.org/pdf/1612.03473
135. A locally tuned model will
typically lead to overfitting
as the dynamics shift
available at
https://arxiv.org/pdf/1612.03473
136. We want a model that is
robust to a large number
of known dynamics …
available at
https://arxiv.org/pdf/1612.03473
137. Version 2.02
January 16, 2017
243,882
28,009
Case Outcomes
JusticeVotes
Current Version of #PredictSCOTUS
1816-2015
138. Version 2.02
January 16, 2017
Current Version of #PredictSCOTUS
1816-2015
case accuracy
70.2%
71.9%
justice accuracy
142. Our Model Against the Null Models
Some commentators had suggested using a heuristic rule of
‘always guess reverse’ as a baseline
(Null Model 1 ) the always guess Reverse model
Turns out it is a lousy
model prior to ~1950
Because the reversal rate
is not stable over time
143. Our Model Against the Null Models
(Null Model 2 ) memory window = inf
This is our model against Null Model 2
What about memory window that selects the most frequent
historical outcome?
(Green = our model out performs)
144. Our Model Against the Null Models
(Null Model 3 ) finite memory window = 10
We in-sample optimize using future information to select a
null model that is among the best performing of all null models
as it is using in-sample info this is a deeply unfair null
145. Over past century, we outperform
M=10 by nearly 5% and have
significant temporal stability at both
the justice, case, term level
154. expert
forecast
crowd
forecast
learning problem is to discover how to blend streams of intelligence
algorithm
forecast
ensemble method
ensemble model
via back testing we can learn the weights
to apply for particular problems
155. By the way, you
might ask why does
one care about
marginal improvements
in prediction ?
#Fin(Legal)Tech
156. Given our ability to offer
forecasts of judicial
outcomes, we wondered
if this information could
inform an event driven
trading strategy ?
157. Revise + Resubmit @
http://arxiv.org/abs/1508.05751
available at
http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2649726
167. M. Bommarito. & D. Katz. A Mathematical Approach to the
Study of the United States Code. Physica A: Statistical Mechanics
and its Applications, 389(19), 4195-4200 (2010).
168. D. Katz & M. Bommarito. Measuring the complexity of the law: the United
States Code. Artificial intelligence and law, 22(4), 337-374. (2014)
171. Paper 1
(Currently Under Review)
M. Bommarito., D. Katz,
Measuring the Temperature
and Diversity of the U.S.
Regulatory Ecosystem (2016)
V available at
https://arxiv.org/pdf/1612.09244.pdf
https://papers.ssrn.com/sol3/papers2.cfm?abstract_id=2891494
2
173. “A Form 10-K is an annual report
required by the U.S. Securities and
Exchange Commission (SEC), that gives
a comprehensive summary of a
company's financial performance.”
174. Data Collection and Pre-Processing
Form 10-K contains
a range of relevant
information about
regulatory exposure
(and other risks)
175. Data Collection and Pre-Processing
“…the Financial Executives
Research Foundation reveals
a mean and median 2015
expense of $1.8M and
$522,205, respectively.”
Real Resources are used to
produced these reports:
176. Data Collection and Pre-Processing
Countervailing Incentives:
Report to get a form of
‘securities fraud insurance’
177. Data Collection and Pre-Processing
Countervailing Incentives:
Report to get a form of
‘securities fraud insurance’
Do not enumerate all risks
under the sun because it
may scare investors
178. Data Collection and Pre-Processing
data is *not* perfect
but it is large scale
characterization of
the manner in which
regulations impact
companies
179. Text of the
10-k for
Company i
in year y
Synonym +
Fuzzy String
Matching to Act,
Agency, U.S. Code
and CFR Masterlist
Gramm Leach
Bliley
Financial
Services
Modernization
Act
GLBA
Graham Leach Bliley
Financial Services
Modernization” Act
Gramm Leach
Bliley Financial
Services
Modernization
Act of 1999
Data Collection and Pre-Processing
190. 0
1
1
0
0
0
0
0
1
…
Encoding
Regulatory
Bitstring for
Company i
in year y
Text of the
10-k for
Company i
in year y
Synonym +
Fuzzy String
Matching to Act,
Agency, U.S. Code
and CFR Masterlist
Gramm Leach
Bliley
Financial
Services
Modernization
Act
GLBA
Graham Leach Bliley
Financial Services
Modernization” Act
Gramm Leach
Bliley Financial
Services
Modernization
Act of 1999
A Mean - Field Characterization
201. In the spirit of
complex systems /
physics, we would
like to try offer a
generalization …
202. We are
undertaking
a version of
this approach
Riley Crane and Didier Sornette. "Robust dynamic
classes revealed by measuring the response function of
a social system." Proceedings of the National Academy
of Sciences 105, no. 41 (2008): 15649-15653.
205. μ
σ
τ auto-correlation
(can be thought of as memory)
set each parameter to base zero at
time = and indext 0
variance2
mean (H, M, L)
(+, -)
(H, M, L)
207. (‘M’ ‘H’, ‘+’)
Anti Kickback
Fairness In Asbestos Injury Resolution
American Clean Energy And Security
Pension Funding Equity
Medicare, Medicaid, And Schip Benefits Improvement
Clusters that have a Similar
Behaviorial Signature
(Not necessariy topically similar)
(‘H’ ‘H’, ‘+’)
Patient Protection And Affordable Care
Secure And Fair Enforcement For Mortgage Licensing
Dodd Frank Wall Street Reform And Consumer Protection
Energy Independence And Security
Tax Relief, Unemployment Insurance Reauthorization
208. In sum, we are able to use this
framework to classify any
regulation by its behavior
upon the broader
regulatory ecosystem
211. Once we have a distance metric
or some manner to encode edges…
212. we can generate a network
projection of the overall
company landscape …
213. Network Generation Procedure
a) Calculate the Hamming distance matrix as described
in Paper #1 over Acts
b) Threshold the matrix by removing all elements whose
distances are greater than D (D=3 in this figure)
c) Generate graph from resulting edge-weighted
adjacency matrix, where edge weight = 1/(1 + d)
d) Layout is Fruchterman-Reingold weighted showing only
giant component
224. Network Science + Law
Exploring the ‘Evolution’ of the Law and its Institutions
Three Forms of (Legal) Prediction
Experts, Crowds + Algorithms
Mean-Field Measurement of Risk + Legal Complexity
Measuring Complex Systems
225. We hope to work with
some of you to advance
the state of the science
in this field
226. Associate Professor of Law
The Law Lab @ Illinois-Tech
Affiliated Faculty
Stanford CodeX
Center for Legal Informatics
Founder + Director
232. Michael J. Bommarito II
@ mjbommar
computationallegalstudies.com
lexpredict.com
bommaritollc.com
illinois tech - chicago kent college of law@
thelawlab.com
233. Daniel Martin Katz
@ computational
computationallegalstudies.com
lexpredict.com
danielmartinkatz.com
illinois tech - chicago kent college of law@
thelawlab.com