A public lecture by Prof Tony Bryk (President of the Carnegie Foundation for the Advancement of Teaching) on the Design-Educational Engineering and Development (DEED) approach to systemic school improvement, through the use of Improvement Science concepts and Networked Improvement Communities.

1. Anthony S. Bryk
University of Bristol
May 21, 2014

2. 2
Triple Aims of Educational Improvement
EFFICIENCY
EFFECTIVENESS
ENGAGEMENT
Be0er
Use
of
Resources
Deeper
Learning
For
All
Students
More
Relevant

3. Last Decade: Performance Management
Using Accountability to Drive Change
– Set targets
– Create incentives
– Collect data/dashboards
– Hold individuals accountable “Go
figure
it
out
or
else…”

4. Last Decade: Evidence-based
Practice Movement
An academic has
an idea
He/she design and fine
tunes an intervention
An RCT field trial
(5 years later)
Evidence
it can work
Reviewed by What Works Clearing House
Goes on an “approved list”
Districts required or “incented”
to buy only from approved list
Educators “Implement with Fidelity”
Practice Improves!
But there is a problem,
actually many problems.

5. Last Decade: Communities of Practice
5

6. Great
Dynamism,
But
PuLng
All
of
This
Together
6
Resembles
a
Tower
of
Babel
We
have
to
find
a
be0er
way
to
accelerate
learning
to
improve.

7. I. Analogical Scavenger
in hot pursuit!

9. 9
We can
accomplish more
together, than
even the best of us
can do alone.
Complex systems
problems that we
now seek to solve

10. A Better Way: Integrating Two Big Ideas
• Improvement Science Disciplines Efforts
joined to
• Structured Networks that Accelerate the Learning
A
Shi&
Toward
Learning
Fast
to
Implement
Well.

11. II.

12. Two Networked Improvement Communities
! 1. Extraordinary high failure rates in developmental
mathematics courses in community colleges—a
gatekeeper to opportunity.
– Can colleges dramatically increase the success of
these students to and through college-level math in
one year of instruction?
! 2.The weak and incoherent supports by which
individuals learn to teach in public school settings.
– Can districts learn how to support new teachers to
learn faster, better and hold on to them?

13. 13
III. Forming As a Community:
Building Education’s Capacity to Improve

14. The Six Core Principles
14
! 1. Make the work problem-specific and user-centered.
! 2. Variation in performance is the core problem to address.
! 3. See the system that produces the current outcomes.
! 4. We cannot improve at scale what we cannot measure.
! 5. Anchor practice improvement in disciplined inquiry.
! 6. Accelerate improvements through networked communities.

15. Taken Together
15
• Disciplined Inquiry
• Rudiments a scientific community
• Aim: systematic practice improvement

16. The Six Core Principles
16
! 1. Make the work problem-specific and user-centered.
! 2. Variation in performance is the core problem to address.
! 3. See the system that produces current outcomes.
! 4. We cannot improve at scale what we cannot measure.
! 5. Anchor practice improvement in disciplined inquiry.
! 6. Accelerate improvements through networked communities.

17. How Do We Heal Medicine? Atul Gawande April, 2012
See
the
System

18. 18

19. Gawande’s Closing Observation
Making systems work is the great task of my
generation of physicians and scientists.
But I would go further and say that making systems
work — whether in healthcare, education, climate
change, making a pathway out of poverty — is the
great task of our generation as a whole.

20. And at the heart of making systems work
is the problem of complexity…
When you are confronted by any complex social system
with things about it that you’re dissatisfied with and
anxious to fix, you cannot just step in and set about fixing
with much hope of helping.
This realization is one of the sore discouragements …You
cannot meddle with one part of a complex system without
the almost certain risk of setting off disastrous events that
you hadn’t counted on.
If you want to fix something you are first obliged to seek
to understand it…the whole system.
-­‐
Lewis
Thomas,
1974
Biologist
and
Essayist

21. What
We
Should
Work
On:
Task
complexity
Organizaonal
complexity
Complexity
of
work
in
21st
century
instuons
Need
to
“see
the
system”

22. The Invisible Complexity Schooling
22
The
Invisible
Complexity
of
Schooling

23. What
We
Should
Work
On:
Task
complexity
Organizaonal
complexity
Consequences:
• Breakdowns
• Wide
variability
in
outcomes
Complexity
of
work
in
21st
century
instuons
Need
to
“see
the
system”

24. What
We
Should
Work
On:
Task
complexity
Organizaonal
complexity
Consequences:
• Breakdowns
• Wide
variability
in
outcomes
Need
to
idenfy
high-­‐
leverage
problems
Design/develop/
refine
quality
work
processes
Complexity
of
work
in
21st
century
instuons
Need
to
“see
the
system”

25. I.
Problem-­‐
&
User-­‐Centered
• What
we
tend
to
do
now:
a
general
issue
comes
into
view
and
we
jump
on
soluons.
! What
is
the
specific
problem
we’re
trying
to
solve?

26. 60-­‐70%
Students
assigned
to
developmental
math
course.
80%
Percent
of
these
students
that
never
get
past
this
gate.
500,000
students
in
every
cohort
will
never
complete
a
college
math
requirement.
26
A Specific High-Leverage Problem to Solve

27. What
We
Should
Work
On:
How
We
Work
On
It:
Task
complexity
Organizaonal
complexity
Consequences:
• Breakdowns
• Wide
variability
in
outcomes
Need
to
idenfy
high-­‐
leverage
problems
Design/develop/
refine
quality
work
processes
Inherent
indeterminism
Must
learn
our
way
into
improvement;
“Change
it
to
understand
it.”
Complexity
of
work
in
21st
century
instuons
Need
to
“see
the
system”

28. V.
Engage
in
Disciplined
Inquiry
• Design-­‐development
orientaon,
iterave
cycles
– The
Driving
Principle:
Quick,
minimally
intrusive,
and
an
empirical
warrant
–
Mantra:
Learn
Fast,
Fail
Fast,
Improve
Fast!

29. Framing
the
Learning
to
Improve
Challenge
Current Situation Resistant Indifferent Ready
Low
confidence:
Good idea,
But how to
make it work
????
Limited
Capacity
Very Small
Scale Test
Very Small
Scale Test
Very Small
Scale Test
Good
base
Capacity
Very Small
Scale Test
Very Small
Scale Test Small Scale
Test
High
confidence:
Good idea
èè
Can execute
Limited
Capacity
Very Small
Scale Test Small Scale
Test
Large Scale
Test
Good
base
Capacity
Small Scale
Test
Large Scale
Test
Implement

30. Framing
the
Learning
to
Improve
Challenge
Current Situation Resistant Indifferent Ready
Low
confidence:
Good idea,
But how to
make it work
????
Limited
Capacity
Very Small
Scale Test
Very Small
Scale Test
Very Small
Scale Test
Good
base
Capacity
Very Small
Scale Test
Very Small
Scale Test Small Scale
Test
High
confidence:
Good idea
èè
Can execute
Limited
Capacity
Very Small
Scale Test Small Scale
Test
Large Scale
Test
Good
base
Capacity
Small Scale
Test
Large Scale
Test
Implement

31. Framing
the
Learning
to
Improve
Challenge
Current Situation Resistant Indifferent Ready
Low
confidence:
Good idea,
But how to
make it work
????
Limited
Capacity
Very Small
Scale Test
Very Small
Scale Test
Very Small
Scale Test
Good
base
Capacity
Very Small
Scale Test
Very Small
Scale Test Small Scale
Test
High
confidence:
Good idea
èè
Can execute
Limited
Capacity
Very Small
Scale Test Small Scale
Test
Large Scale
Test
Good
base
Capacity
Small Scale
Test
Large Scale
Test
Implement

32. Framing
the
Learning
to
Improve
Challenge
Current Situation Resistant Indifferent Ready
Low
confidence:
Good idea,
But how to
make it work
????
Limited
Capacity
Very Small
Scale Test
Very Small
Scale Test
Small Scale
Test
Good
base
Capacity
Very Small
Scale Test
Small Scale
Test Modest
Scale Test
High
confidence:
Good idea
èè
Can execute
Limited
Capacity
Small Scale
Test Modest
Scale Test
Large Scale
Test
Good
base
Capacity
Modest
Scale Test
Large Scale
Test
Implement
Wide-spread

33. 33
Of Educational Reform
Going Fast, but Learning Slow.

34. 34
We Are in
Good Company
Shil
to
Learning
Fast
to
Implement
Well

35. What
We
Should
Work
On:
How
We
Work
On
It:
Task
complexity
Organizaonal
complexity
Consequences:
• Breakdowns
• Wide
variability
in
outcomes
Need
to
idenfy
high-­‐
leverage
problems
Design/develop/
refine
quality
work
processes
Inherent
indeterminism
Must
learn
our
way
into
improvement;
“Change
it
to
understand
it.”
Complexity
of
work
in
21st
century
instuons
Need
to
“see
the
system”

36. Developing
a
Quality
Process
Reliably
at
Scale
Develop
A
Change
Test
under
mulple
condions
Test
under
increasingly
varied
condions
Make
the
change
permanent
Ini@al
Hunches
System
Changes
1
school
1
administrator
5
schools
Many
administrators
Enre
vercal
team
A
more
diverse
group
of
administrators
District
Wide
All
administrators
Seeing
Task
Complexity
Seeing
Organizaonal
Complexity
Learning
to
improve
feedback
conversaons
between
principals
and
new
teachers
PLAN
DO
ACT
STUDY

37. What
We
Should
Work
On:
How
We
Work
On
It:
Task
complexity
Organizaonal
complexity
Consequences:
• Breakdowns
• Wide
variability
in
outcomes
Need
to
idenfy
high-­‐
leverage
problems
Design/develop/
refine
quality
work
processes
Inherent
indeterminism
Must
learn
our
way
into
improvement;
“Change
it
to
understand
it.”
Can’t
always
see
all
the
consequences
of
what
we
do.
Centrality
of
measurement:
How
will
you
know
Δ
is
an
improvement?
Complexity
of
work
in
21st
century
instuons
Need
to
“see
the
system”

38. You
Cannot
Improve
at
Scale
What
You
Cannot
Measure
• Need
measureable
targets
– But,
you
just
can
not
stand
at
the
end
of
the
line.
• We
need
process
measures
ed
to
intermediate
targets
and
key
process
changes.

39.
TraditionalSequenceStatway
Effects: Time to Complete a College Level Math Course
1
Year
2
Years
Triple the
success
rate in half
the time.
6%
51%
15%

40. Producve
Persistence
Belonging
Uncertainty:
Supporve
social
es
Target:
How
do
we
measure
it?
Mindsets
about
the
value
of
math
Mindsets
about
potenal
to
learn
math
Anxiety
Regulaon
Study
Skills
Conceptual
Task:
reduce
40+
concepts
to
5
core
ideas
focus
on
underlying
malleable
causes
+
change
evidence
Prac@cal
Measurement:
reduce
900
items
to
26
“you
have
3
minutes”
A
Primary
Driver:
key
intermediate
outcome

41. 41
!
Testing a Change Idea: A Starting Strong Package to
Enhance Productive Persistence

42. What
We
Should
Work
On:
How
We
Work
On
It:
Task
complexity
Organizaonal
complexity
Consequences:
• Breakdowns
• Wide
variability
in
outcomes
Need
to
idenfy
high-­‐
leverage
problems
Design/develop/
refine
quality
work
processes
Inherent
indeterminism
Must
learn
our
way
into
improvement;
“Change
it
to
understand
it.”
Can’t
always
see
all
the
consequences
of
what
we
do.
The
Goal:
Quality
reliably
at
scale
• Can
we
get
an
idea
to
work?
• What
will
it
take
to
make
it
work
in
many
other
contexts?
Centrality
of
measurement:
How
will
you
know
Δ
is
an
improvement?
Complexity
of
work
in
21st
century
instuons
Need
to
“see
the
system”

43. II.Variation in Performance Is the
Problem to Solve
• Crical
queson
is
not:
“What
Works?”
But
rather:
“How
to
advance
quality
among
diverse
teachers
engaging
varied
populaons
of
students
and
working
in
different
organizaonal
contexts?”
Goal:
Achieve
quality
reliably
at
scale.

44. Belonging Uncertainty and Stereotype
12%
13%
14%
28%
40%
7%
11%
14%
50%
71%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Never
Hardly
Ever
Somemes
Frequently
Always
Pathways
Dropout
All
students
Black
students
“How
olen,
if
ever,
do
you
wonder:
‘Maybe
I
don't
belong
here?’”
N
=
714
math
students

45. Variation in Pathways Success
Rates by College (n=19)
45
1
23
4
5
6
7
8
9
11
1213
14
15
17
18
19
0%
50%
100%
0% 50% 100%
StatwayStudents
Non-Statway Matched Comparisons
No improvement
line
We also have a failure, why?
What can we learn?
Triple success rate
line

46. VI.
Accelera@ng
Improvement:
Tap
the
Power
of
Networks
• A
source
of
innovaon
• Mulple
fast
tests/refinements
• Improvement
diffusion
A
Learning
Educa/onal
System

47. A
A
Improvement
Networks:
Accelerate
Learning
in
Pracce
for
Improvement
A
B
A
A
A
B
A
A
A
B
A
A
A
B
C
(Englebart,1994)

48. A System of Social Learning to Improve:
Embracing Disciplined Inquiry
Transla@onal
Research
Interven@ons
(Alpha
Labs)
Will
it
work
in
our
context
and
with
out
students,
and
if
so,
how?
Expert
Prac@@oner
Knowledge
(Subnet)
Building
robust
clinical
knowledge
about
effecve
materials
and
instruconal
pracces
(PDSA).
Learning
from
Network
Data
(Hub
Analy@cs)
Learning
from
observed
variability.
Discerning
the
unseen.

49. The Network Improvement Paradigm
49
Researchers vs Users
“Knowers” “Doers”
All Improvers
What Works!
How to Make It Work!
Replicability as the new
Gold Standard.
“Script it” vs. “Every
situation is unique”
Develop Quality
Processes to
Support Complex Work
Individual Autonomy
As Most Prized Norm
Working Together We
Can Accomplish More
"
Implement Fast and Scale Wide
Learn Fast to
Implement Well
"
Focus on
Standard Effect Size
Focus on Sources of
Variability in Performance
"
"
"
"
Developing
Practice-Based
Evidence

50. The Network Improvement Paradigm
“The problem that is managing quality is not
just an intellectual endeavor; it is a pragmatic
one.The point is not just to know what makes
things better or worse; it is to make things
actually better.”
–Dr. Don Berwick, Founder
Institute for Healthcare
Improvement
Learning Fast to ImplementWell to
Achieve Quality Reliably at Scale.
50

51. It is all about accelerating how we learn
in and through practice to improve.
Our thanks to our foundation partners in this work: Carnegie, Gates,
Hewlett, Kresge and Lumina.
Also the Institute of Education Sciences and the National Science
Foundation