Learning to Improve: * How America\'s Social Institutions Can Get Better at Getting Better. November, 2014

Learning to Improve:* How America's Social Institutions Can Get Better at Getting Better November, 2014 * To

be published by Harvard Education Press, 2/2015

A “friend of evidence” for 40 plus years

2

My Starting Points § We confront a growing chasm between unmet social needs, rising aspirations and what our social institutions can routinely accomplish.

§ Current R&D strategies—too slow, too expensive and

the findings often of limited value to the on the ground problems people are trying to solve.

§ Our goal: Learning faster and better in order

to achieve quality outcomes reliably at scale. 3

Current Paradigm: Evidence-based Movement An academic has an idea

He/she design and fine tunes an intervention

An RCT field trial (5 years later)

Evidence it can work

Reviewed by What Works Clearing House Goes on an “approved list”

Districts required or “incented” to buy only from approved list

Educators “Implement with Fidelity”

Practice Improves! But there are major practical and conceptual problems here

The Power and Limits of Evidence-Based Practice §  The genius of randomization –  Being able to isolate the effect of one factor, willfully ignoring everything else.

§  Its limitation –  ”Everything Else” is what actually produces the wide variability (and often unacceptable variability) in outcomes that we continue to observe.

§  A needed complementarity –  Evidence-based practice and practice-based evidence. –  This is key to achieving quality outcomes reliably at scale. 5

Core Principles Organizing Learning to Improve (aka developing practice-based evidence) §  1.

Make the work problem-specific and user-centered.

§  2.

Focus on variation in performance.

§  3.

See the system that produces the current outcomes.

§  4. We cannot improve at scale what we cannot measure. §  5.

Use disciplined inquiry to drive improvement.

§  6.

Accelerate learning through improvement networks. 6

II.Variation in Performance is the Problem to Solve

§ Critical question is not: “What Works?” § It is the Quality Improvement Question: “How to advance effectiveness among diverse practitioners engaging varied populations of children and families and working in different organizational contexts?”

§ Goal: Achieve efficacy with reliability at scale.

An Ideal Case Example § First year results from a

large randomized field trial of Reading Recovery (I3 inititative)

§ Key: a multi-site trial

RCT  (average)  Treatment  Effect:  Reading  Recovery   N=141  schools   16  

14  

12  

It  is  a  success   lets  spread  it!  

10  

8  

6  

4  

2  

0  

-­‐0.5  

-­‐0.3  

-­‐0.1  

0.1  

0.3  

0.5  

0.7  

0.9  

Effect  Size  

1.1  

1.3  

1.5  

1.7  

1.9  

DistribuCon  of  RCT  Treatment  Effects:  Reading  Recovery   N=141  schools   16  

14  

12  

Count  

10  

8  

6  

4  

2  

0  

-­‐0.5  

-­‐0.3  

-­‐0.1  

0.1  

0.3  

0.5  

0.7  

0.9  

Effect  Size  

1.1  

1.3  

1.5  

1.7  

1.9  

DistribuCon  of  RCT  Treatment  Effects:  Reading  Recovery   N=141  schools   16  

14  

12  

Count  

10  

PosiEve     Deviants  

Undesirable/   Weak  Outcomes  

8  

         

         

6  

4  

2  

0  

-­‐0.5  

-­‐0.3  

-­‐0.1  

0.1  

0.3  

0.5  

0.7  

0.9  

Effect  Size  

1.1  

1.3  

1.5  

1.7  

1.9  

III. See the system.

It  is  hard  to  improve    what  we  do  not     fully  understand.  

A  Typical  Approach  to  a  Coaching  IniEaEve   District

School

Salary Policies For Coaches

Hiring & Assignment Policies For Coaches

Coaching

Union Contracts

Credit to: A Framework for Effective Management of School System Performance. Lauren Resnick, Mary Besterfield-Sacre, Matthew Mehalik, Jennifer Zoltners Sherer and Erica Halverson.

And  then  great  things  are  suppose  to   happen…  

Peering  inside  the  “Black  Box”:     the  work  processes  of  coaching  

Does Coaching Work?  

It  is  a  silly  quesEon.  Across  many  different  fields  of   endeavor,  we  know  it  can  be  powerful,  but…     It  is  a  human  and  social  resource  intensive  system.     Even  when  well  planned  out  expect  to  see  variable   quality  in  actual  performance.     What  will  it  take  to  achieve  quality  with  reliability  at   scale?    This  is  the  quesCon  we  need  to  ask  and   where  evidence  must  be  brought  to  bear.      

The Invisible Complexity Schooling The  Invisible  Complexity  of  Schooling  

17  

Achieving  Quality   Reliably  at  Scale  

How Do We Heal Medicine? Atul Gawande April, 2012

Gawande’s Closing Observation

§ Making systems work is the great task of my generation of physicians and scientists.

But I would go further and say that making systems work — whether in healthcare, education, climate change, making a pathway out of poverty — is the great task of our generation as a whole.

19

An Improvement Research Example: The Presenting Problem

60-­‐70%  

Students  assigned  to   developmental  math   course.  

80%  

Percent  of  these   students  that  never   get  past  this  gate.  

500,000   students  

in  every  cohort  will  never   complete  college  math   requirement.  

20

Statway

Traditional Sequence

What Is Possible to Achieve 1  Year  

2  Years    

6%

15%              

51%

  Triple the   success rate in half the time.

It is All About the Details Getting Under the Hood: Root Causes Analysis The  OrienEng   Problem  

     Extraordinarily   high  failure  rates   among  students   assigned  to   developmental   math  instrucEon  

Primary  Causes    for    High  Failure  Rates    

Organizing  Improvement   event   Hypotheses  

Lose  large  #  of  students   at  the  transiEons  

Consolidate  the  courses   into  a  1-­‐year  pathway  

material  +  instrucEon   not  engaging  

Real  world  problems  from   staEsEcs  as  the  organizer  

Embedded  literacy   and  language  barriers     Students  mindsets   undermine  success   Students  “gone”   before  we  know  it   Faculty  pracEces  +   Beliefs  limit  success  

Faculty  development   Psycho-­‐social  intervenEons   aimed  at  “producEve   persistence”   Rapid  analyEcs  capacity  

Emergence of a Working Theory of Improvement The  OrienEng   Problem  

Primary  Causes    for    High  Failure  Rates     Lose  large  #  of  students   at  the  transiEons  

     Extraordinarily   high  failure  rates   among  students   assigned  to   developmental   math  instrucEon  

Course  material  and   instrucEon  are    not   engaging   Students  mindsets   undermine  success   Embedded  literacy   and  language  barriers     Students  “gone”   before  we  know  it  

Organizing  Improvement   Hypotheses   Consolidate  the  courses   into  a  1-­‐year  pathway   Real  world  college-­‐level   math  problems  relevant  to   students  as  the  organizer    Faculty  development   Psycho-­‐social  intervenEons   aimed  at    “producEve  persistence”   A  focus  on    “starEng  strong”    Rapid  analyEcs  capacity  

1. Assessing Change: Initial Evidence of Efficacy of Starting Strong Package

24

!

II. Predictive Analytics and an Elevated Improvement Priority All  students  

Black  students  

Math  Class  Dropout  

80%   70%   60%   50%  

71%  

“How  ogen,  if  ever,  do   you  wonder:  ‘Maybe  I   don't  belong  here?’”

50%   40%  

40%   28%  

30%   20%   10%  

12%  

7%  

13%  11%  

14%  14%  

Hardly  Ever  

SomeEmes  

0%   Never  

Frequently  

Always  

Carnegie  FoundaCon  naConal  survey  of  N  =  714  math  students    

IIIl PDSA Cycle: Rapid, Small Experimental Trials The  Three  QuesCons:     •  What  specifically  are  we  trying  to  accomplish?     •  What  change  might  we  introduce  and  why?     •  How  will  we  know  that  the  changes  are  an  improvement?  

   

PLAN  

DO  

ACT  

STUDY  

Problem:  Declining  Alendance  ager  the  Mid-­‐term  

A Change Idea: Student Group Noticing Routine

Actual observed attendance

Past experience

28

And the network also continue to assess performance context by context… 100%

Positive Deviants

Statway Students

8

Triple success rate line

1 17

50%

6 7 4 5

18 19

14 11 9 3

12

13 2

15 0% 0%

No improvement line

We also have a failure, why? What can we learn?

50% Non-Statway Matched Comparisons

100%

This  is  the  real  work  of  gemng  beler  at  gemng  beler.    

29

VI. Structured Networks Can Accelerate this Learning to Improve

1.  An enormous source of innovation 2.  Social connections accelerate testing and diffusion 3.  Seeing patterns that otherwise look particular 4.  A safe environment for participants to discuss

comparative results • a “learning exchange” • a moral imperative, “if others can, why not us?”

Using Evidence to Get Better at Getting Better “The problem that is managing quality is not just an intellectual endeavor; it is a pragmatic one. The point is not just to know what makes things better or worse; it is to make things actually better.” –Dr. Don Berwick, Founder Institute for Healthcare Improvement Learning Fast to Implement Well to Achieve Quality Outcomes Reliably at Scale. 31

Come Join a Growing Community