A 10 minute presentation of a case study

1. Case study
From evidence to
decisions
Frameworks for Clinical
Guidelines’ Development
Carlos A. Cuello, MD, PhD(c)
Health Research Methodology Program!
Department of Clinical Epidemiology and Biostatistics
March 27, 2014

2. Disclosure
Authors belong to the CE&B department and
to the GRADE / DECIDE working group.
No other conflicts to declare.

3. Usefulness of the Evidence to DECISION table
for clinical practice guidelines: a case study
with a panel of international experts
Carlos Cuello-Garcia, Jan Brozek, Juan José Yepes-Nuñez, Yuan Zhang, Shreyas Gandhi, Arnav
Agarwal, Holger Schünemann
Clinical Epidemiology and
Biostatistics
McMaster University
MacGRADE Center

4. What are these
frameworks?
How can they help?
Why should we use
them?

5. decision
(recommendation)
research
evidence
benefits vs
harms
resource use
patient
values
equity
feasible?
acceptable?
Clinical Guideline

6. FRAMEWORKS
evidence to decision (EtD)

7. just
4 columns
&
conclusions

8. CRITERIA
Problem
Quality of evidence
Benefits & harms
Values
Resource use
Equity
Acceptability
Feasibility
JUDGEMENTS
RESEARCH
EVIDENCE
ADDITIONAL
INFORMATION

9. JUDGEMENTS
RESEARCH
EVIDENCE
ADDITIONAL
INFORMATION
CRITERIA
Problem
Quality of evidence
Benefits & harms
Values
Resource use
Equity
Acceptability
Feasibility

10. JUDGEMENTS
RESEARCH
EVIDENCE
ADDITIONAL
INFORMATION
CRITERIA
Problem
Quality of evidence
Benefits & harms
Values
Resource use
Equity
Acceptability
Feasibility

11. JUDGEMENTS
RESEARCH
EVIDENCE
ADDITIONAL
INFORMATION
CRITERIA
Problem
Quality of evidence
Benefits & harms
Values
Resource use
Equity
Acceptability
Feasibility

12. JUDGEMENTS
RESEARCH
EVIDENCE
ADDITIONAL
INFORMATION
CRITERIA
Problem
Quality of evidence
Benefits & harms
Values
Resource use
Equity
Acceptability
Feasibility

13. CONCLUSIONS
Balance of consequences
Decision /recommendation
Justification
Implementation considerations
Monitoring
Evaluation
Research priorities
…

14. PURPOSE OF THE FRAMEWORK

15. PROS CONS
Inform decision makers’
judgements about the pros and
cons of each option (intervention)
that is considered.

16. The important factors that
determine a decision (criteria)
are considered.

17. Structure discussion and
manage disagreements
Make the basis for decisions transparent
to guideline users

18. Provide a concise summary of the
best available research evidence
to inform judgements about each
criterion

19. A case study . . .

20. Guideline for Allergic
Diseases Prevention

21. INTERNATIONAL
PANEL

22. guidelines on allergy prevention
• Probiotics
• Prebiotics
• Vitamin D

23. guidelines on allergy prevention
• Probiotics
• Prebiotics
• Vitamin D

24. Explanations Prepared by: Holger, Jan, Carlos, Juan
Date: December 10, 2013
Generic EtD framework 1
Evidence to decision framework
Question 1: Should probiotics vs. no probiotics be used in pregnant women?
Population:!pregnant!women!
Option:!probiotics!
Comparison:!no!probiotics!
Setting:3outpatient!
Perspective:3individual!patient
Background: The$intestinal$microbiome$could$play$an$important$role$in$the$immune$system$maturation,$
and$it$has$been$suggested$that$early6life$probiotic$administration,$whether$directly$to$the$infant$or$in$
their$mothers$breast$milk,$may$reduce$the$risk$of$allergies$in$childhood.$The$objective$of$this$question$is$
to$evaluate$the$impact$of$probiotics$administered$to$the$expecting$mothers$on$their$infant.
Subgroup considerations: subpopulation of women at high risk for allergy in a child
CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL INFORMATION
PROBLEM
Is the
problem a
priority?
No Probably no Uncertain Probably yes Yes
X
Allergic diseases represent a spectrum of health conditions and a worldwide burden in different
populations. (1)
Are a large
number of
people
affected?
No Probably no Uncertain Probably yes Yes
X
As many as 40% of the worldwide population is affected by any type of allergy. In infants
prevalence depends highly on the allergic status of their parents, being approximately of 10% in
those without an allergic parent or sibling, versus 20% to 30% in those with the atopic
background in their relatives. (2)
Prepared by: Holger, Jan, Carlos, Juan
Date: December 10, 2013

25. Explanations Prepared by: Holger, Jan, Carlos, Juan
Date: December 10, 2013
Generic EtD framework 2
CRITERIA JUDGEMENTS RESEARCH EVIDENCE
ADDITIONAL
CONSIDERATIONS
VALUES
Is there
important
uncertainty
or variability
about how
much people
value the
main
outcomes?
Important
uncertainty
or
variability
Possibly
important
uncertainty
or
variability
Probably
no
important
uncertainty
or
variability
No
important
uncertainty
or
variability
No known
undesirable
outcomes
X
Detailed judgements
The relative importance or values of the main outcomes of interest:
Outcome Relative importance Certainty of the evidence
Eczema critical low
Asthma/wheezing critical low
Food allergy critical low
Adverse effects critical low
We judged that the outcomes
eczema, asthma and food
allergy are critical for people.
The adverse outcomes are
probably of high importance
and the burden of taking daily
pills is limited. Some
immunocompromised women
might not accept the risk.
BENEFITS&HARMSOFTHEOPTIONS
What is the
overall
certainty of
the evidence
of
effectiveness?
No
included
studies Very low Low Moderate High
X
Summary of findings:
Outcome With
[intervention]
Without
[intervention]
Difference
(per 100)
(95%CI)
Relative
effect (RR)
(95%CI)
Certainty
of the
evidence
(GRADE)
Eczema
(follow-up 1 to 5 years)
365/1520
(24%)
484/1515
(31.9%)
9 fewer per
100 (from 4
fewer to 13
fewer)
RR 0.72 (0.6 to
0.86)
⊕⊝⊝⊝
VERY LOW
Asthma/wheezing
(follow-up 2 to 7 years)
143/992
(14.4%)
139/982
(14.2%)
0 fewer per
100 (from 3
fewer to 3
more)
RR 0.97 (0.77
to 1.22)
⊕⊕⊝⊝
LOW
Food allergy
(follow-up 1 to 2 years)
36/279
(12.9%)
41/284
(14.4%)
1 more per
100 (from 3
fewer to 8
more)
RR 1.08 (0.73
to 1.59)
⊕⊝⊝⊝
VERY LOW
Adverse effects 101/394
(25.6%)
88/397
(22.2%)
3 more per
100 (from 4
fewer to 12
more)
RR 1.13 (0.82
to 1.52)
⊕⊝⊝⊝
VERY LOW
Link to detailed evidence profile
Subgroup considerations:
Link(s) to summary of findings and judgments for subgroups
The data are indirect for all
outcomes because they are
primarily derived from studies
that looked at mixed exposure
in women during pregnancy
and breastfeeding and of
infants after birth. Only 1 RCT
assessed the effect on
eczema in pregnant women
only: RR 0.88 (0.63 to 1.22);
RD 5 fewer per 100 (from 14
fewer to 9 more)
5 RCTs included pregnant
women + later breastfeeding
mothers: RR 0.5 (0.4 to 0.63);
RD 21 fewer per 100 (from 15
fewer to 25 fewer)
5 RCTs included pregnant
women + infants after
birth (follow-up 1 to 5 years):
RR 0.87 (0.72 to 1.04), RD 4
fewer per 100 (from 8 fewer
to 1 more)
3 RCTs included pregnant
women + subsequently
breastfeeding +
infants (follow-up 3 to 4
years): RR 0.78 (0.49 to
1.24); RD 7 fewer per 100
(from 17 fewer to 8 more)
No effects were observed on
asthma/wheezing and food
allergy.
How
substantial
are the
desirable
anticipated
effects?
Don’t
know
Not
important
Somewhat
important
Moderately
important
Very
important
Varies
X X
Detailed judgements
There was some disagreement among panel members
whether the effect is somewhat or moderately important.
How
substantial
are the
undesirable
anticipated
effects?
Don’t
know
Very
important
Moderately
important
Somewhat
important
Not
important
Varies
X
Detailed judgements
No serious adverse effects, and no difference in mild
adverse effects between the groups.
Do the
desirable
effects
outweigh the
undesirable
effects?
No Probably
No
Don’t know Probably
Yes
Yes Varies
X
Detailed judgements
Prepared by: Holger, Jan, Carlos, Juan
Date: December 10, 2013

26. Explanations Prepared by: Holger, Jan, Carlos, Juan
Date: December 10, 2013
Generic EtD framework 2
CRITERIA JUDGEMENTS RESEARCH EVIDENCE
ADDITIONAL
CONSIDERATIONS
VALUES
Is there
important
uncertainty
or variability
about how
much people
value the
main
outcomes?
Important
uncertainty
or
variability
Possibly
important
uncertainty
or
variability
Probably
no
important
uncertainty
or
variability
No
important
uncertainty
or
variability
No known
undesirable
outcomes
X
Detailed judgements
The relative importance or values of the main outcomes of interest:
Outcome Relative importance Certainty of the evidence
Eczema critical low
Asthma/wheezing critical low
Food allergy critical low
Adverse effects critical low
We judged that the outcomes
eczema, asthma and food
allergy are critical for people.
The adverse outcomes are
probably of high importance
and the burden of taking daily
pills is limited. Some
immunocompromised women
might not accept the risk.
BENEFITS&HARMSOFTHEOPTIONS
What is the
overall
certainty of
the evidence
of
effectiveness?
No
included
studies Very low Low Moderate High
X
Summary of findings:
Outcome With
[intervention]
Without
[intervention]
Difference
(per 100)
(95%CI)
Relative
effect (RR)
(95%CI)
Certainty
of the
evidence
(GRADE)
Eczema
(follow-up 1 to 5 years)
365/1520
(24%)
484/1515
(31.9%)
9 fewer per
100 (from 4
fewer to 13
fewer)
RR 0.72 (0.6 to
0.86)
⊕⊝⊝⊝
VERY LOW
Asthma/wheezing
(follow-up 2 to 7 years)
143/992
(14.4%)
139/982
(14.2%)
0 fewer per
100 (from 3
fewer to 3
more)
RR 0.97 (0.77
to 1.22)
⊕⊕⊝⊝
LOW
Food allergy
(follow-up 1 to 2 years)
36/279
(12.9%)
41/284
(14.4%)
1 more per
100 (from 3
fewer to 8
more)
RR 1.08 (0.73
to 1.59)
⊕⊝⊝⊝
VERY LOW
Adverse effects 101/394
(25.6%)
88/397
(22.2%)
3 more per
100 (from 4
fewer to 12
more)
RR 1.13 (0.82
to 1.52)
⊕⊝⊝⊝
VERY LOW
Link to detailed evidence profile
Subgroup considerations:
Link(s) to summary of findings and judgments for subgroups
The data are indirect for all
outcomes because they are
primarily derived from studies
that looked at mixed exposure
in women during pregnancy
and breastfeeding and of
infants after birth. Only 1 RCT
assessed the effect on
eczema in pregnant women
only: RR 0.88 (0.63 to 1.22);
RD 5 fewer per 100 (from 14
fewer to 9 more)
5 RCTs included pregnant
women + later breastfeeding
mothers: RR 0.5 (0.4 to 0.63);
RD 21 fewer per 100 (from 15
fewer to 25 fewer)
5 RCTs included pregnant
women + infants after
birth (follow-up 1 to 5 years):
RR 0.87 (0.72 to 1.04), RD 4
fewer per 100 (from 8 fewer
to 1 more)
3 RCTs included pregnant
women + subsequently
breastfeeding +
infants (follow-up 3 to 4
years): RR 0.78 (0.49 to
1.24); RD 7 fewer per 100
(from 17 fewer to 8 more)
No effects were observed on
asthma/wheezing and food
allergy.
How
substantial
are the
desirable
anticipated
effects?
Don’t
know
Not
important
Somewhat
important
Moderately
important
Very
important
Varies
X X
Detailed judgements
There was some disagreement among panel members
whether the effect is somewhat or moderately important.
How
substantial
are the
undesirable
anticipated
effects?
Don’t
know
Very
important
Moderately
important
Somewhat
important
Not
important
Varies
X
Detailed judgements
No serious adverse effects, and no difference in mild
adverse effects between the groups.
Do the
desirable
effects
outweigh the
undesirable
effects?
No Probably
No
Don’t know Probably
Yes
Yes Varies
X
Detailed judgements
Prepared by: Holger, Jan, Carlos, Juan
Date: December 10, 2013

27. Explanations Prepared by: Holger, Jan, Carlos, Juan
Date: December 10, 2013
Generic EtD framework 3
CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL CONSIDERATIONS
RESOURCEUSE
How large are
the resource
requirements?
Large
costs
Moderate
costs
Small Moderate
savings
Large
savings
Varies
X
Detailed judgements
Prices are likely to vary substantially depending on the
setting. This may be a particularly important
consideration in low and middle-income countries. A
level and type of insurance may play a substantial role
as well.
From a health systems point of view it might also be cost
effective given that probiotic would be used for 9 months
and cost of treatment of eczema may be distributed
across many years.
Extremely limited research evidence (internet searches of drug prices)
Bifidobacterium
bifidum (cost per
person per year
US$) Dose: 1 pill
each day
Lactobacillus
gg (cost per
person per
year US$) 1
pill each day
North-
America
Average $181.16 $341.6
South-
America
Average $174.3 $286
Europe
Average $167.86 $251.56
Fewer office visits would occur as a result of eczema if the
effects on eczema were true.
How large is
the
incremental
cost relative
to the net
benefit?
Very
large
ICER
Large
ICER
Moderate
ICER
Small
ICER
Savings Varies
Detailed judgements
No research evidence
If eczema was reduced the intervention might be cost-
effective given fewer office visits (between $17,400 and
$34,100 to treat 100 people for 1 year or ¾ of that for 9
months) preventing 9 cases of eczema.
In most studies probiotics were used in the last trimester of
pregnancy, which, if used this same way, might reduce the
cost per pregnant woman.
EQUITY
What would
be the impact
on health
inequities?
Increased Probably
increased
Uncertain Probably
reduced
Reduced Varies
X
Detailed judgements
No research evidence
In some settings it may be important to consider equity as
the access may depend on socioeconomic status of the
country or setting where coverage will depend on
policymakers.
ACCEPTABILITY
Is the option
acceptable
to key
stakeholders?
No Probably
No
Uncertain Probably
Yes
Yes Varies
X
Detailed judgements
No research evidence
Prepared by: Holger, Jan, Carlos, Juan
Date: December 10, 2013

28. Explanations Prepared by: Holger, Jan, Carlos, Juan
Date: December 10, 2013
Generic EtD framework 4
CRITERIA JUDGEMENTS RESEARCH EVIDENCE ADDITIONAL CONSIDERATIONS
FEASIBILITY
Is the option
feasible to
implement?
No Probably
No
Uncertain Probably
Yes
Yes Varies
X
Detailed judgements
No research evidence
Prepared by: Holger, Jan, Carlos, Juan
Date: December 10, 2013

29. Explanations Prepared by: Holger, Jan, Carlos, Juan
Date: December 10, 2013
Generic EtD framework 5
Recommendation Should probiotics vs. no probiotics be used in pregnant women (exposing their children in utero)?
Overall balance of consequences Undesirable consequences
clearly outweigh desirable
consequences
Undesirable consequences
probably outweigh desirable
consequences
The balance of desirable and
undesirable consequences indicates
they are very similar*
Desirable consequences
probably outweigh undesirable
consequences
Desirable
consequences
clearly outweigh
undesirable
consequences
! ! ! X !
We recommend against the
option or for the alternative
We suggest not to use the option or to use the
alternative
We suggest using the option We recommend the
option
! ! X !
Panel decisions 3 panel members with potential COI recused themselves from participating in formulating the recommendation. Consensus was obtained from the rest of the team.
Recommendation (text)
The guideline panel suggests using probiotics in pregnant women at high risk for allergy in their children (conditional recommendation, very low quality
evidence).
Remarks and justification Most studies commenced probiotics in the last trimester of pregnancy.
The very low quality evidence for adverse effects indicates that our confidence in the absence of increased adverse effects is low. Future research is needed (see
definitions of very low quality) e.g., generalizing to immune-compromised children
Subgroup considerations Women with high risk of allergy in their children
Women with average risk of allergy in their children
Implementation considerations This recommendation is based on trials investigating the probiotics or mixtures of probiotic listed below. We have not found a difference between these different
probiotics, but that does not mean there is no difference
Monitoring and evaluation
considerations
Research priorities Develop instruments for evaluating the risk of allergy in children as the family history predicts only about 30% of the population risk. There is some evidence that first child
is at higher risk for allergy than subsequent children. Long-term follow-up of long-term effects.
No direct evidence for the use of probiotics in formula – this should be evaluated in future research and is an unmet need.
Prepared by: Holger, Jan, Carlos, Juan
Date: December 10, 2013

30. SURVEY

31. 24 panel members
13 RESPONDED

32. 11 ACCEPTED TO
COMPLETE THE
SURVEY

33. 1. The EtD table helped in the ORGANIZATION of the
development process (i.e., for getting a
recommendation from a body of evidence)
Strongly
disagree
Disagree
Somewhat
disagree
neutral
Somewhat
agree
Agree Strongly agree

34. 2. The EtD is a structured process that is BETTER than
an informal/unstructured discussion
Strongly
disagree
Disagree
Somewhat
disagree
neutral
Somewhat
agree
Agree Strongly agree

35. 3. The EtD helped you UNDERSTAND the process of
getting a recommendation from a body of evidence
Strongly
disagree
Disagree
Somewhat
disagree
neutral
Somewhat
agree
Agree Strongly agree

36. 4. The EtD made you think about concepts like
patient values & preferences, costs & risks of the
intervention, equity, acceptance, and feasibility of the
intervention
Strongly
disagree
Disagree
Somewhat
disagree
neutral
Somewhat
agree
Agree Strongly agree

37. – Anonymous panel member
“It made the process smooth, easy to
understand and significantly reduced
the time it would have otherwise
taken to complete”

38. What are these
frameworks?
How can they help?
Why should we use
them?

39. Gracias
@CharlieNeck
cuelloca@mcmaster.ca