The document provides an overview of the AGREE II tool for assessing clinical practice guidelines. It discusses the six domains of the AGREE II instrument which are used to evaluate guidelines. These domains contain a total of 23 items addressing scope and purpose, stakeholder involvement, rigor of development, clarity and presentation, applicability, and editorial independence. The document outlines how to use the AGREE II tool including scoring guidelines, considerations before assessment, and what each domain evaluates. It aims to guide users on properly applying the AGREE II instrument to appraise clinical practice guidelines in a reliable manner.

1. ‫ـيم‬‫ـ‬‫ـ‬‫ـ‬‫ـ‬‫ح‬‫الر‬‫ـن‬‫ـ‬‫ـ‬‫ـ‬‫ـ‬‫م‬‫الرح‬‫هللا‬ ‫ـم‬‫ـ‬‫ـ‬‫ـ‬‫ـ‬‫س‬‫ب‬
In the name of Allah. Most Gracious, Most Merciful

2. Instructor
Dr. Yasser Sami Amer
MS Pediatrics, MS Healthcare Informatics, CPHQ
Supervisor, EBPU, QMD, KSUMC
CPGs Advisor , KSUHs, AUHs
Member, G-I-N Adaptation & Implementation Working Groups

4.  EBM? – CPG? 

5. EBM ?! – CPGs?!

6. Check Your Handouts @ SOFT COPIES @:- ?????????????
I- PDF of [APPRAISAL OF GUIDELINES FOR RESEARCH & EVALUATION (AGREE) II Instrument] - full document (user's manual
+ instrument)
2- Excel file AGREE concordance calculator (electronic instrument) with THE COMMENT (JPEG file)
3- Link to the AGREE Enterprise website here (YOU are advised to create a free account before the training and browse the
website) [ http://www.agreetrust.org/ ] It contains the online AGREE instrument: My AGREE Plus
4- Samples of AGREE II appraisal reports (solved using the online tool/ my AGREE plus)
Individual appraisals (3)
(i) Lung cancer SIGN
(ii) IVF number of embryos to transfer ASRM
(iii) status epilepticus
Group appraisal (1)
(i) CVAD ( 2 files: scores and comments)
HARD COPIES
1- Learning objectives & training agenda
2-Pre-training questionnaire (SUBMIT PLEASE)
2- PDF for training AGREE (27 pages)
http://agree.machealth.ca/guidelines/PracTutorial.pdf
3- AGREE II Instrument (Paper form) (11 pages)
4- AGREE II Instrument–at-a-glance document (one page)
5- Evaluation Form (SUBMIT PLEASE)

7. Learning objectives
By the end of this session, participants should be able to know:-
PART 1: The Appraisal for Guidelines REsearch and Evaluation II (AGREE II)
Instrument: Assessment of the quality of any CPG
1. Brief overview on EBM, Research Methods and CPGs
2. What is the AGREE II Instrument?
 Domains
 questions/items
 scoring
3. What are the uses of the AGREE Instrument?
4. How to use the AGREE?
5. AGREE II Instrument tools:
 User’s manual @ & Instrument Paper form
 Excel concordance calculator @
 The online tool (My AGREE PLUS) @
PART 2: Working groups (hands on session)
Working groups of 3 – 5 appraisers (or more) and will use the printed paper AGREE II
Instrument to appraise the practice documents provided (Evidence-based Series #4-9:
sections 1 – 3). Also a demonstration for the Excel calculator tool and the online tool
(My AGREE Plus)

8. Uncertainty!.....Probability!
Sir William Osler (1849 – 1919)

9. The Epsom Derby, 1821 (oil on canvas) by Theodore Gericault;
Louvre, Paris, France
Importance of Research
People observe what they expect to observe, until shown otherwise

10. Importance of Research
People observe what they expect to observe, until shown otherwise
• http://www.youtube.com/watch?v=IEqccPhsq
gA&list=PLSC6ykdWzbjHk4fKrmB94yj11dqfkXF
0U
57 years later!
Sallie Gardner (horse) at a Gallop
Eadweard Muybridge in 1878 – 3 seconds silent film, USA

11. Importance of Research
People observe what they expect to observe, until shown otherwise
Sallie Gardner (horse) at a Gallop
Eadweard Muybridge in 1878 – 3 seconds silent film, USA

12. 8/23/2014
CPGs Adaptation & Implementation in
KSUMC: KAUH 2014
12

13. EBM is all about Clinical Decision
Making

14. Level
Evidence
Grade
Recomm.
Strength of
EVIDENCE PYRAMID

15. CPG-ICU-002 (VTE) 2013
8/23/2014
CPGs Adaptation & Implementation in
KSUMC: KAUH 2014
15

16. HWCPG-ENT-001 (ABRS) 2013
8/23/2014
CPGs Adaptation & Implementation in
KSUMC: KAUH 2014
16

17. HWCPG-PHARM-001 (Vanco)
8/23/2014
CPGs Adaptation & Implementation in
KSUMC: KAUH 2014
17

18. EVIDENCE PYRAMID ?
18

19. Evidence Pyramid
SR
RCT
Cohort
Case control
Case series
Case report
Expert opinion
I
II
III
IV
A
B
C
LevelsofEvidence
GradesofRecommendations
MA

21. Strength of
EVIDENCE

22. Research
Descriptive
Single gp, no control gp
Case report
Case series
Comparative
> One gp, control gp.
Observational
(OBSERVE)
Cohort
Case-control
Cross-
sectional
Experimental
(INTERVENE)
RCT
Non-RCT

23. Types of study designs
Comparative !
> One gp, control gp.
TIME

24. Confounder
(other factors)
(another exposure that is a risk
factor for the disease)
OUTCOME
(Effect = Disease)
EXPOSURE
(Cause)

25. Basic concepts of research designs

26. P value < 0.05 AND 95% CI

27. Types of Clinical Research Studies
Clinical Research
Studies
Observational
We OBSERVE
Interventional =
Experimental
(Clinical Trials)
We INTERVENE

28. Observational studies
“….investigators OBSERVE what happens to participants…..!”
“…….investigators assess health outcomes in groups
of participants according to a protocol or research
plan. Participants may receive interventions, which
can include medical products, such as drugs or
devices, or procedures as part of their routine
medical care, but participants are not assigned to
specific interventions by the investigator (as in a
clinical trial).
For example, investigators may observe a group of older adults
to learn more about the effects of different lifestyles on cardiac
health.”
- http://clinicaltrials.gov/ct2/about-studies/learn

29. Observational studies:
another definition
Measures characteristics of the subject
without attempting
to manipulate/influence the subjects.
CAN NOT prove causation, only
association.
-STAT INTRO

30. Researchers measure Exposures &
Outcomes but do not measure any
particular ‘intervention’ as part of the
study
Observational
studies
Prospective
(forward in time)
Retrospective
(backward in time)

31. Observational studies: types
• Cohort
• Case-control
• Cross-sectional

32. Cohort study
An observational study in which a defined group of
people (the cohort) is followed over time.
The outcomes of people in subsets of this cohort are
compared, to examine people who were exposed or
not exposed (or exposed at different levels) to a
particular intervention or other factor of interest.
A prospective cohort study assembles participants and follows
them into the future. A retrospective (or historical) cohort study
identifies subjects from past records and follows them from the
time of those records to the present. Because subjects are not
allocated by the investigator to different interventions or other
exposures, adjusted analysis is usually required to minimize the
influence of other factors (confounders)
http://www.cochrane.org/glossary

33. Interventional studies (clinical trials)
• Participants receive specific interventions according to the
research plan or protocol created by the investigators. These
interventions may be medical products (e.g. drugs or devices);
procedures; or changes to participants' behavior, for example,
diet.
• Compare (i) a new medical approach to a standard one that is
already available or to a placebo that contains no active
ingredients or to no intervention. (ii) compare interventions that
are already available to each other.
• The investigators try to determine the safety and efficacy of the
intervention by measuring certain outcomes in the participants.
• For example, investigators may give a drug or treatment to participants who
have high blood pressure to see whether their blood pressure decreases.

34. Experimental studies: another
definition
Attempt to manipulate or influence
the subjects in order to obtain data.
Properly designed experiments
can be used to prove causation.
-STAT INTRO

35. Clinical trial (Interventional
study)
An experiment to compare the effects of two or
more healthcare interventions. Clinical trial is an
umbrella term for a variety of designs of
healthcare trials, including:
1. Uncontrolled trials
2. Controlled trials
3. Randomised controlled trials

36. Randomized controlled/ clinical trial
(RCT)
• An experiment in which 2 or more
interventions, possibly including a control
intervention or no intervention, are
compared by being randomly allocated
to participants.
• In most trials one intervention is assigned to each individual
but sometimes assignment is to defined groups of individuals
(for example, in a household) or interventions are assigned
within individuals (for example, in different orders or to
different parts of the body)

37. A review of a clearly formulated question
that uses systematic and explicit methods
to identify, select, and critically appraise
relevant research, and to collect and
analyse data from the studies that are
included in the review.
Statistical methods (meta-analysis) may or may not be
used to analyse and summarise the results of the
included studies.
Systematic review
(synonym: systematic overview)

38. Meta-analysis
The use of statistical techniques in
a systematic review to integrate the results
of included studies.
Sometimes misused as a synonym for
systematic reviews, where the review
includes a meta-analysis.

40. Cochrane Review
• Systematic summaries of evidence of the effects
of healthcare interventions, intended to help
people make practical decisions.
• For a review to be called a ‘Cochrane Review’ it
must be in CDSR or CMR.
• Prepared using Review Manager
(RevMan) software provided by the
Collaboration, and adhere to a structured format
described in the Cochrane Handbook for
Systematic Reviews of Interventions.
CDSR= Cochrane Database of SRs, CMR=Cochrane Methodology Register

41. EBM: Clinical questions
(PICO or PICOT)

42. PICOT

43. Best study design to answer
clinical questions

44. PICO example: formulation of
searchable clinical questions

45. A 28-year-old male presents with recurrent
furunculosis (skin boils) for past 8 months;
these episodes have been treated with
drainage and several courses of antibiotics
but keep recurring. He asks if recurrences
can be prevented.
P Population/patient patients with recurrent
furunculosis
I Intervention/indicator prophylactic antibiotics
C Comparator/control no treatment
O Outcome reduction in recurrence
rate of furunculosi

46. A physician thought he heard that after a
myocardial infarction (MI), aspirin may not
improve mortality. He asks the clinical
pharmacist, while on rounds, “Does daily
aspirin really improve mortality, or is there
evidence to the contrary?”
Prognosis
P = postmyocardial infarction patient; I = an
aspirin a day; C = no aspirin; O = improve
mortality
Cohort studies first.

47. CPGs

48. Definition: (old)
“Systematically developed statements to assist
practitioner and patient decisions about
appropriate health care for specific clinical
circumstances” (IOM 1990).
Increasing international interest in the
development and implementation of CPGs.
Clinical Practice Guidelines (CPGs)
48

49. CPGs New Definition
CPGs are “Statements that include
Recommendations intended to optimize
patient care that are informed by a
Systematic Review of evidence and an
assessment of the benefits and harms of
alternative care options”.
Committee on Standards for Developing Trustworthy CPGs
(IOM-AHRQ 2011)
EVIDENCE-BASED CPGs

50. Standards for high quality CPGs
IOM 2011 – G-I-N 2012

51. IOM 2011: 8 Standards for
Developing Trustworthy CPGs
1. Establishing transparency.
2. Management of conflict of interest (COI).
3. GDG composition.
4. CPG – SR intersection.
5. Establishing [E] foundations for and rating strength of
[Rs]. (i.e. E – Rs link)
6. Articulation of [Rs].
7. External review.
8. Updating.

52. Date of download: 8/19/2014
From: Guidelines International Network: Toward
International Standards for Clinical Practice Guidelines
Ann Intern Med. 2012;156(7):525-531. doi:10.7326/0003-4819-156-7-201204030-00009
Copyright © American College of Physicians. All rights reserved.

53. 1
2
3
4
5
6
7
8
9
10
11

55. 8/23/2014 55

57. Purpose
To guide on how to use and apply the
AGREE II for assessing CPGS
Items
Domains
User’s Manual
Website

58. Outlines
• Overview of AGREE II including the items and
domains.
• Present the scoring method of the CPG.
• How to use and apply the AGREE II for assessing
CPG.
• Share the overall scoring.
• The AGREE Enterprise Website

59. What is the AGREE II Tool ?
• Quality (Methodological rigor & transparency; confident in
resulting Recommendations)
59 Using the AGREE II Instrument
Assess
Guide
•CPGs Development
•CPGs Reporting
History
1st AGREE was published in 2003,then refined in 2009 AGREE II
(New scoring “7 point scale” – Items modifications – New
user’s manual) – 11 YEARS !
Can be applied to any
CPG in any Disease
area !

60. • Healthcare providers/ clinicians
• CPG developers/ adapters
• Policy makers
• Educators
Who can use the AGREE II ?

61. Considerations before a CPG
Assessment
• increase the reliability of the assessment
2 – 4 Appraisers
• in full and obtain all related information and
needed documents before undertaking the
AGREE II assessment ( to make a well
informed assessment)
Read CPG first

62. Rating Scale
• All AGREE II Items are rated on the following 7-point scale
Score Meaning
7 (Strongly
Agree)
= If the quality of reporting is exceptional and full criteria
and considerations in User’s manual are met.
1 (Strongly
Disagree)
= No information relevant to AGREE II item OR the concept
is very poorly reported
2 – 6 = when the reporting of the item does not meet the full
criteria or considerations, depending on the completeness
& quality of reporting .

63. The AGREE II Includes….
•Core Items
•(6 Domains)
23
•Overall
Assessment Items2

64. 23 Items in 6 Domains
USER’S MANUAL page 7
DOMAINS No. of Items
1 Scope & Purpose 3
2 Stakeholder Involvement 3
3 Rigour of Development 8
4 Clarity & Presentation 4
5 Applicability 3
6 Editorial Independence 2

65. Items and Domains: A Closer Look
AGREE II:
USER’S MANUAL
Per each Domain (guidance for rating the 23 items)
Pages 11 – 41
User’s Manual Description:
Where to Look:
How to Rate:
Item content includes the following CRITERIA:
Additional CONSIDERATIONS:

66. Domain 1

67. DOMAIN 1. SCOPE AND PURPOSE
1. Objective(s):
Health impact & benefits of a CPG
on target population
Introduction, scope, purpose,
rationale, background & objectives

68. Examples of CPG Objective(s)

69. SIGN Asthma CPG 2012

70. 2. Health Question(s):-
Questions, Scope, Purpose, Rationale
and Background
3. Target population:-
Pt. population, target population,
relevant pt.s, scope and purpose
DOMAIN 1. SCOPE AND PURPOSE

71. Health/ Clinical/ Key Questions
Patient (& disease characteristics)
Intervention(s)
Professionals (Target users)
Outcomes (purpose of the CPGs)
Healthcare settings (& context)
19th March 2013 EBCPGs: Dr. Yasser Sami Amer 71
CPG Scope: PIPOH Model

72. Health/ clinical Questions (PIPOH) Sample
P: Patient/target population:
Pediatric (age 1–18 years) and Adult (age 19 years or older) patients
planned for undergoing surgical procedures. Note: These CPGs do
not specifically address newborn (premature and full-term) infants.
While the CPGs do not address all concerns for patients with renal or
hepatic dysfunction, antimicrobial prophylaxis often does not need
to be modified for these patients when given as a single
preoperative dose before surgical incision.
I: Intervention and practices considered and CPG
Category:
Assessment of Therapeutic Effectiveness and Prevention
Primary antimicrobial prophylaxis (i.e., prevention of an initial
infection) for surgical procedures, including antibiotic choice, dose,
and dosage regimen.
Antimicrobial Prophylaxis in Surgery

73. P: Professionals and Intended Users (target
users/ stakeholders) and Clinical Specialty:
Physicians, Nurses, Allied Health Personnel and
Clinical Pharmacists in the departments of
Surgery and all Surgical subspecialties (Colon
and Rectal Surgery, Gastroenterology, Plastic
Surgery, Urology, Thoracic Surgery, Vascular
Surgery and Neurological Surgery), Obstetrics
and Gynecology, Orthopedic Surgery,
Ophthalmology and Pharmacology.

74. O: Major Outcomes Considered
1. Postoperative infection rates
2. Postoperative morbidity and mortality rates
3. Duration and cost of health care
4. Adverse effects
H: Healthcare settings
King Saud University Hospitals (KKUH/ KAUH) –
Tertiary/ Governmental/ University Hospital –
Departments and clinics (mentioned in the
clinical specialty)

75. SIGN Asthma CPG 2012 supporting material

76. Domain 2

77. 4. Guideline group
Methods, guideline panel list, acknowledgements,
& appendices
5. Patient preference
Scope, methods, guideline panel list, external
review &target population perspectives
6. Target users
Target user & intended user
DOMAIN 2. STAKEHOLDER
INVOLVEMENT

78. Domain 3

79. 7. Systematic methods for E
search
8. Selection Criteria of E
9. Strengths & Limitations of E
10. Methods of Rs
11. Benefits , side effect and risks
in Rs
12. Evidence Links (Gs of Rs –
LoE)
13. External Review
14. Update Procedure
DOMAIN 3. RIGOUR OF DEVELOPMENT
Methods, literature search
strategy & appendices
+ inclusion/ exclusion criteria
+ Evidence tables, clinical
evidence, evidence description
(results), evidence
interpretation (discussion)
Methods, CPG Development
process
 same sections
+ Rs, Key Evidence
+ acknowledgements
+ CPG update, date of CPG

80. Methodology of development
• In same CPG document (in brief or
detailed)
• In a separate document or
supplement (usu. Detailed)
• Not documented

81. Within CPG document (ENT IDSA ABRS CPG)

82. Mentioned but not documented

83. KSUMC CPGs: Adaptation Process Methodology

84. Separate Methodology document
(SIGN 50: A guideline developer’s handbook -2011)

85. NICE: The guideline manual (2012)

86. Domain 4

87. DOMAIN 4. CLARITY OF PRESENTATION
15. Rs are specific
Rs & executive summary
16. Options for management
+ discussion, Treatment (options/alternatives)
17. Recommendation identifiable
Key Rs; separate (e.g. QRG)

88. Domain 5

89. DOMAIN 5. APPLICABILITY
18. Facilitators & barriers
CPG dissemination/ implementation, barriers,
CPG utilization & Quality indicators
19. Tools
+ tools, resources, appendices
20. Resource implications
Method, cost utility, cost effectiveness,
acquisition costs & implications for budgets
21. Monitoring /audit criteria
Rs, Quality indicators & audit criteria

90. Domain 6

91. DOMAIN 6. EDITORIAL INDEPENDENCE
22. Funding body
Disclaimer & funding source
23. Competing interests
Methods, Conflicts of interest (COI),
CPG panel & appendices

92. Overall Assessment

93. OVERALL CPG ASSESSMENT
1. Rate the overall quality of this CPG
2. I would recommend this CPG for use
3. Notes
YES
Yes ,with modification
No

95. Selected
CPG
Domains
Overall
Is CPG
Recommended
for use ?1 2 3 4 5 6
CPG 1
CPG 2
CPG 3
CPG 4
CPG 5
Finally AGREE II Domain Scores Color Coding (Prof. Lubna Al-Ansary)
< 40 % Red >41-70 Yellow >71 Green

96. AGREE
Domains
Scores Table
CPG 1:
SIGN
CPG 2:
EPR3
CPG 3:
GINA
CPG 4: ICSI CPG 5:
Singapore
MOH
Domain 1
60 % 74 % 45 % 61 % 42 %
Domain 2
55 % 56 % 60 % 58 % 63 %
Domain 3 92 % 83 % 79 % 38 % 43 %
Domain 4
95 % 90 % 92 % 75 % 87 %
Domain 5
70 % 82 % 80 % 33 % 58 %
Domain 6
80 % 60 % 22 % 50 % 10 %
Overall
assessment 7 6 5 3 4
Recommende
d for use Yes Yes (w M) Yes (w M) No No

97. Graphical Representation

98. AGREE II DOMAINS
(ENT-ABRS CPG)
IDSA CPG 2012 Canadian CPG 2011
D1: Scope & Purpose
86 % 80 %
D2: Stakeholder Involvement
86 % 69 %
D3: Rigour of Development
90 % 85 %
D4: Clarity & Presentation
92 % 90 %
D5: Applicability
67 % 64 %
D6: Editorial Independence
96 % 76 %
Overall assessment 7 5
Recommended for use
Yes No
This table uses the AGREE II Domain Score Colour Coding proposed by Dr. Lubna Alansary
(< 40% red - > 41 – 70% yellow - > 71 % green)

99. AGREE II DOMAINS CPG1
(Singapore)
CPG2
(WFSBP)
CPG3
(Canada)
D1: Scope & Purpose
90.7% 90.7% 96.3 %
D2: Stakeholder Involvement 72.2% 65 % 81.5 %
D3: Rigour of Development
39. 5% 84.5 % 90.3 %
D4: Clarity & Presentation
98.1% 88.9 % 87.03 %
D5: Applicability
34.7% 23.6% 40.3 %
D6: Editorial Independence
ZERO % 86.1% 55.6 %
Overall assessment
4 5 6
Recommended for use NO NO YES
This table uses the AGREE II Domain score colour coding proposed by Dr Lubna Al-Ansary
(< 40% Red - >41-70% Yellow - >71% Green)

100. The paper tool (handout!)

101. Using the AGREE II Instrument 101

102. AGREE II rater concordance
calculator (EXCEL)

103. AGREE II rater concordance calculator
(McMaster University)

107. AGREE Enterprise website
http://www.agreetrust.org/

108. The online tool
“My AGREE PLUS”

109. My AGREE PLUS
(OR Our AGREE!)

110. My AGREE II Report PDF

112. A tool for evaluation of clinical validity/
credibility of CPGs (in progress)

113. AGREE-REX: Recommendation EXcellence:
Innovations to enhance the capacity of practice
guidelines to improve health and health care systems
Develop a useful, reliable, and valid
knowledge resource to complement the
AGREE II, which will guide the
development, reporting, and evaluation of
the clinical credibility of CPG
recommendations.

114. Acknowledgments

115. Questions?

116. EVALUATION !

117. Dr. Yasser Sami Amer
MSc Pediatrics, MSc HC Informatics, CPHQ
EBCPGs Advisor & Trainer
yasser3amer@yahoo.com

118. AGREE II Workshop: Group photo: August 20th, 2014
King Khalid University Hospital,
King Saud University, Riyadh, Saudi Arabia