Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Meta analysis: Made Easy with Example from RevMan

The most reliable source of Evidence Based Medicine.

  • Login to see the comments

Meta analysis: Made Easy with Example from RevMan

  1. 1. META- ANALYSIS GAURAV KAMBOJ Junior Resident Deptt. of Community Medicine PGIMS, Rohtak
  2. 2. Contents •Introduction •Types of reviews •Definition •Functions of meta-analysis •Conducting Meta-analysis •Writing the research question and protocol •Comprehensive search •Selection of studies •Appraisal (quality assessment) of studies •Data abstraction •Data analysis
  3. 3. •Effect size •Presenting the findings – Forest plot •Heterogeneity •Dealing with heterogeneity •Fixed and random effects model •Meta-regression •Strengths and Weaknesses of meta-analysis •Software for meta-analysis Contents
  5. 5. Rationale for reviews Information explosion • More than 1,00,000 articles are published each year in more than 20,000 journals. • Humanly impossible to read through the articles published in any field. • Concise summaries of literature (Reviews) required, after separating insignificant and unsound from salient and crucial.
  6. 6. Literature reviewing - conceptual relations Systematic review Meta-analysis Narrative review
  7. 7. What is a traditional / narrative review? “ Review articles written by one or more experts based on a convenience sample of studies with no description of the underlying methodology” •Do not statistically combine results from multiple studies •Vote-counting
  8. 8. What is a systematic review? “A review that has been prepared using some kind of systematic approach to minimising biases and random errors, and that the components of the approach will be documented in a materials and methods section” - Chalmers et al • Has only qualitative or both qualitative and quantitative components • Quantitative component is meta-analysis
  9. 9. Meta-analysis “Quantitative and statistical approach for systematically combining results of previous research to arrive at conclusions about the body of research.” -Glass
  10. 10. •The term meta-analysis means ‘an analysis of analysis’. •A particular topic may have been replicated in various ways, using, for example, differently sized samples, and conducted in different countries under different environmental, social and economic conditions. •Sometimes results appear to be reasonably consistent; others less so. •Meta-analysis enables a rigorous comparison to be made rather than a subjective ‘eyeballing’. •If some crucial factors like sample size and methodology are missing then comparison is not feasible. What is meta-analysis?
  11. 11. The Great Debate •1952: Hans J. Eysenck concluded that there were no favorable effects of psychotherapy, starting a raging debate which 25 years of evaluation research and hundreds of studies failed to resolve •1978: To proved Eysenck wrong, Gene V. Glass statistically aggregated the findings of 375 psychotherapy outcome studies Glass (and colleague Smith) concluded that psychotherapy did indeed work. Glass called the method “Meta-analysis”
  12. 12. Functions of Meta-Analysis •Identifies heterogeneity in effects among multiple studies and, where appropriate, provide summary measure •Increases statistical power and precision to detect an effect •Develops ,refines, and tests hypothesis •Reduces the subjectivity of study comparisons by using systematic and explicit comparison procedure •Identifies data gap in the knowledge base and suggest direction for future research •Analyses if and how previous studies have modified knowledge on a certain topic
  13. 13. Forest plots of the meta-analysis addressing the use of antibiotic prophylaxis compared with no treatment in colon surgery
  14. 14. Conducting Meta-Analysis
  15. 15. Conducting a Meta-Analysis Writing the research question and a protocol Comprehensive search Selection of studies Appraisal (quality assessment) of studies Data abstraction Data analysis
  16. 16. Writing the research question and protocol •Research question: • P: the population of interest • I: the intervention or exposure • C: the comparison (in certain situations) • O: the outcome of interest •Protocol: specifying the – • Research question • Search methods • Inclusion and exclusion criteria for studies • Criteria for quality assessment (appraisal) of the studies • Methods of data abstraction and synthesis
  17. 17. Comprehensive Search Hand searching – ‘gold-standard’ for published studies • Computerized databases: • Pubmed/Medline (, • EMBASE, • ScienceDirect , ( • Scirus, ( ), • Cochrane Review- CENTRAL (Cochrane Central Register of Controlled Trials, m) • Trials Register- • Personal references, and emails • Web e.g. ISI Web of Knowledge (, Google Scholar( • Conference programs • Dissertations • Review articles • Government reports, bibliographies
  18. 18. Example: Research Issue Let's say we want to know whether streptokinase is protective for death from acute myocardial infarction. How should we set up a search strategy? We will search pubmed only
  19. 19. The Search • “streptokinase”[text word] OR “acute myocardial infarction” produces ALL articles that contain EITHER streptokinase OR acute myocardial infarction anywhere in the text – inclusive, many • streptokinase [text word] AND “acute myocardial infarction” [text word] will capture only those subsets that have BOTH streptokinase AND acute myocardial infarction anywhere in the text – restrictive, few Next, we shall look at the PUBMED Screen …
  20. 20. Choose your DATABASE here Remember to choose both PUBMED, and MESH for formulating search. Choose PUBMED CENTRAL for free articles!
  21. 21. Keep some, throw out others •Keep the ones with •high levels of evidence •good quality •check with QUOROM guidelines •Usually, MetaAnalysis done with RCTs •Case series, and case reports definitely out Selection problems are major problems
  22. 22. Selection of studies •Explicit Inclusion and exclusion criteria • Study designs: RCTs or CTs with a control group • Subjects: e.g. Females > 18 years of age • Publication types: Journal articles, dissertations, & masters theses • Languages: English • Interventions: e.g. Bone mineral density assessed at femur, spine, and/or radius • Time Frame: Studies published & indexed between January 1966 and December 1998
  23. 23. Factors Affecting Study Quality Non-randomized trials: •Treatment allocation related to prognosis or pre- judgment of appropriateness of treatment Randomized trials: •Inadequate randomization (e.g. alternating assignment) •Lack of stratification on important factors •Lack of or ineffective blinding All trials: •Patient drop-outs, patient switching arms •Missing data •Improper statistical analysis Appraisal of studies
  24. 24. Assessing Study Quality • Quality scores developed by - • Chalmers et al • Jadad et al • None is absolute best. • Little is known about their relative merits and their association with study outcomes. • When studies are excluded from a meta-analysis, reasons for exclusion should be provided for each excluded study • GIGO principle of ‘garbage in, garbage out’
  25. 25. How to score the quality of a study? •Example (scored yes=1, no=0): • Published in a peer-reviewed journal? • Experienced researchers? • Research funded by impartial agency? • Study performed by impartial researchers? • Subjects selected randomly from a population? • Subjects assigned randomly to treatments? • High proportion of subjects entered and/or finished the study? • Subjects blind to treatment? • Data gatherers blind to treatment? • Analysis performed blind? • Use the score to exclude some studies, and/or… • Include as a covariate in the meta-analysis
  26. 26. Checking for Bias •Reporting Bias is a group of related biases potentially leading to over- representation of significant or positive studies in systematic reviews •Studies with significant positive findings - • More likely to be published- Publication bias - over estimation of treatment effects • More likely to be published rapidly - Time lag bias • More likely to be published in English - Language bias • More likely to be cited by others - Citation bias
  27. 27. Identifying Publication Bias •Funnel Plot • Display the studies included in meta-analysis in a plot of effect size against sample size (or some other measure of the extent to which the findings could be affected by the play of chance). • Egger’s Regression Test: • Tests whether small studies tend to have larger effect sizes than would be expected (implying that small studies with small effect sizes have not been published). • Begg’s rank correlation test Both rarely used
  28. 28. Funnel Plot: what and how to read Plots the effect size against the sample size of the study To study a funnel plot, look at its LOWER LEFT corner, that’s where negative or null studies are located If EMPTY, this indicates “PUBLICATION BIAS” Note that here, the plot fits in a funnel, and that the left corner is not all that empty, but we cannot rule out publication bias
  29. 29. An Asymmetric Funnel Plot (indicative of publication bias) (Region of missing studies) Log Odds Ratio -2 -1 0 1 2 Asymmetric plot – •Publication bias •Clinical heterogeneity •Methodological heterogeneity
  30. 30. Meta-analysis in Presence of Publication Bias • Combine the results of larger studies only, which are less likely subject to publication bias. • File-drawer Method / Fail safe N: How many unpublished studies showing a null result are required to change a ‘significant’ meta analysis result to a ‘non-significant’ one? • ‘Trim and Fill’ method: • Tail of the side of funnel plot with smaller studies is chopped off to make the funnel plot symmetrical • Replicated and added back to both sides so the plot becomes symmetrical. • The centre and variability of the filled funnel plot are then estimated (there are complicated statistical methods to do this formally).
  31. 31. An Asymmetric Funnel Plot (indicating publication bias) Log Odds Ratio -2 -1 0 1 2 Trimmed Filled Estimated # missing studies : 5
  32. 32. Precautions •At least two reviewers •Sift and sift again • The first sift – pre-screening - is to decide which studies to retrieve in full. • The second sift – selection - is to look again at these studies and decide which are to be included in your review •Do not collect outcome data at the same time as eligibility information • wasted time and effort - if study is excluded later on • Results can sway decision •Look out for duplicate publications
  33. 33. How to Abstract Data: Guidelines Seven columns created trial: trial identity code trialnam: name of trial year: year of the study pop1: study population deaths1: deaths in study pop0: control population deaths0: deaths in control 22 studies to do meta analysis
  34. 34. 1. Choice of Metric Data Type Outcome Measures Continuous Mean Dichotomous (binary) (displayed in 2x2 table) Odds ratio (OR), Risk ratio (RR), Risk difference (RD) Data analysis
  35. 35. Comparison of OR, RR, and RD Failure Success Total New Treatment 5 95 100 Control 10 90 100 Odds Ratio = (5/95) / (10/90) = 0.48 Risk Ratio = (5/100) / (10/100) = 0.50 (Recall OR  RR when probability is small. OR is generally more extreme (further from 1) than RR.) Risk Difference = (5/100) - (10/100) = -0.05
  36. 36. Effect sizes • The effect size makes meta-analysis possible “ratio of the frequency of the events in the intervention to that in the control group.” • Any standardized index can be an “effect size” (e.g., standardized mean difference, correlation coefficient, odds- ratio) as long as it – • Is comparable across studies (generally requires standardization) • Represents the magnitude and direction of the relationship of interest • Is independent of sample size • Different meta-analyses may use different effect size indices • Studies are weighted according to the inverse of their variance.
  37. 37. gHedges  YExperimental  YControl ((NE 1)SD2 E  (NC 1)SD2 C )) / (NTot  2)  1 3 4(NE  NC )  9     Glass  YExperimental  YControl SDControl ES Calculation: Descriptive Statistics dCohen  YExperimental  YControl (SD2 E  SD2 C ) / 2
  38. 38. Zero Effect Size ES = 0.00 Control Group Intervention Group Overlapping Distributions
  39. 39. Moderate Effect Size Control Group Treatment Group ES = 0.40
  40. 40. Large Effect Size Control Group Intervention Condition ES = 0.85
  41. 41. Presenting the findings - Forest plots •The graphical display of results from individual studies on a common scale is a “Forest plot”. •Each study is represented by a black square and a horizontal line (CI:95%). •The area of the black square reflects the weight of the study / precision of the study (roughly the sample size). •A logarithmic scale should be used for plotting the Relative Risk / Odds Ratio. •Aggregate Effect size – displayed as a ‘diamond’.
  42. 42. Forest plot
  43. 43. Forest plot The impact of fish oil consumption on Cardio-vascular diseases LINE OF NO EFFECT i.e. no statistically significant difference between the study and control group
  44. 44. D'Souza, A. L et al. BMJ 2002;324:1361 Effect of probiotics on the risk of antibiotic associated diarrhoea The label tells what the comparison and outcome of interest are Scale measuring treatment effect. Take care when reading labels! Each study has an ID (author) Treatment effect sizes for each study (plus 95% CI)
  45. 45. Heterogeneity Reviews usually bring together studies that were performed: • By different people • In different settings • In different countries • On different people • In different ways • For different lengths of time • To look at different outcomes Types of heterogeneity •Clinical heterogeneity •Methodological heterogeneity •Statistical heterogeneity
  46. 46. Dealing with statistical heterogeneity •Test for existence of heterogeneity: have low power • Cochrane’s Q – statistic based on chi-square test • I2 statistic – scores heterogeneity between 0% and 100% • 25% - low heterogeneity • 50% - moderate • 75% - high • Presence or absence of heterogeneity influences the subsequent method of analysis: • Fixed- effects model • Random effect model • Meta-regression: to over come heterogeneity
  47. 47. Assessing heterogeneity from a forest plot
  48. 48. Fixed effects model • Conduct, if heterogeneity is absent • Assumes the size of treatment effect be same (fixed) across all studies & variation due to chance • Pooling: Mantel Haenszel OR • Weight = 1/variance = 1/SE2 • When heterogeneity exists we get: • a pooled estimate which may give too much weight to large studies, • A narrow confidence interval • a P-value which is too small. Random effects model • Conduct, if heterogeneity is present • Assumes the size of treatment effect does vary between studies • Der Simonian Laird method (DSL) for Odds’ Ratio • Weight = 1/variance = 1/(SE2+ inter-trial variance) • When heterogeneity exists we get: • a different pooled estimate with a different interpretation, • a wider confidence interval, • a larger P-value
  49. 49. Fixed effects model • When heterogeneity does not exists: • a pooled estimate which is correct, • a confidence interval which is correct, • a P-value which is correct. Random effects model • When heterogeneity does not exist: • a pooled estimate which is correct, • a confidence interval which is too wide, • a P-value which is too large No universally accepted method for choosing. A reasonable approach: 1. Decide whether the assumption of a fixed effects model is plausible. Could the studies all be estimating the same effect? If not, consider a random effects model. 2. If fixed effects assumption is plausible, are the data compatible? Graphical methods: forest plot, Galbraith plot. Analytical methods: heterogeneity test, I2 statistic. If assumption looks compatible with the data, use fixed effects, otherwise consider random effects.
  50. 50. Meta-regression • Allows researchers to explore which types of patient-specific factors or study design factors contribute to heterogeneity. • The estimate of study results is the dependent variable and one or more study-level variables are the independent variables (predictors) • Uses summary data from each trial, such as the average effect size, average disease severity at baseline, and average length of follow-up.
  51. 51. Quality Assessment of MA •PRISMA Statement (formerly QUOROM) : Preferred Reporting Items for Systematic Reviews and Meta- Analyses •MOOSE Statement : proposal for reporting meta analyses of observational studies in epidemiology
  52. 52. Strengths & Weaknesses
  53. 53. Strengths • Comprehensive search strategy: multiple sources of information • Explicit methodology: to ensure reproducibility and transparency • Emphasis on all clinically important outcomes: related to efficacy, safety, and tolerability of the interventions under consideration • Limiting errors: two reviewers at all major steps; limits bias and improves precision
  54. 54. Weaknesses  Good deal of effort  Qualitative distinctions between studies not captured  A good meta-analysis of badly designed studies will still result in bad statistics.  Selection bias  Tends to look at ‘broad questions’ that may not be immediately applicable to individual patients 65
  55. 55. Softwares •Huge Checklist [] •Free Software: •EpiMeta: from Epi Info •Revman: from Cochrane Collaboration •Non-free •Meta module in STATA
  56. 56. Further reading 1. Egger M, Smith GD, Altman DG (eds). Systematic Reviews in Health Care: Meta-analysis in context, 2nd edn. London: BMJ Publishing Group, 2001. 2. Petticrew M, Roberts H. Systematic Reviews in the Social Sciences: A practical guide. Oxford: Blackwell Publishing, 2006. 3. Deeks JJ, Higgins JPT, Altman DG (editors). Chapter 9: Analysing data and undertaking meta-analyses. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from
  57. 57. THANK YOU