SlideShare a Scribd company logo
1 of 21
Download to read offline
METHODS FOR
DEVELOPING
ASSESSMENT
INSTRUMENTS TO
GENERATE USEFUL
DATA IN THE PRESENCE
OF VAGUE COURSE
OBJECTIVES
                       Patrick B. Barlow
                       Tiffany L. Smith
                       Eric Heidel, PhD
                       William Metheny, PhD
On the Agenda
SETTING THE SCENE
• Who? What? Where? When?

WHY?
• Assessment in Graduate Medical Education

FIVE PRACTICAL TIPS
• How we addressed the problem

USE OF ASSESSMENT RESULTS
• How were these methods useful?

DISCUSSION
• Questions?
Who? What? Where? When?

SETTING THE SCENE
Office of Medical
Education, Research, and Development
(OMERAD)
 •   Job Description
      •   Consultation and education
 • What was happening with GME at
   our institution?
 • New Office Structure
      •   PhD students in ESM brought in
      •   Office given the reigns of the clinical
          research skills curricula
Assessment in Graduate Medical Education

WHY DOES THIS NEED TO BE
ADDRESSED?
What is EBM?


                      Clinical
                   Epidemiology
  Biostatistics



              Critical
             Appraisal




Evidence-Based Medicine in GME1,2
What We Know About Resident
Knowledge of Clinical Research Skills?
  • Error rate in reporting and
    interpreting statistics in medicine
                                   3
    is estimated between 30-90%
  • Consistent…

      • Lack of knowledge
                           4,5
      • Lack of confidence




                          An example…
How we addressed the problem

FIVE PRACTICAL TIPS
TIP ONE
Know Your Situation
     •   Learning environment factors
          •   Statistics and research methods
              as a topic
          •   No formal “courses”, nothing is
              “required”
          •   No previous learning
              objectives, syllabus, or
              assessment structure
     •   Work environment factors
          •   Hospital obligations
          •   Attending physician buy-in &
              priorities
TIP ONE
Know Your Situation

     • Population-specific   factors
       • Variable background
         experience
       • Low average competence
         and confidence
       • Realities of being a
         physician
     • Availability   of resources
       • Limited time
       • Limited money
TIP TWO
Clarify Your Purpose
      •   Ask two questions:
           • How will the assessment
             audience benefit from the
             results?
           • How will the students benefit
             from the assessment results?
      •   In our case
           • Audience (OMERAD, GSM
             faculty/administration)
           • Students
             (Residents, fellows, physicians
             , & staff)
TIP THREE
Use What You Have
     •   Gather the Necessary
         Background Data
          •   Existing content
          •   Faculty interviews
          •   Direct observation
          •   Literature
          •   Clinical/Work experience
     •   Three benefits
          •   What instructors think the
              students are learning
          •   What is being taught
          •   Where the gaps are in the
              curriculum
TIP FOUR
Fit the Instrument to Your Purpose, Not
          the Other Way Around
                 • Again, consider situational
                   factors
                 • Resources for types of
                   assessment instruments
                 • What worked for us
                    •   Background knowledge
                             6
                        probe
TIP FIVE
 Get Consistent and Critical Feedback
                               Assessment must be
                               viewed as a never-
            Develop/
             Modify            ending, iterative process
                               • An instrument is
                                 developed or modified
                               • The instrument is
           “Feedback             tested
            Loop” of
           Assessment          • Testing generates
            Practice             feedback
                               • Feedback leads to
Feedback                Test
                                 modifications…
TIP FIVE
 Get Consistent and Critical Feedback
                               Assessment must be
                               viewed as a never-
            Develop/
             Modify            ending, iterative process
                               • An instrument is
                                  developed or modified
                               • The instrument is
           “Feedback              tested
            Loop” of
           Assessment          • Testing generates
            Practice              feedback
                               • Feedback leads to
Feedback                Test
                                  modifications…
                               • These modifications
                                 are tested
An Integrated Assessment Model for a Dynamic Learning
Environment

USE OF RESULTS
Improvements to the Course:
   Learning Objective Development
                                • Multiple sources
                                  of data
                                   • Assessment
                   Concrete        • Experiential
“List of Topics”   Learning        • Evaluation
                   Objectives   • Data identified
                                   • Salient topics
                                   • Missing content
                                   • Student needs
                                • Need for a
 BEFORE            AFTER          responsive
                                  curriculum
Improvements to the Assessment:
 Test blueprint process used to improve the
          assessment instrument
Module 2:Comparing Research Designs
                                                    1.   Start with course
                                                         learning objectives
                                                    2.   Identify test “topics”
    Identify major                                       from learning
    epidemiologic                                        objectives
       research       “after this                   3.   Expand each topic
        designs
                       module                            to as many
                     participants                        “concepts” as
                      should be Apply each               possible
                      able to…” design to their     4.   Collapse list of
                                      own area of
                                       research          concepts to remove
                                                         redundancy
                                                    5.   Create/modify items
Next Steps


• Continue instrument & curriculum
  revisions
• Standardized assessment for
  residents, fellows, physicians on
  clinical research skills and
  statistics.
Questions? Comments?

DISCUSSION
References
1.   Green, M. L. (2000). Evidence-based medicine training in graduate
     medical education: past, present and future. Journal of evaluation in
     clinical practice, 6(2), 121–38. Retrieved from
     http://www.ncbi.nlm.nih.gov/pubmed/10970006
2.   Stewart, M.G. (2001). ACGME Core Compentencies Accreditation
     Council for Graduate Medical Education. Retrieved from
     http://www.acgme.org/acWebsite/RRC_280/280_coreComp.asp
3.   Novack, L., Jotkowitz, A., Knyazer, B., & Novack, V. (2006). Evidence-
     based medicine: assessment of knowledge of basic epidemiological and
     research methods among medical doctors. Postgraduate Medical
     Journal, 82(974), 817–822. Retrieved from
     http://pmj.bmj.com/content/82/974/817.abstract
4.   West, C. P., & Ficalora, R. D. (2007). Clinician Attitudes Toward
     Biostatistics. Mayo Clinic Proceedings, 82(8), 939–943. Retrieved from
     http://www.mayoclinicproceedings.com/content/82/8/939.abstract
5.   Windish, D. M., Huot, S. J., & Green, M. L. (2007). Medicine Residents’
     Understanding of the Biostatistics and Results in the Medical Literature.
     JAMA: The Journal of the American Medical Association, 298(9), 1010–
     1022. Retrieved from http://jama.ama-
     assn.org/content/298/9/1010.abstract
6.   Angelo, T. & Cross, K.P. (1993). Classroom Assessment Techniques. San
     Francisco: Jossey-Bass.
7.   Fink, L.D. (2003). Creating Significant Learning Experiences: An
     integrated approach to designing college courses. San Francisco, CA:
     John Wiley & Sons, Inc.

More Related Content

What's hot

Identifying and formulating a research question: Ayurveda Perspective
Identifying and formulating a research question: Ayurveda Perspective Identifying and formulating a research question: Ayurveda Perspective
Identifying and formulating a research question: Ayurveda Perspective Kishor Patwardhan
 
Credibility, validity, reliability and transferability
Credibility, validity, reliability and transferabilityCredibility, validity, reliability and transferability
Credibility, validity, reliability and transferabilityJohn Marsden
 
Notes from frankel and wallen
Notes from frankel  and wallenNotes from frankel  and wallen
Notes from frankel and wallenRose Jedin
 
Research Method for Business chapter 8
Research Method for Business chapter  8Research Method for Business chapter  8
Research Method for Business chapter 8Mazhar Poohlah
 
Research Methods: Basic Concepts and Methods
Research Methods: Basic Concepts and MethodsResearch Methods: Basic Concepts and Methods
Research Methods: Basic Concepts and MethodsAhmed-Refat Refat
 
Research methods - PSYA1 psychology AS
Research methods - PSYA1 psychology ASResearch methods - PSYA1 psychology AS
Research methods - PSYA1 psychology ASNicky Burt
 
INTRODUCTION TO DATA COLLECTION INSTRUMENTS (DCIs)
INTRODUCTION TO DATA COLLECTION INSTRUMENTS (DCIs)INTRODUCTION TO DATA COLLECTION INSTRUMENTS (DCIs)
INTRODUCTION TO DATA COLLECTION INSTRUMENTS (DCIs)Auver2012
 
Lesson 21 designing the questionaire and establishing validity and reliabilty
Lesson 21 designing the questionaire and establishing validity and reliabiltyLesson 21 designing the questionaire and establishing validity and reliabilty
Lesson 21 designing the questionaire and establishing validity and reliabiltymjlobetos
 
Qualitative vs Quantitative
Qualitative vs QuantitativeQualitative vs Quantitative
Qualitative vs Quantitativeeilire91
 
Action research by Prof.. vinita shrouty
Action research  by Prof.. vinita shroutyAction research  by Prof.. vinita shrouty
Action research by Prof.. vinita shroutyvinita katpatal
 

What's hot (20)

Modesofinquiry
ModesofinquiryModesofinquiry
Modesofinquiry
 
Identifying and formulating a research question: Ayurveda Perspective
Identifying and formulating a research question: Ayurveda Perspective Identifying and formulating a research question: Ayurveda Perspective
Identifying and formulating a research question: Ayurveda Perspective
 
Credibility, validity, reliability and transferability
Credibility, validity, reliability and transferabilityCredibility, validity, reliability and transferability
Credibility, validity, reliability and transferability
 
Notes from frankel and wallen
Notes from frankel  and wallenNotes from frankel  and wallen
Notes from frankel and wallen
 
Research Method for Business chapter 8
Research Method for Business chapter  8Research Method for Business chapter  8
Research Method for Business chapter 8
 
Research process and sampling
Research process and samplingResearch process and sampling
Research process and sampling
 
Research Methods: Basic Concepts and Methods
Research Methods: Basic Concepts and MethodsResearch Methods: Basic Concepts and Methods
Research Methods: Basic Concepts and Methods
 
Research methods - PSYA1 psychology AS
Research methods - PSYA1 psychology ASResearch methods - PSYA1 psychology AS
Research methods - PSYA1 psychology AS
 
Survey design report
Survey design reportSurvey design report
Survey design report
 
INTRODUCTION TO DATA COLLECTION INSTRUMENTS (DCIs)
INTRODUCTION TO DATA COLLECTION INSTRUMENTS (DCIs)INTRODUCTION TO DATA COLLECTION INSTRUMENTS (DCIs)
INTRODUCTION TO DATA COLLECTION INSTRUMENTS (DCIs)
 
Research process
Research processResearch process
Research process
 
Tools of Education Research- Dr. K. Thiyagu
Tools of Education Research- Dr. K. ThiyaguTools of Education Research- Dr. K. Thiyagu
Tools of Education Research- Dr. K. Thiyagu
 
Research methodology methods and techniques 2004
Research methodology   methods and techniques 2004Research methodology   methods and techniques 2004
Research methodology methods and techniques 2004
 
Methods of collecting data
Methods of collecting dataMethods of collecting data
Methods of collecting data
 
Research tools and techniques
Research tools and techniquesResearch tools and techniques
Research tools and techniques
 
Qualitative data analysis
Qualitative data analysisQualitative data analysis
Qualitative data analysis
 
Chapter10
Chapter10Chapter10
Chapter10
 
Lesson 21 designing the questionaire and establishing validity and reliabilty
Lesson 21 designing the questionaire and establishing validity and reliabiltyLesson 21 designing the questionaire and establishing validity and reliabilty
Lesson 21 designing the questionaire and establishing validity and reliabilty
 
Qualitative vs Quantitative
Qualitative vs QuantitativeQualitative vs Quantitative
Qualitative vs Quantitative
 
Action research by Prof.. vinita shrouty
Action research  by Prof.. vinita shroutyAction research  by Prof.. vinita shrouty
Action research by Prof.. vinita shrouty
 

Viewers also liked

New Benchmark 500 Uploads!
New Benchmark 500 Uploads!New Benchmark 500 Uploads!
New Benchmark 500 Uploads!Pat Barlow
 
The Development of the Biostatistics & Clinical Epideimiolgy Skills (BACES) A...
The Development of the Biostatistics & Clinical Epideimiolgy Skills (BACES) A...The Development of the Biostatistics & Clinical Epideimiolgy Skills (BACES) A...
The Development of the Biostatistics & Clinical Epideimiolgy Skills (BACES) A...Pat Barlow
 
Application of assessment and evaluation data to improve a dynamic graduate m...
Application of assessment and evaluation data to improve a dynamic graduate m...Application of assessment and evaluation data to improve a dynamic graduate m...
Application of assessment and evaluation data to improve a dynamic graduate m...Pat Barlow
 
Bad Statistics, Bad Reporting, Bad Impact on Patients: The Story of the PACE ...
Bad Statistics, Bad Reporting, Bad Impact on Patients: The Story of the PACE ...Bad Statistics, Bad Reporting, Bad Impact on Patients: The Story of the PACE ...
Bad Statistics, Bad Reporting, Bad Impact on Patients: The Story of the PACE ...Julie Rehmeyer
 
Brief Look at Association vs causation
Brief Look at Association vs causationBrief Look at Association vs causation
Brief Look at Association vs causationPat Barlow
 
Comparing research designs fw 2013 handout version
Comparing research designs fw 2013 handout versionComparing research designs fw 2013 handout version
Comparing research designs fw 2013 handout versionPat Barlow
 
Lecture of epidemiology
Lecture of epidemiologyLecture of epidemiology
Lecture of epidemiologyAmany El-seoud
 
Learning by doing aalhe presentation handout
Learning by doing aalhe presentation handoutLearning by doing aalhe presentation handout
Learning by doing aalhe presentation handoutPat Barlow
 
Commonly used Statistics in Medical Research Handout
Commonly used Statistics in Medical Research HandoutCommonly used Statistics in Medical Research Handout
Commonly used Statistics in Medical Research HandoutPat Barlow
 
Common measures of association in medical research handout
Common measures of association in medical research handoutCommon measures of association in medical research handout
Common measures of association in medical research handoutPat Barlow
 

Viewers also liked (20)

Life style
Life styleLife style
Life style
 
New Benchmark 500 Uploads!
New Benchmark 500 Uploads!New Benchmark 500 Uploads!
New Benchmark 500 Uploads!
 
Quality
QualityQuality
Quality
 
The Development of the Biostatistics & Clinical Epideimiolgy Skills (BACES) A...
The Development of the Biostatistics & Clinical Epideimiolgy Skills (BACES) A...The Development of the Biostatistics & Clinical Epideimiolgy Skills (BACES) A...
The Development of the Biostatistics & Clinical Epideimiolgy Skills (BACES) A...
 
Application of assessment and evaluation data to improve a dynamic graduate m...
Application of assessment and evaluation data to improve a dynamic graduate m...Application of assessment and evaluation data to improve a dynamic graduate m...
Application of assessment and evaluation data to improve a dynamic graduate m...
 
Social marketing
Social marketingSocial marketing
Social marketing
 
Auditing1
Auditing1Auditing1
Auditing1
 
Bad Statistics, Bad Reporting, Bad Impact on Patients: The Story of the PACE ...
Bad Statistics, Bad Reporting, Bad Impact on Patients: The Story of the PACE ...Bad Statistics, Bad Reporting, Bad Impact on Patients: The Story of the PACE ...
Bad Statistics, Bad Reporting, Bad Impact on Patients: The Story of the PACE ...
 
Complex sampling design & analysis
Complex sampling design & analysisComplex sampling design & analysis
Complex sampling design & analysis
 
Research Made Easy
Research Made EasyResearch Made Easy
Research Made Easy
 
Medical statistics2
Medical statistics2Medical statistics2
Medical statistics2
 
Regression analysis
Regression analysisRegression analysis
Regression analysis
 
Brief Look at Association vs causation
Brief Look at Association vs causationBrief Look at Association vs causation
Brief Look at Association vs causation
 
Sociology
SociologySociology
Sociology
 
Comparing research designs fw 2013 handout version
Comparing research designs fw 2013 handout versionComparing research designs fw 2013 handout version
Comparing research designs fw 2013 handout version
 
Data management & statistics in clinical trials
Data management & statistics in clinical trialsData management & statistics in clinical trials
Data management & statistics in clinical trials
 
Lecture of epidemiology
Lecture of epidemiologyLecture of epidemiology
Lecture of epidemiology
 
Learning by doing aalhe presentation handout
Learning by doing aalhe presentation handoutLearning by doing aalhe presentation handout
Learning by doing aalhe presentation handout
 
Commonly used Statistics in Medical Research Handout
Commonly used Statistics in Medical Research HandoutCommonly used Statistics in Medical Research Handout
Commonly used Statistics in Medical Research Handout
 
Common measures of association in medical research handout
Common measures of association in medical research handoutCommon measures of association in medical research handout
Common measures of association in medical research handout
 

Similar to Methods for developing assessment instruments to generate useful data in t…

WORKPLACE BASED ASSESSMENT TOOL KIT THREE.ppt
WORKPLACE BASED ASSESSMENT TOOL KIT THREE.pptWORKPLACE BASED ASSESSMENT TOOL KIT THREE.ppt
WORKPLACE BASED ASSESSMENT TOOL KIT THREE.pptAbdul Jabbar Arif
 
CBME and Assessment
CBME and Assessment CBME and Assessment
CBME and Assessment jakinyi
 
Evidence based practice
Evidence based practiceEvidence based practice
Evidence based practiceArifa T N
 
KEY STEPS OF EVIEN.pptx
KEY STEPS OF EVIEN.pptxKEY STEPS OF EVIEN.pptx
KEY STEPS OF EVIEN.pptxBasitRamzan1
 
03-Planning Research.ppt
03-Planning Research.ppt03-Planning Research.ppt
03-Planning Research.pptFridahchungu
 
Evaluation of Medical literature
Evaluation of Medical literatureEvaluation of Medical literature
Evaluation of Medical literatureRamesh Parajuli
 
setting used in the study typical of clinical settings.docx
setting used in the study typical of clinical settings.docxsetting used in the study typical of clinical settings.docx
setting used in the study typical of clinical settings.docxwrite5
 
Testing a Test: Evaluating Our Assessment Tools
Testing a Test: Evaluating Our Assessment ToolsTesting a Test: Evaluating Our Assessment Tools
Testing a Test: Evaluating Our Assessment ToolsEddy White, Ph.D.
 
1. IDENTIFICATION OF RESEARCH PROBLEUM.ppt
1. IDENTIFICATION OF RESEARCH PROBLEUM.ppt1. IDENTIFICATION OF RESEARCH PROBLEUM.ppt
1. IDENTIFICATION OF RESEARCH PROBLEUM.pptSavitaHanamsagar
 
WQD2011 – INNOVATION – BRONZE WINNER – Tawam Hospital - Innovative Approaches...
WQD2011 – INNOVATION – BRONZE WINNER – Tawam Hospital - Innovative Approaches...WQD2011 – INNOVATION – BRONZE WINNER – Tawam Hospital - Innovative Approaches...
WQD2011 – INNOVATION – BRONZE WINNER – Tawam Hospital - Innovative Approaches...Dubai Quality Group
 
Keys to success with assessment and evaluation
Keys to success with assessment and evaluationKeys to success with assessment and evaluation
Keys to success with assessment and evaluationFrank Cervone
 
RESEARCH METHODOLOGY FOR UNDERGRADUATES.pptx
RESEARCH METHODOLOGY FOR UNDERGRADUATES.pptxRESEARCH METHODOLOGY FOR UNDERGRADUATES.pptx
RESEARCH METHODOLOGY FOR UNDERGRADUATES.pptxROBIN VAVACHAN
 
Developing a fundable research question: 27 June 2017
Developing a fundable research question: 27 June 2017 Developing a fundable research question: 27 June 2017
Developing a fundable research question: 27 June 2017 RDSLondon
 
Evidence Based Practice - EBP
Evidence Based Practice - EBPEvidence Based Practice - EBP
Evidence Based Practice - EBPSaira Ambreen
 
quantitative nor qualitative research.docx
quantitative nor qualitative research.docxquantitative nor qualitative research.docx
quantitative nor qualitative research.docxwrite22
 
quantitative nor qualitative research.docx
quantitative nor qualitative research.docxquantitative nor qualitative research.docx
quantitative nor qualitative research.docxstirlingvwriters
 

Similar to Methods for developing assessment instruments to generate useful data in t… (20)

York EBSC NSCA July 2009
York EBSC NSCA July 2009York EBSC NSCA July 2009
York EBSC NSCA July 2009
 
WORKPLACE BASED ASSESSMENT TOOL KIT THREE.ppt
WORKPLACE BASED ASSESSMENT TOOL KIT THREE.pptWORKPLACE BASED ASSESSMENT TOOL KIT THREE.ppt
WORKPLACE BASED ASSESSMENT TOOL KIT THREE.ppt
 
Siobhan and evelyn's presentation 15th april
Siobhan and evelyn's presentation 15th aprilSiobhan and evelyn's presentation 15th april
Siobhan and evelyn's presentation 15th april
 
CBME and Assessment
CBME and Assessment CBME and Assessment
CBME and Assessment
 
Evidence based practice
Evidence based practiceEvidence based practice
Evidence based practice
 
KEY STEPS OF EVIEN.pptx
KEY STEPS OF EVIEN.pptxKEY STEPS OF EVIEN.pptx
KEY STEPS OF EVIEN.pptx
 
model-of-human-occ-and-prac-dev-LLL event-London Region-heasman.pdf
model-of-human-occ-and-prac-dev-LLL event-London Region-heasman.pdfmodel-of-human-occ-and-prac-dev-LLL event-London Region-heasman.pdf
model-of-human-occ-and-prac-dev-LLL event-London Region-heasman.pdf
 
From Passively Received Wisdom to Actively Constructed Knowledge:
From Passively Received Wisdom to Actively Constructed Knowledge:From Passively Received Wisdom to Actively Constructed Knowledge:
From Passively Received Wisdom to Actively Constructed Knowledge:
 
03-Planning Research.ppt
03-Planning Research.ppt03-Planning Research.ppt
03-Planning Research.ppt
 
Evaluation of Medical literature
Evaluation of Medical literatureEvaluation of Medical literature
Evaluation of Medical literature
 
setting used in the study typical of clinical settings.docx
setting used in the study typical of clinical settings.docxsetting used in the study typical of clinical settings.docx
setting used in the study typical of clinical settings.docx
 
Testing a Test: Evaluating Our Assessment Tools
Testing a Test: Evaluating Our Assessment ToolsTesting a Test: Evaluating Our Assessment Tools
Testing a Test: Evaluating Our Assessment Tools
 
1. IDENTIFICATION OF RESEARCH PROBLEUM.ppt
1. IDENTIFICATION OF RESEARCH PROBLEUM.ppt1. IDENTIFICATION OF RESEARCH PROBLEUM.ppt
1. IDENTIFICATION OF RESEARCH PROBLEUM.ppt
 
WQD2011 – INNOVATION – BRONZE WINNER – Tawam Hospital - Innovative Approaches...
WQD2011 – INNOVATION – BRONZE WINNER – Tawam Hospital - Innovative Approaches...WQD2011 – INNOVATION – BRONZE WINNER – Tawam Hospital - Innovative Approaches...
WQD2011 – INNOVATION – BRONZE WINNER – Tawam Hospital - Innovative Approaches...
 
Keys to success with assessment and evaluation
Keys to success with assessment and evaluationKeys to success with assessment and evaluation
Keys to success with assessment and evaluation
 
RESEARCH METHODOLOGY FOR UNDERGRADUATES.pptx
RESEARCH METHODOLOGY FOR UNDERGRADUATES.pptxRESEARCH METHODOLOGY FOR UNDERGRADUATES.pptx
RESEARCH METHODOLOGY FOR UNDERGRADUATES.pptx
 
Developing a fundable research question: 27 June 2017
Developing a fundable research question: 27 June 2017 Developing a fundable research question: 27 June 2017
Developing a fundable research question: 27 June 2017
 
Evidence Based Practice - EBP
Evidence Based Practice - EBPEvidence Based Practice - EBP
Evidence Based Practice - EBP
 
quantitative nor qualitative research.docx
quantitative nor qualitative research.docxquantitative nor qualitative research.docx
quantitative nor qualitative research.docx
 
quantitative nor qualitative research.docx
quantitative nor qualitative research.docxquantitative nor qualitative research.docx
quantitative nor qualitative research.docx
 

More from Pat Barlow

REVISED 5-14: Curriculum vitae Barlow
REVISED 5-14: Curriculum vitae BarlowREVISED 5-14: Curriculum vitae Barlow
REVISED 5-14: Curriculum vitae BarlowPat Barlow
 
Common measures of association in medical research (UPDATED) 2013
Common measures of association in medical research (UPDATED) 2013Common measures of association in medical research (UPDATED) 2013
Common measures of association in medical research (UPDATED) 2013Pat Barlow
 
Commonly Used Statistics in Medical Research Part I
Commonly Used Statistics in Medical Research Part ICommonly Used Statistics in Medical Research Part I
Commonly Used Statistics in Medical Research Part IPat Barlow
 
Commonly Used Statistics in Survey Research
Commonly Used Statistics in Survey ResearchCommonly Used Statistics in Survey Research
Commonly Used Statistics in Survey ResearchPat Barlow
 
What's Significant? Hypothesis Testing, Effect Size, Confidence Intervals, & ...
What's Significant? Hypothesis Testing, Effect Size, Confidence Intervals, & ...What's Significant? Hypothesis Testing, Effect Size, Confidence Intervals, & ...
What's Significant? Hypothesis Testing, Effect Size, Confidence Intervals, & ...Pat Barlow
 
Retrospective application of systems thinking and isomorphism to a complex mu...
Retrospective application of systems thinking and isomorphism to a complex mu...Retrospective application of systems thinking and isomorphism to a complex mu...
Retrospective application of systems thinking and isomorphism to a complex mu...Pat Barlow
 
Comparing Research Designs
Comparing Research DesignsComparing Research Designs
Comparing Research DesignsPat Barlow
 

More from Pat Barlow (7)

REVISED 5-14: Curriculum vitae Barlow
REVISED 5-14: Curriculum vitae BarlowREVISED 5-14: Curriculum vitae Barlow
REVISED 5-14: Curriculum vitae Barlow
 
Common measures of association in medical research (UPDATED) 2013
Common measures of association in medical research (UPDATED) 2013Common measures of association in medical research (UPDATED) 2013
Common measures of association in medical research (UPDATED) 2013
 
Commonly Used Statistics in Medical Research Part I
Commonly Used Statistics in Medical Research Part ICommonly Used Statistics in Medical Research Part I
Commonly Used Statistics in Medical Research Part I
 
Commonly Used Statistics in Survey Research
Commonly Used Statistics in Survey ResearchCommonly Used Statistics in Survey Research
Commonly Used Statistics in Survey Research
 
What's Significant? Hypothesis Testing, Effect Size, Confidence Intervals, & ...
What's Significant? Hypothesis Testing, Effect Size, Confidence Intervals, & ...What's Significant? Hypothesis Testing, Effect Size, Confidence Intervals, & ...
What's Significant? Hypothesis Testing, Effect Size, Confidence Intervals, & ...
 
Retrospective application of systems thinking and isomorphism to a complex mu...
Retrospective application of systems thinking and isomorphism to a complex mu...Retrospective application of systems thinking and isomorphism to a complex mu...
Retrospective application of systems thinking and isomorphism to a complex mu...
 
Comparing Research Designs
Comparing Research DesignsComparing Research Designs
Comparing Research Designs
 

Recently uploaded

Neurological history taking (2024) .
Neurological  history  taking  (2024)  .Neurological  history  taking  (2024)  .
Neurological history taking (2024) .Mohamed Rizk Khodair
 
Generative AI in Health Care a scoping review and a persoanl experience.
Generative AI in Health Care a scoping review and a persoanl experience.Generative AI in Health Care a scoping review and a persoanl experience.
Generative AI in Health Care a scoping review and a persoanl experience.Vaikunthan Rajaratnam
 
SGK RỐI LOẠN KALI MÁU CỰC KỲ QUAN TRỌNG.pdf
SGK RỐI LOẠN KALI MÁU CỰC KỲ QUAN TRỌNG.pdfSGK RỐI LOẠN KALI MÁU CỰC KỲ QUAN TRỌNG.pdf
SGK RỐI LOẠN KALI MÁU CỰC KỲ QUAN TRỌNG.pdfHongBiThi1
 
PAIN/CLASSIFICATION AND MANAGEMENT OF PAIN.pdf
PAIN/CLASSIFICATION AND MANAGEMENT OF PAIN.pdfPAIN/CLASSIFICATION AND MANAGEMENT OF PAIN.pdf
PAIN/CLASSIFICATION AND MANAGEMENT OF PAIN.pdfDolisha Warbi
 
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdf
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdfSGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdf
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdfHongBiThi1
 
Mental health Team. Dr Senthil Thirusangu
Mental health Team. Dr Senthil ThirusanguMental health Team. Dr Senthil Thirusangu
Mental health Team. Dr Senthil Thirusangu Medical University
 
historyofpsychiatryinindia. Senthil Thirusangu
historyofpsychiatryinindia. Senthil Thirusanguhistoryofpsychiatryinindia. Senthil Thirusangu
historyofpsychiatryinindia. Senthil Thirusangu Medical University
 
ayurvedic formulations herbal drug technologyppt
ayurvedic formulations herbal drug technologypptayurvedic formulations herbal drug technologyppt
ayurvedic formulations herbal drug technologypptPradnya Wadekar
 
CPR.nursingoutlook.pdf , Bsc nursing student
CPR.nursingoutlook.pdf , Bsc nursing studentCPR.nursingoutlook.pdf , Bsc nursing student
CPR.nursingoutlook.pdf , Bsc nursing studentsaileshpanda05
 
How to cure cirrhosis and chronic hepatitis naturally
How to cure cirrhosis and chronic hepatitis naturallyHow to cure cirrhosis and chronic hepatitis naturally
How to cure cirrhosis and chronic hepatitis naturallyZurück zum Ursprung
 
Male Infertility, Antioxidants and Beyond
Male Infertility, Antioxidants and BeyondMale Infertility, Antioxidants and Beyond
Male Infertility, Antioxidants and BeyondSujoy Dasgupta
 
Unit I herbs as raw materials, biodynamic agriculture.ppt
Unit I herbs as raw materials, biodynamic agriculture.pptUnit I herbs as raw materials, biodynamic agriculture.ppt
Unit I herbs as raw materials, biodynamic agriculture.pptPradnya Wadekar
 
DNA nucleotides Blast in NCBI and Phylogeny using MEGA Xi.pptx
DNA nucleotides Blast in NCBI and Phylogeny using MEGA Xi.pptxDNA nucleotides Blast in NCBI and Phylogeny using MEGA Xi.pptx
DNA nucleotides Blast in NCBI and Phylogeny using MEGA Xi.pptxMAsifAhmad
 
SGK NGẠT NƯỚC ĐHYHN RẤT LÀ HAY NHA .pdf
SGK NGẠT NƯỚC ĐHYHN RẤT LÀ HAY NHA    .pdfSGK NGẠT NƯỚC ĐHYHN RẤT LÀ HAY NHA    .pdf
SGK NGẠT NƯỚC ĐHYHN RẤT LÀ HAY NHA .pdfHongBiThi1
 
EXERCISE PERFORMANCE.pptx, Lung function
EXERCISE PERFORMANCE.pptx, Lung functionEXERCISE PERFORMANCE.pptx, Lung function
EXERCISE PERFORMANCE.pptx, Lung functionkrishnareddy157915
 
SGK ĐIỆN GIẬT ĐHYHN RẤT LÀ HAY TUYỆT VỜI.pdf
SGK ĐIỆN GIẬT ĐHYHN        RẤT LÀ HAY TUYỆT VỜI.pdfSGK ĐIỆN GIẬT ĐHYHN        RẤT LÀ HAY TUYỆT VỜI.pdf
SGK ĐIỆN GIẬT ĐHYHN RẤT LÀ HAY TUYỆT VỜI.pdfHongBiThi1
 
Bulimia nervosa ( Eating Disorders) Mental Health Nursing.
Bulimia nervosa ( Eating Disorders) Mental Health Nursing.Bulimia nervosa ( Eating Disorders) Mental Health Nursing.
Bulimia nervosa ( Eating Disorders) Mental Health Nursing.aarjukhadka22
 

Recently uploaded (20)

Neurological history taking (2024) .
Neurological  history  taking  (2024)  .Neurological  history  taking  (2024)  .
Neurological history taking (2024) .
 
Generative AI in Health Care a scoping review and a persoanl experience.
Generative AI in Health Care a scoping review and a persoanl experience.Generative AI in Health Care a scoping review and a persoanl experience.
Generative AI in Health Care a scoping review and a persoanl experience.
 
SGK RỐI LOẠN KALI MÁU CỰC KỲ QUAN TRỌNG.pdf
SGK RỐI LOẠN KALI MÁU CỰC KỲ QUAN TRỌNG.pdfSGK RỐI LOẠN KALI MÁU CỰC KỲ QUAN TRỌNG.pdf
SGK RỐI LOẠN KALI MÁU CỰC KỲ QUAN TRỌNG.pdf
 
PAIN/CLASSIFICATION AND MANAGEMENT OF PAIN.pdf
PAIN/CLASSIFICATION AND MANAGEMENT OF PAIN.pdfPAIN/CLASSIFICATION AND MANAGEMENT OF PAIN.pdf
PAIN/CLASSIFICATION AND MANAGEMENT OF PAIN.pdf
 
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdf
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdfSGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdf
SGK RỐI LOẠN TOAN KIỀM ĐHYHN RẤT HAY VÀ ĐẶC SẮC.pdf
 
Mental health Team. Dr Senthil Thirusangu
Mental health Team. Dr Senthil ThirusanguMental health Team. Dr Senthil Thirusangu
Mental health Team. Dr Senthil Thirusangu
 
historyofpsychiatryinindia. Senthil Thirusangu
historyofpsychiatryinindia. Senthil Thirusanguhistoryofpsychiatryinindia. Senthil Thirusangu
historyofpsychiatryinindia. Senthil Thirusangu
 
ayurvedic formulations herbal drug technologyppt
ayurvedic formulations herbal drug technologypptayurvedic formulations herbal drug technologyppt
ayurvedic formulations herbal drug technologyppt
 
CPR.nursingoutlook.pdf , Bsc nursing student
CPR.nursingoutlook.pdf , Bsc nursing studentCPR.nursingoutlook.pdf , Bsc nursing student
CPR.nursingoutlook.pdf , Bsc nursing student
 
GOUT UPDATE AHMED YEHIA 2024, case based approach with application of the lat...
GOUT UPDATE AHMED YEHIA 2024, case based approach with application of the lat...GOUT UPDATE AHMED YEHIA 2024, case based approach with application of the lat...
GOUT UPDATE AHMED YEHIA 2024, case based approach with application of the lat...
 
How to cure cirrhosis and chronic hepatitis naturally
How to cure cirrhosis and chronic hepatitis naturallyHow to cure cirrhosis and chronic hepatitis naturally
How to cure cirrhosis and chronic hepatitis naturally
 
Male Infertility, Antioxidants and Beyond
Male Infertility, Antioxidants and BeyondMale Infertility, Antioxidants and Beyond
Male Infertility, Antioxidants and Beyond
 
Unit I herbs as raw materials, biodynamic agriculture.ppt
Unit I herbs as raw materials, biodynamic agriculture.pptUnit I herbs as raw materials, biodynamic agriculture.ppt
Unit I herbs as raw materials, biodynamic agriculture.ppt
 
DNA nucleotides Blast in NCBI and Phylogeny using MEGA Xi.pptx
DNA nucleotides Blast in NCBI and Phylogeny using MEGA Xi.pptxDNA nucleotides Blast in NCBI and Phylogeny using MEGA Xi.pptx
DNA nucleotides Blast in NCBI and Phylogeny using MEGA Xi.pptx
 
SGK NGẠT NƯỚC ĐHYHN RẤT LÀ HAY NHA .pdf
SGK NGẠT NƯỚC ĐHYHN RẤT LÀ HAY NHA    .pdfSGK NGẠT NƯỚC ĐHYHN RẤT LÀ HAY NHA    .pdf
SGK NGẠT NƯỚC ĐHYHN RẤT LÀ HAY NHA .pdf
 
EXERCISE PERFORMANCE.pptx, Lung function
EXERCISE PERFORMANCE.pptx, Lung functionEXERCISE PERFORMANCE.pptx, Lung function
EXERCISE PERFORMANCE.pptx, Lung function
 
SGK ĐIỆN GIẬT ĐHYHN RẤT LÀ HAY TUYỆT VỜI.pdf
SGK ĐIỆN GIẬT ĐHYHN        RẤT LÀ HAY TUYỆT VỜI.pdfSGK ĐIỆN GIẬT ĐHYHN        RẤT LÀ HAY TUYỆT VỜI.pdf
SGK ĐIỆN GIẬT ĐHYHN RẤT LÀ HAY TUYỆT VỜI.pdf
 
How to master Steroid (glucocorticoids) prescription, different scenarios, ca...
How to master Steroid (glucocorticoids) prescription, different scenarios, ca...How to master Steroid (glucocorticoids) prescription, different scenarios, ca...
How to master Steroid (glucocorticoids) prescription, different scenarios, ca...
 
Bulimia nervosa ( Eating Disorders) Mental Health Nursing.
Bulimia nervosa ( Eating Disorders) Mental Health Nursing.Bulimia nervosa ( Eating Disorders) Mental Health Nursing.
Bulimia nervosa ( Eating Disorders) Mental Health Nursing.
 
Immune labs basics part 1 acute phase reactants ESR, CRP Ahmed Yehia Ismaeel,...
Immune labs basics part 1 acute phase reactants ESR, CRP Ahmed Yehia Ismaeel,...Immune labs basics part 1 acute phase reactants ESR, CRP Ahmed Yehia Ismaeel,...
Immune labs basics part 1 acute phase reactants ESR, CRP Ahmed Yehia Ismaeel,...
 

Methods for developing assessment instruments to generate useful data in t…

  • 1. METHODS FOR DEVELOPING ASSESSMENT INSTRUMENTS TO GENERATE USEFUL DATA IN THE PRESENCE OF VAGUE COURSE OBJECTIVES Patrick B. Barlow Tiffany L. Smith Eric Heidel, PhD William Metheny, PhD
  • 2. On the Agenda SETTING THE SCENE • Who? What? Where? When? WHY? • Assessment in Graduate Medical Education FIVE PRACTICAL TIPS • How we addressed the problem USE OF ASSESSMENT RESULTS • How were these methods useful? DISCUSSION • Questions?
  • 3. Who? What? Where? When? SETTING THE SCENE
  • 4. Office of Medical Education, Research, and Development (OMERAD) • Job Description • Consultation and education • What was happening with GME at our institution? • New Office Structure • PhD students in ESM brought in • Office given the reigns of the clinical research skills curricula
  • 5. Assessment in Graduate Medical Education WHY DOES THIS NEED TO BE ADDRESSED?
  • 6. What is EBM? Clinical Epidemiology Biostatistics Critical Appraisal Evidence-Based Medicine in GME1,2
  • 7. What We Know About Resident Knowledge of Clinical Research Skills? • Error rate in reporting and interpreting statistics in medicine 3 is estimated between 30-90% • Consistent… • Lack of knowledge 4,5 • Lack of confidence An example…
  • 8. How we addressed the problem FIVE PRACTICAL TIPS
  • 9. TIP ONE Know Your Situation • Learning environment factors • Statistics and research methods as a topic • No formal “courses”, nothing is “required” • No previous learning objectives, syllabus, or assessment structure • Work environment factors • Hospital obligations • Attending physician buy-in & priorities
  • 10. TIP ONE Know Your Situation • Population-specific factors • Variable background experience • Low average competence and confidence • Realities of being a physician • Availability of resources • Limited time • Limited money
  • 11. TIP TWO Clarify Your Purpose • Ask two questions: • How will the assessment audience benefit from the results? • How will the students benefit from the assessment results? • In our case • Audience (OMERAD, GSM faculty/administration) • Students (Residents, fellows, physicians , & staff)
  • 12. TIP THREE Use What You Have • Gather the Necessary Background Data • Existing content • Faculty interviews • Direct observation • Literature • Clinical/Work experience • Three benefits • What instructors think the students are learning • What is being taught • Where the gaps are in the curriculum
  • 13. TIP FOUR Fit the Instrument to Your Purpose, Not the Other Way Around • Again, consider situational factors • Resources for types of assessment instruments • What worked for us • Background knowledge 6 probe
  • 14. TIP FIVE Get Consistent and Critical Feedback Assessment must be viewed as a never- Develop/ Modify ending, iterative process • An instrument is developed or modified • The instrument is “Feedback tested Loop” of Assessment • Testing generates Practice feedback • Feedback leads to Feedback Test modifications…
  • 15. TIP FIVE Get Consistent and Critical Feedback Assessment must be viewed as a never- Develop/ Modify ending, iterative process • An instrument is developed or modified • The instrument is “Feedback tested Loop” of Assessment • Testing generates Practice feedback • Feedback leads to Feedback Test modifications… • These modifications are tested
  • 16. An Integrated Assessment Model for a Dynamic Learning Environment USE OF RESULTS
  • 17. Improvements to the Course: Learning Objective Development • Multiple sources of data • Assessment Concrete • Experiential “List of Topics” Learning • Evaluation Objectives • Data identified • Salient topics • Missing content • Student needs • Need for a BEFORE AFTER responsive curriculum
  • 18. Improvements to the Assessment: Test blueprint process used to improve the assessment instrument Module 2:Comparing Research Designs 1. Start with course learning objectives 2. Identify test “topics” Identify major from learning epidemiologic objectives research “after this 3. Expand each topic designs module to as many participants “concepts” as should be Apply each possible able to…” design to their 4. Collapse list of own area of research concepts to remove redundancy 5. Create/modify items
  • 19. Next Steps • Continue instrument & curriculum revisions • Standardized assessment for residents, fellows, physicians on clinical research skills and statistics.
  • 21. References 1. Green, M. L. (2000). Evidence-based medicine training in graduate medical education: past, present and future. Journal of evaluation in clinical practice, 6(2), 121–38. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/10970006 2. Stewart, M.G. (2001). ACGME Core Compentencies Accreditation Council for Graduate Medical Education. Retrieved from http://www.acgme.org/acWebsite/RRC_280/280_coreComp.asp 3. Novack, L., Jotkowitz, A., Knyazer, B., & Novack, V. (2006). Evidence- based medicine: assessment of knowledge of basic epidemiological and research methods among medical doctors. Postgraduate Medical Journal, 82(974), 817–822. Retrieved from http://pmj.bmj.com/content/82/974/817.abstract 4. West, C. P., & Ficalora, R. D. (2007). Clinician Attitudes Toward Biostatistics. Mayo Clinic Proceedings, 82(8), 939–943. Retrieved from http://www.mayoclinicproceedings.com/content/82/8/939.abstract 5. Windish, D. M., Huot, S. J., & Green, M. L. (2007). Medicine Residents’ Understanding of the Biostatistics and Results in the Medical Literature. JAMA: The Journal of the American Medical Association, 298(9), 1010– 1022. Retrieved from http://jama.ama- assn.org/content/298/9/1010.abstract 6. Angelo, T. & Cross, K.P. (1993). Classroom Assessment Techniques. San Francisco: Jossey-Bass. 7. Fink, L.D. (2003). Creating Significant Learning Experiences: An integrated approach to designing college courses. San Francisco, CA: John Wiley & Sons, Inc.