1) The document discusses competency-based assessment and e-portfolios in residency training programs in the Netherlands. It implemented an e-portfolio system called the Maastricht Manchester Transactional Support System (MMTSS) to address problems with traditional workplace-based assessment. 2) The MMTSS uses an online platform to facilitate competency-based assessments aligned with CanMEDS roles. It provides automated feedback reports, allows comparison to peer performance over time, and supports documentation of learning activities. 3) A small-scale pilot of the MMTSS found that it promoted meaningful workplace-based assessment and feedback that was transparent and user-friendly. Ensuring its long-term, purposeful

1. Competency-based assessmentCompetency-based assessment
and e-portfolio in residency trainingand e-portfolio in residency training
Maastricht UniversityMaastricht University
Manchester UniversityManchester University
Marjan Govaerts
J. Donkers; H.Brackel; B. Verhoeven;
C. Van der Vleuten; E. Driessen; T. Dornan

2. Residency Training NetherlandsResidency Training Netherlands
Nationwide reformNationwide reform
Competency-based: CanMEDS rolesCompetency-based: CanMEDS roles
Workplace-based assessmentWorkplace-based assessment
PortfolioPortfolio
– Guidance of competence developmentGuidance of competence development
– Evidence-based decision makingEvidence-based decision making

3. Implementation ProblemsImplementation Problems
Workplace-based assessment: timeWorkplace-based assessment: time
consuming and administrative overloadconsuming and administrative overload
– Meaningful feedback is lackingMeaningful feedback is lacking
– Documentation of feedback/performance isDocumentation of feedback/performance is
lackinglacking
Portfolio: limited valuePortfolio: limited value
– Information in portfolio non-transparentInformation in portfolio non-transparent
– Credible (evidence-based) norms are lackingCredible (evidence-based) norms are lacking

4. Maastricht Manchester TSSMaastricht Manchester TSS
E-portfolio with built-in online supportE-portfolio with built-in online support
system for Workplace Assessmentsystem for Workplace Assessment
Rating forms attuned to educational goalsRating forms attuned to educational goals
– Competency-based (CanMEDS)Competency-based (CanMEDS)
Limited number of rating instrumentsLimited number of rating instruments
– Mini-CEX; OSATS; MSF (incl. self assessment)Mini-CEX; OSATS; MSF (incl. self assessment)
– Numeric scores for behavioral objectives + narrativeNumeric scores for behavioral objectives + narrative
feedbackfeedback

5. Maastricht Manchester TSSMaastricht Manchester TSS
1.1: 3.51.1: 3.5
1.2: 4.21.2: 4.2
2.1: 5.02.1: 5.0
2.2: 4.32.2: 4.3
7.4: 3.47.4: 3.4
ScoresScores
Rating formRating form
CC
AA
NN
MM
EE
DD
SS
LongitudinalLongitudinal
developmentdevelopment
Communicator
Cohort scoresCohort scores
Performance
peers

6. MMTSS feedbackMMTSS feedback
Medical expert
Professional
Manager
Communicator
Collaborator
Scholar Health advocate
Individual versus
peer group
Medical expert
Communicator
Collaborator
Scholar
Health advocate
Manager
Professional
Individual over time

7. MMTSS feedbackMMTSS feedback
Narrative feedback
Performance ratings
(others vs self)

8. Maastricht Manchester TSSMaastricht Manchester TSS
Rating forms attuned to educational goalsRating forms attuned to educational goals
– Competency-based template (CanMEDS)Competency-based template (CanMEDS)
Aggregation of scores across rating instrumentsAggregation of scores across rating instruments
– Automated feedback reports on competenceAutomated feedback reports on competence
developmentdevelopment
Collection of performance rating data inCollection of performance rating data in
databasedatabase
– Evidence-based norms and decision makingEvidence-based norms and decision making
Comparison of individual performance with aggregatedComparison of individual performance with aggregated
performance data of peer groupperformance data of peer group

9. MMTSS E-portfolioMMTSS E-portfolio
Documentation (surgical) procedures /Documentation (surgical) procedures /
learning activitieslearning activities
Reflective practice and self-directedReflective practice and self-directed
learninglearning
– Critical incident reportsCritical incident reports
– Self assessment and analysisSelf assessment and analysis
– Personal Development PlansPersonal Development Plans
…….other artefacts/’evidence’ illustrating.other artefacts/’evidence’ illustrating
achievement of professional competenceachievement of professional competence

10. Results small scale pilotResults small scale pilot
Promotes workplace based assessmentPromotes workplace based assessment
Provides meaningful feedbackProvides meaningful feedback
– Insight in strengths and weaknessesInsight in strengths and weaknesses
– AcceptableAcceptable
– TransparentTransparent
User-friendlyUser-friendly

11. ChallengesChallenges
How to encourage meaningful use of theHow to encourage meaningful use of the
portfolio afterportfolio after large scale implementationlarge scale implementation??
Organizational support requiredOrganizational support required
– Purposeful use by residents, teachersPurposeful use by residents, teachers
Reflection and self-directed learningReflection and self-directed learning
– Quality improvement of residency trainingQuality improvement of residency training
Flexible, tailor-made, cost-efficientFlexible, tailor-made, cost-efficient

12. Thank you for your attention.Thank you for your attention.
Questions?Questions?