š«ā jodhpur 24Ć7 BEST GENUINE PERSON LOW PRICE CALL GIRL SERVICE FULL SATISFACT...
Ā
Usability Testing Medical Devices
1. UPA 2012
Usability Testing Medical Devices Las Vegas, NV
Dan Berlin Chris Hass
Experience Research Director SVP of Experience Design
@banderlin @chrishass
dberlin@madpow.net chass@madpow.net
2. Usability activities are NOT clinical Trials
Generally the process will be less stringent:
ā¢āÆ More loosely defined, faster
ā¢āÆ Will likely involve a relatively small number of people per activity
ā¢āÆ Initial studies stress simulation, with no medicinal agents or clinical
impact.
ā¢āÆ Dovetail with interaction design, visual design, and marketing activities
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 2
Ā
3. Itās going to feel like a clinical trial
ā¢āÆ Extremely detailed āUsability Specificationā documents
ā¢āÆ āUsability Validationā (Risk assessment and amelioration)
ā¢āÆ āUsability engineering fileā / āDesign Efficiency Fileā
ā¢āÆ Know Federal and International regulations:
ā¢āÆ FDA Quality System Regulation
ā¢āÆ HE: 75 2009
ā¢āÆ ISO/IEC 62366:2007
ā¢āÆ Knowledge of (and inspiration for)
emerging standards
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 3
Ā
4. Medical Device Manufacturers may be unfamiliar
with your practices and lack of regulations
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 4
Ā
5. Safeguarding participants may be more challenging
ā¢āÆ Your safety protocols must be exacting, fully defined,
piloted, and effective
ā¢āÆ Even simulated product use may put participants at risk
ā¢āÆ Target audiences may be physically or cognitively
vulnerable
ā¢āÆ Activity environments may be unusual
ā¢āÆ Work with client and medical care specialists
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 5
Ā
6. Reporting responsibilities, guidelines, formats, and
requirements will be more complex
ā¢āÆ Reporting guidelines, formats, and requirements will likely
be extensive and complex
ā¢āÆ Industry reporting standards (CIF)
ā¢āÆ Specialized reporting formats
ā¢āÆ Raw data may be reviewed again and again and again. . .
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 6
Ā
7. Research locales may be more challenging
Device use environments may be:
ā¢āÆ Unfamiliar
ā¢āÆ Sterile
ā¢āÆ Stressful
ā¢āÆ Germ-laden
ā¢āÆ Emotionally distressing
ā¢āÆ Gross
ā¢āÆ Secure
ā¢āÆ Rarely observable
ā¢āÆ Complex
ā¢āÆ or otherwise unusual to non-medical personnel
ā¢āÆ (And fainting can be a factor!)
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 7
Ā
8. Client Relationship
Source: http://robot6.comicbookresources.com/2010/07/take-a-look-at-the-avengers-assemble-painting-by-alex-ross/
ā¢āÆ A robust set of client team members is essential ā
as is knowing their roles:
āāÆ Legal
āāÆ Medical advisor
āāÆ Safety advisor
āāÆ Compliance
āāÆ Adverse Event Specialist
āāÆ Physician is helpful, but not enough
āāÆ The usual suspects
ā¢āÆ Clients may ask for proof of insurance
āāÆ But too much insurance could be a bad thing
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 8
Ā
9. FDA Relationship
Source: http://news.petpardons.com/fda-denies-rumors-over-
possible-discovery-of-chicken-jerky-toxin/
ā¢āÆ If the study is for FDA approval ā learn about
the clientās relationship with the FDA:
āāÆ What are the relevant regulations?
āāÆ Who is the clientās liaison to the FDA?
āāÆ What is the clientās relationship with the FDA?
āāÆ Has the client met with the FDA?
āāÆ Have they received comments from the FDA?
āāÆ Does the client understand usability testing?
āāÆ Are we allowed to talk to the FDA?
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 9
Ā
10. Working with an IRB
ā¢āÆ IRB = Institutional Review Board Source: http://www.bioedge.org/index.php/bioethics/bioethics_article/
who_is_watching_the_irb_watchdogs_asks_nature
ā¢āÆ For FDA approval, typically work with an independent IRB
ā¢āÆ Their job: to ensure that research conducted on human subjects is carried
out correctly and humanely
ā¢āÆ Not as scary as you may think
āāÆ We are NOT evaluating medical efficacy or safety
āāÆ But we DO have to be conscientious
ā¢āÆ Have your ducks in a row and it will be a smooth process
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 10
Ā
11. Working with an IRB
ā¢āÆ Adds about $2000-$5000 to the cost of the study
Source: http://www.bioedge.org/index.php/bioethics/bioethics_article/
who_is_watching_the_irb_watchdogs_asks_nature
ā¢āÆ Adds a day or two to the project for dealing with
the IRB
ā¢āÆ Be cognizant of any non-disclosure agreements
ā¢āÆ Prior to the study, signature pages should be
signed-off, circulated, and returned in a timely
manner
ā¢āÆ Everyone on the team will take this very
seriously!
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 11
Ā
12. IRB Documents - Forms
Source: http://blog.pennysaverusa.com/tag/tax-tips/
ā¢āÆ Indemnification Agreement: So the IRB doesnāt get sued
ā¢āÆ Financial Disclosure: The researchers donāt have conflicts of interest
ā¢āÆ Investigator Statement: The researcher code of conduct
ā¢āÆ Satellite Site Application: May be needed if the study involves multiple
locations
ā¢āÆ Shipping & Invoice: Where documents are sent
ā¢āÆ Study Application: Detailed information about the study
ā¢āÆ Curriculum Vitae for all researchers
ā¢āÆ Other forms that your IRB requires
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 12
Ā
13. IRB Documents ā Study and Device Info
ā¢āÆ Recruitment Screener: Be very precise Source: http://healthcareit.etisbew.com/
ā¢āÆ Recruitment Flyer: To hang in doctorās office to aid with recruiting
ā¢āÆ Moderatorās Guide: A second guide may be needed if there are child
participants
ā¢āÆ Consent and asset forms: For adults and children, respectively
ā¢āÆ Safety Concerns: Outline potential risks
ā¢āÆ Device Background: Detailed explanation of the device including
indication and new features
ā¢āÆ Device Instructions for Use: If they are also included in the study
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 13
Ā
14. IRB Documents ā Validation Plan
ā¢āÆ A detailed explanation of how the study will
validate that the device is usable
āāÆ If youāre used to winging it, you canāt do that
here
āāÆ Lay out every question and how it will be
interpreted
āāÆ Lay out a rationale for every metric
āāÆ Break down the pieces of the tasks and what
will be observed
āāÆ A detailed explanation of how the will be
analyzed and presented
ā¢āÆ Quantitative measures (asked and observed)
āāÆ Rating scales
āāÆ Yes/No observations
Source:
Ā hDp://www.vnet5.org/pub/approach/uva.html
Ā
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 14
Ā
15. Chart
Ā Color
Ā
Quantitative Measures
%
Ā of
Ā chart
Ā that
Ā is
Ā red
Ā
%
Ā of
Ā chart
Ā that
Ā is
Ā black
Ā
ā¢āÆ Rating Scales
āāÆ Asked āāÆ Observed
ā¢āÆ Ease of use ā¢āÆ Usability
ā¢āÆ Instructions for ā¢āÆ Help needed
Use clarity ā¢āÆ Number of
ā¢āÆ Design errors
ā¢āÆ Ergonomics
ā¢āÆ Yes/No observations
āāÆ Did the participant do this, that, AND the other thing?
Ra,ng
Ā No
Ā Help
Ā Needed
Ā LiDle
Ā Diļ¬culty
Ā Some
Ā Help
Ā
Ā Much
Ā Help
Ā Task
Ā Failure
Ā
Explana,on
Ā Completed
Ā with
Ā no
Ā help
Ā 1-Āā2
Ā small
Ā errors
Ā 3-Āā5
Ā errors
Ā Moderator
Ā gets
Ā involved
Ā Failure
Ā
Small
Ā errors:
Ā Doesn't
Ā press
Ā the
Ā green
Ā buDon
Ā
Doesn't
Ā remove
Ā cap
Ā properly
Ā
Etc.
Ā
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 15
Ā
16. Recruiting
ā¢āÆ Give your recruiter a heads up LONG before the
study starts
āāÆ And start recruiting early
ā¢āÆ Make a flyer to be hung in doctorās offices and
distributed electronically which includes:
āāÆ The purpose of the study
āāÆ Why they would want to participate
āāÆ Compensation Source: http://www.toptenz.net/top-10-wwii-propaganda-posters.php
āāÆ Recruiter contact information
ā¢āÆ During the project, any screener changes need to be
re-approved by the IRB
ā¢āÆ Many institutions canāt āpromoteā a study that is
under their auspices
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 16
Ā
18. Recruiting
ā¢āÆ Donāt be shy ā makes friends with local hospitals
and clinics
āāÆ Nurses and admins are the gateway
āāÆ Have the sanitized IRB approval in-hand
ā¢āÆ Find groups that can be a gateway to your desired
audience
Source: http://www.toptenz.net/top-10-wwii-propaganda-posters.php
āāÆ Healthcare institutions
āāÆ Local and national support groups
āāÆ Federal agencies
āāÆ Community support groups
āāÆ Craigslist & Social Media
ā¢āÆ Plan for surprises ā like extensions to the timeline
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 18
Ā
19. Working with Children
ā¢āÆ From a legal standpoint, a child is someone under 18
years old
Source: http://evilstaring.com/2011/06/08/amazingly-evil-child-3/
ā¢āÆ Children cannot consent to participate in the study,
they can only assent
āāÆ An assent form explains the details of the study, in the
first person
ā¢āÆ You will treat a 17 year old different than an 8 year old
āāÆ Be cognizant of the protocol to ensure you are
consistent in your data capture
ā¢āÆ The parent conundrum: should they participate?
āāÆ At what point does it become co-discovery and is that Source: http://cautionmagnet.blogspot.com/2011/05/confused-kids.html
okay?
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 19
Ā
20. Working with Children
ā¢āÆ Put the child at ease:
āāÆ Explain the lab (to a point)
āāÆ Ask if s/he would like a break Source: http://lovefunnypictures.com/funny-pic-bush_funny_face.php
āāÆ Give reminders that s/he is doing a great job in the study
āāÆ Structure the study so that it isnāt drudgery
āāÆ Donāt make them remember rating scales
āāÆ Not all children like to be spoken to like a child
ā¢āÆ Be extra vigilant for signs that the participant is telling you want you
want to hear
āāÆ As well as signs of fatigue and confusion
ā¢āÆ You donāt need to be their friend, but donāt scare them away
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 20
Ā
21. Moderatorās Guide
ā¢āÆ Be precise with a tightly written protocol Source: http://tips.tntdental.com/?tag=website-navigation
ā¢āÆ Must be well-defined in advance and piloted
āāÆ No changes to the protocol without revisiting the IRB, the client, the FDA
approval minutes, etc.
ā¢āÆ Clients may be used to clinical trials and view usability methods as āthinā
āāÆ Weāre not running a clinical trials, but thatās no excuse for not adequately
preparing
ā¢āÆ Adhere to the protocol!
āāÆ Any changes require IRB approval
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 21
Ā
22. Moderatorās Guide
ā¢āÆ The moderatorās guide should capture: Source: http://tips.tntdental.com/?tag=website-navigation
āāÆ The tasks
āāÆ Steps in the tasks
āāÆ What should be observed
āāÆ The questions to be asked of the participants
āāÆ Observer rating scales
ā¢āÆ Basically, a shortened version of the Validation Plan
ā¢āÆ Be precise in the metrics and methods
āāÆ They cannot vary
āāÆ Data reporting requirements will be significant
āāÆ Gaps in the reporting will be very visible
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 22
Ā
23. Notes Grid
ā¢āÆ We are using āmicro-tasksā and want to
capture precise information
ā¢āÆ The notes grid should facilitate the analysis
Source: http://beta.armagetronad.net/design/tron.html
yet be easy to use
āāÆ Every step and question on its own row
āāÆ Have clear study sections (introduction, etc.)
āāÆ Grey out areas that can be ignored
ā¢āÆ Team note taking: moderator concentrates on
certain items, note taker does the rest
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 23
Ā
25. Lab Preparation
ā¢āÆ Capture the participants from two
different angles
āāÆ A close-up view of the device interaction
(over-the-shoulder)
āāÆ Straight-on to capture the entire field
ā¢āÆ Have all device supplies at the ready
āāÆ Alcohol wipes, extra needles, batteries, Source: http://www.ebaumsworld.com/pictures/view/218095/
etc.
āāÆ Youāll use more supplies than you expect
ā¢āÆ Get a first aid kit and nitrile gloves
āāÆ Have a first aid plan and practice
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 25
Ā
26. Participant Safety
ā¢āÆ As with any usability, participant comfort and safety is the
top priority Source: http://en.wikipedia.org/wiki/File:Sign_first_aid.svg
ā¢āÆ Must be mindful of participants with health conditions
āāÆ Will they need accessibility accommodations?
āāÆ Will they have an aide/caretaker?
āāÆ How will the condition affect their participation?
ā¢āÆ What you can do to ensure participant comfort and safety:
āāÆ Have a robust first aid kit (and possibly medical personnel)
āāÆ Provide water, juice, sugary, and non-sugary snacks
āāÆ Ensure they know that they will be working with a placebo
āāÆ Accessible food
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 26
Ā
27. Participant Safety
ā¢āÆ Warn participants of safety hazards while using the device, if itās
part of the protocol Source: http://en.wikipedia.org/wiki/File:Sign_first_aid.svg
ā¢āÆ Have a plan of action if someone is injured
ā¢āÆ Establish a path of notification if a participant is injured
āāÆ Reporting may have many phases: reporting it internally, then to the
client, to the FDA, and the IRB
ā¢āÆ Educate yourself about the condition
ā¢āÆ Ensure the protocol does not exacerbate a disease or condition
ā¢āÆ Be vigilant about fatigue, confusion, disorientation, dehydration, &
stress
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 27
Ā
28. Adverse Effects
ā¢āÆ An adverse effect (AE) event is when a participant
tells you about a side effect or other experience
while using a device or drug
ā¢āÆ This is critical information for the client and must
be reported, typically immediately
Source: http://www.cartoonstock.com/directory/d/Din.asp
ā¢āÆ Some clients may require you to go through AE
training
ā¢āÆ Before the study begins, determine to whom AEs
are reported When in doubt,
report!
ā¢āÆ AEs can be complex to identify, but are VERY
important
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 28
Ā
29. Adverse Effects
ā¢āÆ In the course of a usability study for a medication website, a participant
mentions that she stopped taking the drug. She explains that the blister
packs and pills sometimes arrived crushed.
ā¢āÆ While testing an infusion pump, the participant mentions that the tube
always falls out of his previously used pump, which is the clientās
product.
ā¢āÆ During a study, a participant mentions that her arms started to itch when
she stopped taking the clientās medication.
ā¢āÆ A male participant mentions that his wife got pregnant two weeks after
he started taking the clientās medication, even though she used the birth
control pill.
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 29
Ā
30. Analysis
ā¢āÆ The setup from the validation plan, moderatorās
guide, and notes grid should make analysis easy
Source: http://www.webseoanalytics.com/blog/how-to-conduct-market-
research-competitor-analysis/
ā¢āÆ Dealing with gaps in the data can be tricky
āāÆ Did the question get asked?
āāÆ Was the behavior observed?
āāÆ Do we have to go back to the recordings?
ā¢āÆ Make sure you HAVE the recordings!
ā¢āÆ Rely on quantitative measures
āāÆ What percentage of participants passed the task?
āāÆ What ease of use ratings did participants give?
āāÆ How many participants committed the most
common errors?
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 30
Ā
31. Analysis
ā¢āÆ Remember to think on a micro-task scale ā every little
step of using the device should be observed
āāÆ Itās difficult to watch and report on every step of a
complex interaction and document it precisely, Source: http://www.webseoanalytics.com/blog/how-to-conduct-market-
research-competitor-analysis/
especially live
āāÆ Tallying needs to be specific, yet practical
ā¢āÆ Be mindful of capturing and organizing the important
qualitative data
āāÆ This can be especially tough when dealing with many
participants
āāÆ Summarize qualitative data directly after the session ā
donāt wait
ā¢āÆ Know what youāre reporting ā there may be data to
help marketing
ā¢āÆ Sometimes the goal of these studies is to identify
marketing claims, which can raise concerns
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 31
Ā
32. Thank You!
ā¢āÆ Conducting a usability study for a medical
device is similar to a typical study, except
with much stricter protocols Questions/Comments?
ā¢āÆ Your safety and the safety & comfort of the Dan Berlin
participants are paramount dberlin@madpow.net
@banderlin
ā¢āÆ Use quantitative measures and report
percentages of participants that did or Chris Hass
didnāt perform an action chass@madpow.net
@chrishass
ā¢āÆ Itās all about proper planning!
Usability
Ā Tes,ng
Ā Medical
Ā Devices
Ā 32
Ā