This presentation identifies a way to use Risk Management to determine the extent and scope of a validation project, including what validation documents are needed and what should be tested. One validation size does not fit all validation projects! Using the Quality/Regulatory Risk and Functionality/Distribution Risk identifies an Overall System Risk. The Overall System Risk and the Type of System Change determine the needed Validation documents. A methodology to identify the extent of validation test scripts is discussed too.

1. How Much Validation is Enough?
The Breadth and the Depth
Robert Sturm
rmsturm@hotmail.com

2. Estimating the Dilbert Way (1 of 2)

3. Estimating the Dilbert Way (2 of 2)

4. Agenda
• Brief Overview of Validation and Verification
• Do we have to Validate the System?
• Identify Overall System Risk
• Identify Type of System Change
• Identify Needed Validation Documents
• Identify Extent of Test Scripts
• Workshop

5. DISCLAIMER
• Everything in this presentation
represents my opinion and not
that of my employer.
• It’s all about RISK!

6. Validation and Verification
• Verify each part: Test and document results
• Validation = ∑ verified parts
It’s the entire process!
VERIFY

7. Potential Validation Documents
(GAMP 5)
Validation Plan Validation Summary
Data Migration (DM)
User Risk Analysis
Requirements PQ Protocol/ Scripts
Functional Specs OQ Protocol/ Scripts
Design specs IQ Protocol/ Scripts
Trace Matrix

8. Key Validation Deliverables
• Number of validation documents
– How many?
vs.
• Extent of the test scripts
– What and how much to test?
vs.

9. How Much? FDA: : Use Risk

10. How Much? QA vs. IT
VALIDATION
IT &
QA
CUSTOMER

11. Validation Estimating Workflow
1 2 3 4
Do we Determine Determine Determine
have to Quality & Functionality Overall
validate? Regulatory & Distribution System
Risk Risk Risk
8 7 6 5
Identify Identify Determine Determine
extent of Validation Document Type of
test scripts Documents Level System
Change
START

12. 1 Do We Have to Validate?
• Is this a GxP system - GLP, GCP or GMP?
– Laboratory, Clinical or Manufacturing
• Does the system affect Patient Safety,
Product Quality or Data Integrity?

13. 2 What is the Quality & Regulatory Risk?
• Evaluate the criticality of the data processed by the
system as Low, Medium or HIGH risk
• For each risk category identify examples of GMP, GCP
and GLP records in the system
– HIGH: GMP (Master records, batch records, etc.)
– Medium: GCP (Clinical trial patient data)
– Low: GLP (Training records, facilities records, etc.)
• Use the highest identified risk level

14. 3 What is the Functionality Risk?
• Assign values based on functions the system
performs
– Electronic data capture 3
– Data entry/modification/calculation 3
– Submission creation 2
– Data analysis and reporting 2
– Data/document storage 1
– Data browsing 1

15. 3 What is the Distribution Risk?
• Custom system 5
• Multi-industry, limited use 4
• Regulated industry, limited use 3
• Regulated industry, broad use 2
• Multi-industry, broad use 1

16. 3 What is the Complexity?
• Add Functionality and Distribution Risk
• Complexity Level
HIGH = 7 to 8
Medium = 4 to 6
Low = 2 to 3

17. 4 Determine Overall System Risk
Quality/Regulatory Complexity System Value
HIGH + HIGH = HIGH 10
HIGH + Medium = HIGH 9

18. 4 Determine Overall System Risk
Quality/Regulatory Complexity System Value
HIGH + HIGH = HIGH 10
HIGH + Medium = HIGH 9
HIGH + Low = HIGH 8
Medium + HIGH = Medium 7

19. 4 Determine Overall System Risk
Quality/Regulatory Complexity System Value
HIGH + HIGH = HIGH 10
HIGH + Medium = HIGH 9
HIGH + Low = HIGH 8
Medium + HIGH = Medium 7
Medium + Medium = Medium 6
Medium + Low = Medium 5
Low + HIGH = Medium 4

20. 4 Determine Overall System Risk
Quality/Regulatory Complexity System Value
HIGH + HIGH = HIGH 10
HIGH + Medium = HIGH 9
HIGH + Low = HIGH 8
Medium + HIGH = Medium 7
Medium + Medium = Medium 6
Medium + Low = Medium 5
Low + HIGH = Medium 4
Low + Medium = Low 3
Low + Low = Low 2

21. HIGH
Drug Clin
Safety Data Elec Data
MedDRA
WHO Mgt Cap
Drug SAS ERP
Drug
Clin Stability Doc Submission
Supplies Publishing
Software
Upgrade Mgt
MEDIUM
Hardware/
R Drug Network
Clin Trial
Coding Upgrade
I Mgt
S Excel
K
Scanner
Low
Low MEDIUM HIGH
COMPLEXITY

22. HIGH
Drug Clin
Safety Data Elec Data
MedDRA
WHO Mgt Cap
Drug SAS ERP
Drug
Clin Stability Doc Submission
Supplies Publishing
Software
Upgrade Mgt
MEDIUM
Hardware/
R Drug Network
Clin Trial
Coding Upgrade
I Mgt
S Excel
K
Scanner
Low
Low MEDIUM HIGH
COMPLEXITY

23. H-l H-m H-H
HIGH
Drug Clin
Safety Data Elec Data
MedDRA
m-H WHO Mgt Cap
Drug SAS ERP
Drug
m-m Clin Stability Doc Submission
Supplies Publishing
Software
Upgrade Mgt
MEDIUM
m-l Hardware/
R Drug Network
Clin Trial Mgt
Coding Upgrade
I l-H
S l-m Excel
K
l-l Scanner
Low
Low MEDIUM HIGH
COMPLEXITY

24. What do we know at this point?
• We know the overall system risk level
–H - H, H - M, H - L, M - H, M - M, ….
L – M, L – L
• We have a value for the overall system risk
level
Are we there yet?

25. 5 What Kind of Change is Happening?
• Release (Major) – x.0: Adding functionality;
– Change value of 1.0
• Upgrade (Minor) – x.y: Changing functionality;
– Change value of 0.8
• Patch – x.y.z: Not changing functionality;
– Change value of 0.5
• Hot Fix – x.y.z.w: Smell for smoke;
– Change value of 0.2

26. 6 Determine Overall Document Level
System Risk Value Release Upgrade Patch Hot Fix
Change value: 1.0 0.8 0.5 0.2
H - H
H - M
H - L
M - H
M - M
M - L
L - H
L - M
L - L

27. 6 Determine Overall Document Level
System Risk Value Release Upgrade Patch Hot Fix
Change value: 1.0 0.8 0.5 0.2
H - H 10 10.0 8.0 5.0 2.0
H - M 9 9.0 7.2 4.5 1.8
H - L
M - H
M - M
M - L
L - H
L - M
L - L

28. 6 Determine Overall Document Level
System Risk Value Release Upgrade Patch Hot Fix
Change value: 1.0 0.8 0.5 0.2
H - H 10 10.0 8.0 5.0 2.0
H - M 9 9.0 7.2 4.5 1.8
H - L 8 8.0 6.4 4.0 1.6
M - H 7 7.0 5.6 3.5 1.4
M - M
M - L
L - H
L - M
L - L

29. 6 Determine Overall Document Level
System Risk Value Release Upgrade Patch Hot Fix
Change value: 1.0 0.8 0.5 0.2
H - H 10 10.0 8.0 5.0 2.0
H - M 9 9.0 7.2 4.5 1.8
H - L 8 8.0 6.4 4.0 1.6
M - H 7 7.0 5.6 3.5 1.4
M - M 6 6.0 4.8 3.0 1.2
M - L 5 5.0 4.0 2.5 1.0
L - H 4 4.0 3.2 2.0 0.8
L - M 3 3.0 2.4 1.5 0.6
L - L 2 2.0 1.6 1.0 0.4

30. 7 Identify Validation Documents
Document Value 10 - 9 8-7 6-4 3-0
Validation Plan I I I C O
User Requirements I C C C C O
Risk Analysis I C C C O
Functional Specs I C C O O O
Design specs I C O O O
Trace matrix I C C C O
I = Individual, C = Combined, O = Optional
IT’S A GREY AREA!

31. 7 Identify Validation Documents
Document Value 10 - 9 8-7 6-4 3-0
IQ Protocol/Scripts I C I C C C O
OQ Protocol/Scripts I C I C C C O
PQ Protocol/Scripts I C I C C C O
DM Protocol/Scripts I I C C C O
Validation Summary I I I C O
Production Release I O I O I O O
Memo
I = Individual, C = Combined, O = Optional

32. Now where are we?
• We identified what validation documents are
needed for the project
• We have QA buy-in
Validation documents

33. 8 Identify Extent of Test Scripts
• Determine level of testing
• Apply a Functional Risk Assessment to all user
requirements
• Prioritize requirements as HIGH (H),
Medium (M) or Low (L)
• Assess requirements as regulatory/ business
risk Critical (C) or Not critical (N)
• High and Critical are the highest risk

34. 8 Prioritize Requirements (Reqmnt)
• HIGH - H (Mandatory): Reqmnt must be present for
the system to operate
• Medium - M (Desirable to have): Reqmnt need not
be present for the system to operate but it is a ‘nice
to have” feature
• Low - L (Low impact/may be useful): Reqmnt need
not be present for the system to operate, may be
useful and has low impact to the prime user
department
• Null - N (Not used): Reqmnt will not be used and will
not be validated

35. 8 Prioritize Risk of the Requirement
• Critical (C): If this Reqmnt is not done there is
a risk of non-compliance. Reqmnt is critical
for business reasons (data are correct, system
is Web available, data entry and output are
correct, etc.)
• Not critical (N): Reqmnt has no regulatory or
business risk associated with it

36. 8 To Test or Not to Test?
Priority Risk Test (Y/N)
(H/M/L) (C/N)
H C Y
M C Y
L C Y
-- -- --
H N Y
M N N
L N N
N N N

37. Assessing Changes to a Validated
System for Company XYZ
Description Priority Risk Test
(H/M/L) (C/N) (Y/N)
Changes to the safety database schema H C Y
and tables
Ability to capture partial dates M N N
Able to print reports for FDA submissions H C Y
on a representation of the revised
MedWatch form
Customer fixes for expedited reports M N N
Lab test name encoding from Case Form L N N
Electronic Submission: Updated feature -- N N
not used

38. Interactive Exercise
• Our vendor sends us a minor upgrade to our
Drug Safety system
• It affects our existing requirements
• What validation documents are needed?
• What should be tested?

39. The Next Step Is Yours
OR

40. Bonus Information: References
• “General Principles of Software Validation, Final Guidance for
Industry and FDA Staff”, FDA, Jan 11, 2002. www.fda.gov
• Annex 11 Computerised Systems
• Computer Validation: The 100 Worst Mistakes You Can Make,
Tamara Follett, 2003
• The Computer System Risk Management and Validation Life Cycle, R.
Timothy Stein, 2006 (many references)
• GAMP 5 A Risk-Based Approach to Compliant GxP Computerized
Systems, ISPE, 2008
• “GIP Guidance Module 1: Validation and Verification”, HIMSS, 2011,
himss.learn.com/learncenter.asp?id=178409&page=184
• Google “Software Compliance Science John Murray FDA” for a PDF
file: www.softwarecpr/.../download.asp

41. Bonus Information: References
• “RAMP: An Approach to Risk-Based Computer System Validation
and Part 11 Compliance”, Drug Information Journal, Vol 41, pp 69-
79, 2007 (Use to estimate Quality, Regulatory, Functionality and
Distribution Risk)
• “Validation of Software for Regulated Processes”, Assoc. for
Advancement of Medical Instrumentation, 13-Dec 2007
• “Effective and Practical Risk Management Options for Computerised
System Validation,” R. D. McDowall, Quality Assurance Journal, Vol
9, Issue 3, 2005
• IT Pharma Validation Europe: http://www.ccs-inspired.com
• Ask About Validation: http://www.askaboutvalidation.com
• Risk Doctor Network: http://www.risk-doctor.com
• Compliance Webinars: http://www.complianceonline.com/

42. Four Assessment Questions
• Does the system automate a process or
manage date regulated by these regulations?
– GMP (21CFR 210 and 211)
– GCP (21 CFR 312)
– GLP (21 CFR 58)
– PDMA (21 CFR 203)
• If yes to any of the above, it is GxP

43. Bonus Information: Validation Life Cycle
Folder Organization
1 Admin
IT Computer System Validation and Document Flow: Key Validation Documents 2 Pre xQ
3 IQ
** EVALUATE ** ** PLAN ** 4 During Xq
2 7 - Configuration & Training
- Major release 1 1 1
- SOPs and WIs
A B K Validation/
- Added functionality Qualification Project Supplier Audit - Migration
Plan 5 OQ
6 PQ
7 Post xQ
8 Post Impl
** REQUIREMENTS ** ** EXECUTION ** ** EXECUTION **
** DEPLOYMENT **
IDENTIFY REQUIREMENTS WRITE PROTOCOL & EXECUTE TESTS SUMMARIZE TEST RESULTS
TEST SCRIPTS WRITE FINAL REPORTS
3 3 3
7
Installation
2 IQ Installation Qualification Installation Validation/
2 Qualification
User 2 Protocol & Test Scripts Qualification Test Summary Report Qualification
D (Validation Env.) Execution
Requirement C (Val Env.) Summary Report
Specifications (Val Env.)
2 7
4 4
Functional 2 4 4 4
E F G F G 5
Specifications Training Records
Design Operational
5
Specifications 5 Qualification
Summary Report
Operational Operational 7
Qualification Protocol Qualification Test
& Test Scripts Execution Support Model
6
Performance
2 6
4 Qualification
4
Performance Summary Report 7
Requirements Trace H I
Matrix Qualification Test Production Release
6 Execution memo with
7 Workarounds, if any
Performance
Qualification Protocol 7 J
& Test Scripts Pre
J
3 ** MAINTENANCE **
3
IQ 3 IQ Summary Post Implementation
IQ Protocol & IQ Execution Report
(Prod. Env.) (Prod Env.) 8
Test Scripts
Optional: C-Risk Analysis, D-Training Plan,
D-Training Plan, Final WIs/ SOPs
(Production Env.)
E-Detailed Design Specs., F-Configuration Spec.,
F-Configuration Spec., 4 7 7
4
F F J Post J
G-Training & Manuals, H-Migration Qualification Protocol,
G-Training & Manuals, H-Data Migration Protocol, 8
CHANGE
I-Draft WIs/ SOPs, J-Data Migration Report Report
J-Migration Qualification CONTROL
Administrative: A-GxP Assessment, B-IT Statement of Work,
K-MS Project Plan
** RETIREMENT **