Sarat Addanki's Presentation at the UK TestExpo 2014.

1. Testing as a Managed Service Using
SLA’s and KPI’s
Sarat Addanki
Head of Practice
CONNECT WITH US:

2. Corporate overview
03
CONNECT WITH US:
Agenda
01
04
2 Public | Copyright © 2014 Prolifics
06
05
02
SLAs
KPIs
Sample SLA
reporting
Q&A
Demo

3. Best-in-class architects and
specialty experts:
BPM, IIB, DataPower, WSRR,
and other IBM products and
tools.
CONNECT WITH US: 3
Prolifics at a Glance
5-Years Compound
Annual Growth Rate
19%
Employees
Worldwide
1,500
Global Presence
Years in
Business
35+
United States, United Kingdom, Germany, India
20+
Technology
Accelerators
550+
Technical
Certifications
Over 10 Technology and
Solutions Awards since 2009
including Business Agility,
Customer Integration and Digital
Experience
Over 150 global customers
are currently Fortune
1000 companies
96%
industry-high 96% success
rate of deploying robust
solutions.
Offices
14
Awards
IBM Technology Expertise
Fortune 1000
91%
Rate of Repeat Engagements
Public | Copyright © 2014 Prolifics

4. CONNECT WITH US:
Our Services
Public | Copyright © 2014 Prolifics 4
• Application
Development,
Maintenance and
Support
• Outsourced Product
Development
• Application
Modernization
• Portal Content &
Collaboration
• Test Advisory
• Functional Testing
• Packaged
Apps
• Custom Apps
• Cloud
• Test Automation
• Middleware and
SOA
• Performance Engg
• Mobile Testing
BPM & Connectivity
User Experience
Security
Application Performance
Management
Enterprise Content
Management
Business/Data Analytic

5. CONNECT WITH US:
How Prolifics Can Help You
5
 Adapt faster than the
competition with continuous
process improvement
 Transform your applications
to increase business agility
 Create best-in-class
customer and employee
experiences. Become more
social and collaborative
 Gain actionable insight
with an intelligent approach
to security
BUSINESS PROCESS
MANAGEMENT
SOA INTEGRATION USER EXPERIENCE SECURITY
 Drive business growth with
proactive IT management
 Maximize IT investments
with end-to-end testing
 Optimize business
performance with smarter,
faster decisions
 Increase efficiency within
your organization and
achieve “straight through
processing”
APPLICATION
INFRASTRUCTURE
TEST ADVISORY,
APPLICATION TESTING &
TEST AUTOMATION
BUSINESS ANALYTICS
OPERATIONAL DECISION
MANAGEMENT (ODM)
 Simplify employee and
customer access to the
documents they need when
they need them
 Get the most out of your IT
to create competitive
advantage.
 Leverage global expertise
to support your IT
environment
 Optimize your software
spend to achieve business
value
ENTERPRISE CONTENT
MANAGEMENT
ADVISORY SERVICES MANAGED SERVICES SOFTWARE
Public | Copyright © 2014 Prolifics

6. Technology Government
CONNECT WITH US: 6
Prolifics Customers
Telecommunications Insurance
HealthCare
Healthcare
6
Financial Services
Utilities
Retail & Distribution Media & Entertainment
Public | Copyright © 2014 Prolifics

7. Evaluate value
Measure quality
Why Managed Service and SLAs?
Measure
productivity
Incident DRE
Mgmt.
Companies are moving from Outsourcing Model to Managed Services Model as it allows them to govern the vendor
services with the help of mutually decided upon Service Level Agreements (SLA’s).
CONNECT WITH US: 7
Continues improvement
Determine cost of
testing
Create
accountability
Process
Compliance
Within
Budget
% of
Automation
Business
Process
Knowledge
Test Design
Efficiency
Manage vendor
SLA’s are vital signs.
You can tell where a
testing organization is
headed and also
measure it’s maturity
level

8. CONNECT WITH US:
Key Performance Indicators (KPI’s)
8
Test case effectiveness
Test progress plan vs actual
Cost of testing as a % development
Skill set matrix
Quality of development
No. of regression test cases updated per release
Quantifiable regression execution

9. SLAs and Measurement Intervals
Monthly
Quarterly
Quarterly
Quarterly
Monthly
Monthly
CONNECT WITH US:
Defect Removal Effectiveness
Process Compliance
Business Process Knowledge
General Incident Management
Automated Regression library - Execution
Regression Library - Updates
1234
56
78
9
10
11
12
13
14
15
16
17
Offshore Technical Skillset
Effort per Defect
Savings thru Defects
Through by Project types
Resource Utilization
Submission of SLA Report
Review of RCA
Test Design Efficiency
Delivery within budget
Sizing and Estimation Turnaround
Metrics Accuracy
Monthly
Monthly
Monthly
Monthly
Monthly
Monthly
Monthly
Monthly
Quarterly
Quarterly
Quarterly
S. No
SL
A
Non SLA
Metrics
Measurement
Interval
9

10. DRE Process Compliance Business Process Knowledge
CONNECT WITH US:
SLA Target,
95%
General Incident Mgmt. Automated Regression
Library - Execution
SLA Target,
100%
Regression Library - Updates
100%
98%
100%
91%
100%
100%
Snapshot of SLA Metrics
June, 2014
10
SLA Target,
100%
SLA Target,
98%
SLA Target,
90%
SLA Target,
98%

11. Description Formula
Target Frequency
CONNECT WITH US:
Measure of the quality delivered in a
given engagement by measuring the
number of defects that seeped into
production
Performance = Number of non rejected
defects found prior to production / (total
number of non rejected defects
registered during development life cycle
from requirements phase up to and
including warranty period) * 100
Critical and High - 95%
Medium – 90 %
Low – 85 %
Monthly
Defect Removal
Effectiveness
11

12. Process Compliance Description
A quarterly audit to measure vendor’s
compliance with SV&T key process
areas
Formula
Significant deficiency score = Sum of % non-compliance
of critical process
activities/number of audit assessments
Compliance score = Sum of % compliance of
key process activities/number of audit
assessments
Measurement
Interval
Monthly
Target
<2% significant deficiencies on critical process activities
≥90% compliance on key process activities
CONNECT WITH US:
12

13. 13
Quarterly assessment to measure service provider’s
staff proficiency in McKesson business process
knowledge and knowledge management.
Performance = Count of number of people
rated at different expertise levels
(Advanced, Proficient) segmented by
McKesson business process
Monthly
CONNECT WITH US:
15
Business Process (“BP”) Knowledge
Description
Formula
Measurement
Interval
Target
For each BP rated as Critical or High:
1. A minimum of 2 resources rated at advanced
2. A minimum of 2 rated as proficient
For each BP rated as Medium or Low:
1. A minimum of 1 resources rated at advanced level
2. A minimum of 1 rated as proficient level

14. CONNECT WITH US:
General Incident
Management
Description
A quarterly assessment of incident closure
rates within the prescribed response time
Formula
14
Measurement Interval
Target
Performance = Number of occurrences where
incident is closed on or before the expected
closure date/Number of total occurrences –
Segmented by severity
Quarterly
High Severity Issue - Respond within 2 hours (from the
time of notification). Resolve within 8 hours.
Medium Severity Issue - Respond within 4 hours (from
the time of notification). Resolve within 24 hours.
Low Severity Issue - Respond within 8 hours (from the
time of notification). Resolve within 5 business days
unless both parties agree to incorporate the resolution in
the next scheduled release (not to exceed 90 days)

15. Description
Formula
Measurement
Interval
Target
CONNECT WITH US:
Automated Regression Test Library
2 week rolling cycle for periodic
runs and on demand cycle
based on specific request
Performance = Actual test
completion date ≤ expected
completion date
Quarterly
15
>=90% Successful execution of Test Automated
Regression test Cases
<=10% Successful execution of Test automated
regression test cases through manual approach in
case of Data setup issues.

16. 16
Offshore BI/BW Skill set Advanced Proficient
CONNECT WITH US:
Offshore Technical Skillset 1
Offshore SAP Technical Skill set Advanced Proficient Basic
12
0
1
3
10
2
3
0
1
0 0
1
2
6
2
0 0
23
5
3
1
6
0 0 0 0
11
0 0 0
12
4
2
6
11
2
1 1
0
25
20
15
10
5
0
1
4
2
1
2
1
2 2 2
1
2
3
2
1
3
5
4
3
2
1
0
SAP BO SAP BW Datastage SAP BODS SAP HANA

17. Offshore HRIS team Skillset Advanced Proficient
Offshore Bench Strength
CONNECT WITH US:
Offshore Technical Skillset 2
1
Offshore Vistex Skill set
Advanced Proficient Basic
2
2
3
1 1
3
0
0
2
Offshore Legacy & Automation Skill
1
0
Advanced Proficient Basic
1 1
3
3
0
2
3
1
0
2
1
0
set
1
2
0 0
3
4
3
2
1
4
3
2
1
0
0
Acumax FSCM SCORE TANDEM QTP WMI
0
VISTEX
CHARGEBACKS
VISTEX CONTRACTS VISTEX MDM VISTEX REBATES
1
2
2
2
0 1 2
SAP ABAP
5
SAP BI/BW, BO
SAP SD
SAP FI
No. of Resources
17
2
1
0
2 2
1
2 2
0
0
Peoplesoft Manual Testing ETL Testing

18. CONNECT WITH US:
Effort per Defect & DRE
FY13YTD, FY14YTD & April ’14, May ‘14
18
DRE Effort per Defect
99% 99.30%
98% 98%
50
39
43
63
70
60
50
40
30
20
10
0
100%
95%
90%
85%
80%
75%
70%
FY13 YTD FY14 YTD April '14 May '14

19. CONNECT WITH US:
19
Submission of SLA Report
Reporting Period Target Date Publishing Date Published on time
Jan 2014 2/5/2014 2/5/2014 Yes
Feb 2014 3/5/2014 3/3/2014 Yes
Mar 2014 4/5/2014 4/2/2014 Yes
Apr 2014 5/5/2014 5/1/2014 Yes
May 2014 6/5/2014 6/4/2014 Yes

20. CONNECT WITH US:
20
Review of RCA’s
This SLA is included to track RCA’s created vs. percentage of RCA’s reviewed and signed off.
Total of 20 RCA’s are created till May 2014.
5 RCA’s are added in May 2014, two for SO 3163 (One Medium, One Low), one medium defect each for SO
2344, SO 4921, SO 3264.
20 20
20
18
16
14
12
10
8
6
4
2
0
Total RCA's Created Total RCA's Reviewed
No. of RCA's

21. Description Formula
Target Frequency
CONNECT WITH US:
A quarterly assessment to measure
turnaround time for estimation and
sizing of testing initiatives
Performance = number of sizing
requests closed on or before
scheduled closure date/number of
sizing requests submitted – segmented
by project type (large, medium and
small)
Large Projects – 10 Business Days
Small & Medium projects – 5 Business Days
Quarterly
Estimation Turnaround
Time
21

22. Metrics Accuracy
Description Formula
Target Frequency
CONNECT WITH US:
A quarterly assessment of accuracy of
metrics provided by service provider.
McK SV&T will identify ≥30% of
metrics to audit. Prolifics will provide
data used to calculate metric. Prolifics
will provide/ identify source of data for
analysis.
Number of measurement items
reported accurate/Total number of
measurement items reported.
≥95% accuracy of reported
metrics Quarterly
22

23. Performance will be measured on 2 dimensions:
Significant Deviation Measure =( Number of
projects where actual QA cost exceeds
budgeted/forecasted QA cost by 25%/Total
number of projects)*100
Accuracy Measure =( Number of projects where
actual QA cost is within +/- 10% of
budgeted/forecasted QA cost/Total number of
projects)*100
CONNECT WITH US:
Description
Formula
Budget = baseline budget + delta cost.
Baseline budget includes effort and cost
estimated for test lifecycle activities, Delta
cost includes Service Level Credit to cost
and effort caused due to authorized
changes to scope or other project
constraints
.
Delivery within
Budget
Target Frequency
Significant deviation measure =
0% Accuracy measure = >= 95% Quarterly
23

24. CONNECT WITH US:
Delivery within Budget
No specific data point for this, based on the comments from
RTM’s.
This metric is difficult to accurately track at this point as the
total budget for Prolifics and SV&T employees is combined
SVTCs and RTMs do the estimates in most
cases
In BI space, (which is completely outsourced to Prolifics) all
SO’s have been delivered with in the budget
TARGET ACTUAL (YTD)
95% 100%
24