TL 9000 Measurements and Requirements Interactive Workshop

Copyright 2009 QuEST Forum. All Rights Reserved.
1
1
TL 9000 Measurements and Requirements
Interactive Workshop
APAC Best Practices Conference
Tokyo, Japan
16 April, 2015
Ken Koffman – JDSU
John Wronka – Alcatel-Lucent

Topics for Today
• Questions were provided in advance in
the following Areas:
– General
– QuEST Forum Membership
– TL 9000 Certification
– TL 9000 “Adder” Applications
– Measurements
– Data Submissions
• Time Permitting:
– PDR Strategic Initiative
– Member Value Survey 2

“GENERAL” QUESTIONS
Copyright 2009 QuEST Forum. All Rights ReservedCopyright
2009 QuEST Forum. All Rights Reserved.
3

Acquiring TL 9000 Certification
• Tell me about how my business can
become TL 9000 certified
• How long does certification take?
• How many personnel are required?
4

Copyright 2012 QuEST Forum. All rights reserved. 5
TL 9000 Implementation Path
6. Continual
Improvement
5. External
Assessment
4. Assess and review
3. Process
Improvement
1.Management
Commitment
• Understand TL 9000 and Customer Needs,
Set Goals, Commit Resources
• Core Team Selection and Training, Gap
Assessment, Project Planning
• Process Mapping, Data Collection and
Analysis, Process Changes to Meet New
Requirements
• Internal Audit, Management Review,
Corrective and Preventive Action
• Certification Body Pre-
Assessment
and Registration Audit
Measurements
Reporting – Often a
gating item. Plan at
least 9 months to
prepare and submit.
Start
Suggested
Implementation
Time Line
4-6 Months
3-5 Months
Ongoing
2. Organizational
Preparation
2-3 Months
1-2 Months

6
I would like to have realistic examples of
measures to meet TL 9000 Requirements:
Access Examples Utilizing “Links” ( tl9000.org/links.html).
Use “Contact Us” to Suggest New Examples for IGQ Examples Subteam
Scroll Down

What Kinds of Questions Does QuEST Forum
Get on TL 9000 Measurements and Trends?
7
http://tl9000.org/alerts/qa/overview.html

General Questions –
ISO 9001:2015 and TL 9000
Requirements R6
• The requirements for documentation and
records have been reduced version by version
in ISO 9001 QMS. On the other hand, in the TL
9000 system which include ISO 9001,
documentation is mandatory for many
requirements. In the future, will TL 9000 be
aligned to ISO 9001:2015 in which the
documentation requirements are supposed to
be much more reduced?
8

ISO 9001:2015
Status Update
9
CD available
2013
DIS available
05/2014
FDIS expected
AFTER 07/2015
ISO 9001:2015 to
be published
09/2015
3-Year Transition
Proposed
ISOStandardsDevelopmentProcess
IAF Informative Document - Transition Planning
Guidance for ISO 9001:2015
Published
TC 176 SC 2/WG 23 Communication &
Product Support – Ashok Dandekar
(QuEST Forum Representative)
International ballot on the DIS closed on Oct. 10, 2014 –
Results were APPROVE DIS by 90%
ISO/NP TS 9002 “Guidelines for the
application of ISO 9001” may be
published in 2015

QuEST Forum ISO 9001:2015 Position

What actions have QuEST Forum
taken regarding ISO 9001:2015?
Solicited, reviewed and submitted
comments on ISO 9001:2015 CD
QuEST Forum agreed continued
alignment to ISO 9001
QuEST Forum Academy ISO
9001:2015 Webinar by Nigel Croft
Solicited, reviewed and submitted
comments on ISO 9001:2015 DIS
Mapped R5.5 TL 9000 Adders to ISO
9001:2015 DIS clauses (now 10!)
08/2013
06/2014
07/2014
07/2014
11/2014
May
REPEAT
upon
release of
the FDIS!

“Key” Detailed Changes:
• New version of Figure 1; ISO 9001 still follows PDCA
• “documented information” remains
• “procedures” and “records” are not mentioned instead “maintain
documented information” and “retain documented information”
• A quality manual and a management representative are no longer
required
• “products and services” replaces products
• ISO 9000 will be updated to address changes in terms & definitions
• “applicability” replaces “exclusions”
• risks are to be determined and addressed, but there is no requirement
for formal risk management or risk approach
• New/different requirements from ISO 9001:2008 include organizational
context and knowledge management
• Annex A to provide general guidance on implementation and other
Annexes will be included such as a summarized analysis of key
enhancements between ISO 9001:2008 and ISO 9001:2015
13
ISO 9001:2015 Revision Overview

General Questions –
Third Party Process
• How does QuEST Forum take
action with all certification
bodies in order to have a
uniform level of TL 9000
certification audits?
• Or, is there no level difference of
the audit among all certification
bodies in the world?
14

ABCB Sub-Team Review:
Overview of Sub-Team Charter
• CB Statistics
• AB Approval / Re-approval
• Code of Practice
– Including TL 9000 Auditor
Qualification and Experience
Requirements for CBs
• Validation Audits
Copyright 2011 QuEST Forum. All Rights Reserved
16

Validation Audits
• Plan: 2008-9
– Decision to investigate the effectiveness of
CB audits in identifying major non-
conformities
• Do: 2010-11
– ABs conducted validation audits at randomly
selected sites
• Check: 2012-13
– ABCB Sub-team analyzed data and
developed recommendations
Copyright 2011
17

Action Thresholds
Copyright 2009 QuEST Forum. All Rights Reserved Copyright
18
90%
Failures
Certs=800 Sites
Problem
Sites High Midpoint Low
Site=1700 12 0 22.1% 11.0% 0.0% 3rd Party Process Function Well
1 32.7% 16.5% 0.4% Individual Issue, 3rd Party Process OK
2 40.8% 21.9% 2.9% Investigation Required
3 47.6% 27.1% 6.6% 3rd Party Process Review Required
4 53.4% 32.1% 10.8%
5 58.4% 36.8% 15.1%
6 62.7% 41.1% 19.6%
7 66.6% 45.3% 23.9%
8 70.0% 49.1% 28.2%
9 73.0% 52.7% 32.4%
10 75.7% 56.0% 36.4%
QuEST Forum Validation Audit
Confidence Interval
Sites
Audited
Problemsite count
fromVA Audits
Problemsite
estrapolationestimates
5 of 12 audits Failed:
Team determines there
is a high risk of CB
audits failing to detect
some major
non-conformities to
TL 9000

Validation Audits
• Act: 2014-15 – Approved
1. Code of Practice Changes
• Additional qualifications for new auditors
• Additional continuing education requirements
• CBs to identify and respond to soft grading
• CBs to collect additional pre-audit information from sites
• On site audits required for installation processes
2. QF to ensure validity of product category selections
3. Optional checklist for Measurements auditing
4. Optional webinars
– Auditor Rotation – REJECTED by OSWG Vote
19

Validation Audits
• Act: 2014-15 – Proposals Under Discussion
1. Upgrade CB Technical Review of audit reports
2. Prepare a list of optional questions for CB auditors
3. MHB Checklist improvements
4. Anonymous feedback on CBs
5. Library of recorded webinars
6. Provide additional training for CB auditors
7. Reconsider AB oversight requirements
8. Perform further validation audits
20

General Section – Other Questions
• What is the difference between "Tier 1 Supplier" and "Tier 2
Supplier"?
– It depends!!!
• Does QuEST Forum have any plans to set standard objectives for
each measurement based on PDR Industry Statistics?
– No
• Thoughts on estimating future measurement trends based data
collected from all organizations. What is the most effective way?
– Depends on maturity level, data systems in place, connection
with improvement initiatives
• What part of the TL 9000 Adders should be emphasized or
enhanced in the future?
– Some of the newer “adders” such as security, sustainability,
risk management, supplier performance management
21

QUEST FORUM MEMBERSHIP
QUESTIONS
22

TL 9000 Certification & QuEST
Forum Membership Questions
• TL 9000 Certification does not require
Forum membership
• QuEST Forum membership does not
require TL 9000 Certification
23

QuEST Forum Membership –
Other Questions
• Can an Operator Receive IA/BIC/WIC for their
suppliers and the suppliers of other operators?
– Yes
• Can a Forum member or TL 9000 certified supplier
obtain industry statistics solely for the carrier which
it supplies products?
– No
• Does being a QuEST Forum member reduce the
personnel required to be TL 9000 certified?
– Not directly, but QuEST Forum membership provides access
to SMEs and best practices.
24

TL 9000 CERTIFICATION
QUESTIONS
25

TL 9000 Certification –
Audit Days
26
I recognize that TL 9000 certification audit is conducted
in conjunction with ISO 9001:
• How long is the additional audit time for TL 9000 specific part?

Is there a case where a service provider
requires its suppliers to require their
suppliers to meet TL 9000?
• Yes! - See Examples
– “The Supplier shall determine which of its subcontractors and vendors
(Quality Parties) should be registered to TL 9000 and ensure that each of
those timely registers to TL 9000”.
– “In the event Seller uses a contract manufacturer or subcontractor to
fulfill the terms of the Agreement, Seller shall….. Commit to contract
manufacturer or subcontractor registration to TL 9000 Quality
Management System Standard”
• 7.4.1.C.2 Supplier Performance Management –
– requires identified “key suppliers” to align towards TL 9000 OR
other appropriate QMS
– But does stops short of mandating TL 9000 for suppliers
27

TL 9000 Certification
- Scope Questions
• The clause 3.2 b) in the Requirement Handbook requires in the Scope
Statement that if not all products/product lines are included in the registration,
then either the included or excluded products/product lines shall be listed. I
would like to suggest for the list to be included in the Registration Profile.
– Suggestion has been discussed but becomes difficult for large
certifications
• Is it possible to limit the scope of certification to the products for a specific
customer who requires us to be TL 9000 registered?
– No, but you are required to submit only data for customers where you
have confidence in the data.
• Once an organization gets TL 9000 certified, must the organization supply all
products that meet TL 9000 requirement to customers? Or, can the
organization decide which products should meet TL 9000 requirement s or
not. (For an example, is it possible for product A to meet TL 9000 and Product
B to meet only ISO 9001)
– Yes-you can limit the scope of your products or organizations and
activities to either TL or ISO.Copyright 2009 QuEST Forum. All Rights ReservedCopyright
28

TL 9000 Certification
- Other Questions
• An organization wants to limit its scope of registration to products for one
customer (a service provider). But, because the customer is new, it has not yet
supplied any products to the customer. In this case, is it possible to submit
preceding three month data as all "0"?
The organization has been supplying similar products to the other customer.
But the customer is out of scope, so the organization thinks it cannot use the
latter customer's data because it's out of scope.
- Data must be submitted before you can be TL 9000 certified
• Can we participate in the global telecommunications industry if we are not TL
9000 certified?
– Yes of course, but…
– Customer requirement’s for TL 9000 must also be considered
29

APPLICATION OF TL 9000
ADDERS
QUESTIONS
30

Inputs to TL 9000 from Other
Standards
• What part of the items in TL 9000
Requirements and Measurements reflect
the standards in CMMI and Bellcore?
31

TL 9000 is based on ISO 9001
ISO
9001
Telcordia
(Bellcore)
ISO/IEC
12207
(Software)
ISO
9004
TL 9000
Later Versions Incorporated Concepts
From CMMI, Automotive and ITIL

33
CMMI and TL 9000
Full TL 9000 Certification Approximates CMMI Level 3. Standards
Comparisons including CMMI can be found here .

7.1.C.2 Disaster Recovery
• We have Business Continuity Plan for disaster
recovery planning. Could it be applied for this
requirement? The head office is responsible for the
Plan as a whole. Should the responsible function in the
head office be audited for investigation? The head
office will be out of scope for the certification at this
time. So, we would like to exclude this requirement
from the certification. Is it possible?
• Answer:
– BPC is a major part of disaster recovery
– Yes it should be audited
– This requirement can’t be excluded
34

7.1.C.3 Product Security
• As we are merely producing and supplying the
products based on the specification which the
Telecom Service Provider requires, the responsibility
for the product security should be on the Provider. So
we would like to exclude this requirement. Is it
possible?
• Answer:
– In this case, if the supplier is doing any form of development
work, even if to the service provider specification, this
requirement can’t be excluded.
– If there is no design work, and for example providing contract
manufacturing (product category 8 ) – it could be excluded
35

7.3.1HS.1 Migration Planning
• As we do not intentionally change the OS of the
product and just follow the customer requested
specification, and the network operation is completely
under control of the customer (Telecom Service
Provider), so we do not have relevant activities of
migration.
• Answer:
– If your company does any part of the design and the
development, then migration will be required, and this
requirement can’t be excluded.
36

7.5.5.S.1 Software Virus Protection
• As we merely produce/supply customer (service
provider) specified products, we take it for
granted that the responsibility for virus
protection is on the customer. Therefore, we
would like to exclude this requirement if
possible.
• Answer:
– This requirement is difficult to exclude if there is any
type of SW in the product. For example if you install any
of the SW or have to replicate it, this requirement would
apply. In these cases, procedures to prevent infections
would be required.
37

8.2.4.H.1 Periodic Retesting
8.2.4.H.2 Content of Testing
8.2.4.H.3 Frequency of Testing
• Although our production plant is out of
the certification scope, our quality control
department takes care of the product
quality. So we recognize that we should
apply these requirements into our QMS.
(This is just a confirmation.)
• Answer:
– Correct!
38

MEASUREMENT
QUESTIONS
39

• What if the "units shipped" period does not actually
encompass the past full 12 months, because of a
newly introduced product which has become
general availability only within a few months ? And
also if the BRR basis shipping period does not
reach 18 months for a new product?
• Answer:
– In both cases, report only the data for the
months available up to and including the 12
month period
40
TL Measurements Questions –
Partial Shipping Period

• As per the product category 6.2.1.2.4 Wireless Terminal Software Applications,
is it not necessary to register this product category for a company who
designs and manufactures complex wireless user terminals and also designs
& installs basic control software for the terminals when shipping the terminals,
but does not design nor supply application software after shipment?
• Answer:
– Based on Product Category 5.2, it is correct that a registration would be required,
but based on the question it appears that the proper category would be 6.2.1.2.2
(Smart Phone)
• MHB3.5.2n) says that the organization shall provide its suppliers all necessary
information it possesses to allow those organizations to generate their TL 9000
measurements. In this sentence, do "those organizations" stand for "the
organization's suppliers? I believe YES, otherwise, it will cause a confusion.
• Answer:
– Yes, "those organizations" means "those suppliers".
41
TL Measurements –
Other Questions:

Measurements Questions –
Linked Certifications
• My company and its subsidiary company are now on the way to get TL 9000
certifications in parallel, that means to get different certifications. And they have
already got different TL 9000 ID's for RMS registration. The main function of my
company is design and development of wireless user terminals and the subsidiary
company is the production function of the terminals. Some of the measurement data
of these two companies are overlapped. So, the subsidiary company would like to
submit Proxy-xxxx's for those measurement data using the TL ID of its parent
company(my company), for the preceding data submissions(including the three
month data submission) before successful completion of the certification. The
certification process of the parent company is still on-going(not completed). Will it
be allowed for the subsidiary to use Proxies during both company are in on-going
certification process? If it is not allowed, there would occur a discrepancy of TL
9000 measurement statistics because of the duplicated data submissions. Please
give me advice or opinion to this situation.
• Answer:
- If they are the same company, registrations can be linked. If there are multiple
registrations where one registration wants to use data from the other registration(s),
they can be linked. See section 4.2.9. The other registration(s) would need to
provide the receiving registration with DSRs that can be viewed during an audit.
This is a useful with common data systems.
42

DATA SUBMISSION
QUESTIONS
43

Data Submission Questions
- Initial Certification
• Do the three consecutive months of data imply measurement
data as defined in the Measurement Handbook must be
submitted to the TL 9000 Measurement Administrator?
– Yes – Data must be submitted to achieve certification
• As for the three consecutive months data, should the data be
submitted month by month?, or submit at once for three
consecutive month?
– Either way is acceptable initially for certification. Going forward
must be monthly
• Shall an organization submit the Data Submission Receipts for
three consecutive month to the certification body before the
initiation of the certification audit?
– CB has access to them once set up in the registration profile
44

- Customer Data
• There is a concern about that we cannot submit data without
the permission of customer because some of the field data are
customer's own.
– There should be no concern. All data submissions are anonymous.
Reports also don’t show submissions.
• If a customer does not supply data for a specific measurement,
can it be exempted?
– You are obligated to report data for all customers where you have
confidence in their data. . If you do not have confidence in the
data from one or more customers you exclude those customers. If
you don't have confidence in data from all customers for a
particular month, you would report 0/0.
• How does an organization, who supply OEM products to the
specific customers, submit the measurement data?
– Report data in the organization’s product categories
45

Repository Systems
- Identities of Companies
- Contact Information
- Certification Body
- Scope of Registration and
Registration Status
- No identities (anonymous
reporting)
- Data segmented by product
category and reporting month
RMS MRS
Registration Management
System (RMS)
PUBLIC information used to manage TL
9000 Registrations
Measurement Repository
System (MRS)
PRIVATE - Database of TL 9000
Registered Organizations DATA

47
Secure Data Submission Process
Secure Network
Internet
Switch
Front-end
Web Server
Registration
Management
System (RMS)
Measurements
Repository
System (MRS)
Network Firewall
Encrypted Data
Processed by
Software with
No Human
Intervention
Raw Data Stored in
Encryption Format
Company Data
Encrypted Submission
HTTPS /
Encrypted Tunnel Connection
5/6/2015
Copyright 2015 QuEST Forum. All Rights
Reserved
47

- Other
• As for the product delivery, the delivery to the end customer is
done through our sales company which is out of the
certification site. In this case, can we regard the sales company
as customer, because we control delivery time for the sales
company？
– Yes
• If, for example, we have data for three models of a product
family, shall we submit the data to sum up all?, or separately?
– Either way. In the end all data is aggregated anyway
• How do you annualize data for the product that does not reach
12 months after the first shipment?
– Monthly and smoothed data is annualized by the Afactor.
48

- Other
• What is the reason why the field data and delivery data for the
products submission is required for the products before
certification is completed? The purpose of measurement data
submission under TL 9000 QMS system should be to monitor
the field data of the product and to improve the QMS by PDCA
cycle. So, this should be after completion of certification. Why
data submission needed before certification?
• Answer:
– The external assessor must verify that processes are in place to
collect, validate and submit the required data.
49

Data Submission Questions -
Advisories
• What is the purpose of advisories?
• What shall an organization do if an advisory is found
in the DSR? Does it need to prepare evidences for
each Advisory to prove if any problem exists or not?
• How are the Advisories defined and how are the
conditions set to issue Advisories?
50

Copyright © 2014 QuEST Forum
Measurements Advisories (page 1)
QuEST Forum strives to continually improve the value of the data provided through TL 9000 Performance
Data Reports, which include industry statistic trends (Industry Average, Monthly Average, Best-in-Class,
Worst-in-Class) as applicable for each product category and TL 9000 measurement. The more accurate
and complete the data, the more useful it is to TL 9000 certified organizations and QuEST Forum
members.
A QuEST Forum initiative to further enhance data entry checking to identify and correct potential data
errors before inclusion in the Measurements Repository System maintained by The University of Texas
at Dallas, was completed in 2009. The result of this initiative was an “Advisor” software application
that notifies TL 9000 data submitters with “advisories” indicating a potential data integrity issue that
should be investigated. The existence of an advisory does not prevent the data from being successfully
submitted. However, Organizations are expected to review any advisories identified and verify
integrity of the submitted data. If errors are found, the Organization’s data is required to be corrected
and resubmitted. Also, during registration and surveillance audits, the Certification Body will be
expected to follow-up with the organization to insure data integrity, as appropriate, for any
measurement flagged with an advisory.

Measurements Advisories (page 2)
The “Advisor” checks look at data across measurements within submissions and across submissions over time to better
validate the integrity of the organization’s submission. Advisories will be highlighted on the summary status of Data
Submission Receipts (DSRs). Each advisory is assigned an integer identifier, and is briefly described at the end of the DSR.
Some advisories may be common to all measurements , while others may apply to certain measurements. After
investigation of a potential problem indicated by an advisory, it is very possible that the organization may find the data is
indeed correct and no resubmission is required. For example, when multiple measurements in the same submission have
the same Normalization Unit (NU) defined in Table A-3 of the Product Category Table, if the measurement submission
includes differing values for the denominators (i.e. NU’s) for those measurements, an advisory will be generated.
However, this difference may be legitimate (e.g. for NPR and SONE where in a particular product category customers
don’t report outage data, but do report problems). In cases where advisory investigation does identify a problem, the
organization will correct it, resubmit and maintain records of this process for review in registration and surveillance
audits.
More detailed information on the Advisories can be found in the paper referenced on page:
http://tl9000.org/alerts/data_submissions.html

Advisory # 1 - The calculated Measurement over the
smoothed period is perfect – no longer applied as of
August 2011
Advisory # 2 - This submission is the same as month,
year
Advisory # 3 - Normalization Units differ across
measurements that should be the same per Table A-2
– no longer applied as of July 2014
Advisory # 4 - Normalization Units changed greater
than 25% from the prior month
Advisory # 5 - Data Element >150% of the highest
value reported over the previous 11 months
Advisory # 6 - Data Element <50% of the lowest value
reported over the previous 11 months
Advisory # 7 - Data Element >125% of the highest
Advisory # 8 - Data Element <75% of the lowest
Advisory # 9 - Not yet Assigned
Advisory # 10 - Data Element >120% of the highest value
Advisory # 11 - Data Element <80% of the lowest value
reported over the previous 11 months and is 3x the
average value in the same 11 months – reduced from 4x
as of August 2011
reported over the previous 11 months and is 3x the
average value in the same 11 months – reduced from 4x
as of August 2011
Advisory # 14 - Normalization Units are the same across
measurements that should be different per Table A-2
Advisory # 15 - “Exempt” has been submitted for a
measurement not listed as exempt on the organization’s
TL 9000 profile - added August 2011
Measurements Advisories
This list of advisories is current as of July 28, 2014
For the current list see DSR Advisories referenced on page: http://tl9000.org/alerts/data_submissions.html

Advisory # 16 – Downtime reported less than minimum expected
Advisory # 17 – Downtime reported less than minimum expected
Advisory # 18 - Outage frequency reported is less than minimum
expected
expected
Advisory # 20 - NPRs should normally be greater than NEOs
Advisory # 21 - NEOs and SOs should normally be equal
Advisory # 22 - Downtown reported less than minimum expected
expected
Advisory # 24 - Input value greater than 95% of maximum value
expected
Measurements Advisories – Added July 2014
This list of advisories is current as of July 28, 2014
For the current list see DSR Advisories referenced on page: http://tl9000.org/alerts/data_submissions.html

What is expected of CB Auditors?
 Adhere to the QuEST Forum Code of Practice for TL 9000 Registrars
 Assure there is a documented system in place to collect, validate and report TL 9000 measurements
 Investigate and validate that the system in place is implemented and effective
 Ensure that the rational for all measurements exceptions are valid
 Verify that Scope and Product Categories are appropriate for the Registration
 Ensure that measurements submissions are made for all product categories under the scope of registration
 Ensure that measurements submissions are made on time
 Understand measurements advisories
 Ensure that all measurements advisories are thoroughly investigated

PDR Improvement
Executive Board
Strategic Initiative

Improve Member Satisfaction with Performance Data
And Performance Data Reports (PDRs).
EB SI #5 - PDR Improvement Project OverviewTasks
• Data Anomaly
Investigations
• Advisors/Error Checks
• Product Category
Verification Process
• Industry Reports
• Normalization Floors
• “How To” White Paper
Updates
• Continued Workshops
At BPC’s
• Regional Webinar
Delivery
• Agreement with EB #3
For Goal/Path Forward
•Identify Holes
•Identify Potential
Registrants
• Identify “Similar” PC’s
• PC Aggregation
Initiative #1
Improve PDR Accuracy,
Create Industry Reports
Initiative #2
Maximize PDR
Usage Value
Initiative #3
“Fill The Pipe”, i.e.
Expand PDR Scope
Survey
Feedback
Initiatives

TL 9000 Industry Reports
58
• Network Field Maintenance Study
• Wireless Infrastructure Product Family Trends
• Then and Now: On-Time Delivery Measurement Study
2009 vs. 2013
• Repair Service Trends Study
• Network Operator Quality Improvement Trends Study
• Quality Improvements in Simple and Complex Wireless
Devices Study
• Edge Router Product Category Trends Study
• Wireless Return Rate Study
• Problem Report Fix Response Time Study
• On-Time Delivery Measurement Study
All Ten Industry Reports can be found on the
QuEST Forum Web Site

Network Field Maintenance Study
59
7.3.1 Product Category
Measurements studied:
• NPR4 (Problem Reports)
• FRT4 (Fix Response Time)
• OFR4 (Overdue Fix Response Time)
• OTI (On Time Service)
• SQ (Service Quality)
• SSO (Support Service Caused Outages)

60
Problem Report Industry Average
Industry Average has Strong Performance
showing an improvement trend
0
0.0005
0.001
0.0015
0.002
0.0025
0.003
NumberProblemReportsperService
Month
7.3.1 NPR4 Industry Average
7.3.1-IA
Linear (7.3.1-IA)

61
Fix Response Time Industry Average
Better than 95% in 2014
Showing an improvement trend
50
55
60
65
70
75
80
85
90
95
100
%FixedOnTime
Month
7.3.1 FRT4 Industry Average
7.3.1-IA
Linear (7.3.1-IA)

62
OFR4 Industry Average
Consistent Performance
50
55
60
65
70
75
80
85
90
95
100
%OverdueFixResponseTime
Month
7.3.1 OFR4 Industry Average
7.3.1-IA
Linear (7.3.1-IA)

63
On Time Service Industry Average
Approaching 100% in 2014
50
55
60
65
70
75
80
85
90
95
100
%OnTimeService
Month
7.3.1 On Time Service Industry Average
7.3.1-IA
Linear (7.3.1-IA)

64
Service Quality Industry Average
Improving to less than 0.5%
0
0.5
1
1.5
2
2.5
%DefectiveServiceTransactions
Month
7.3.1 Service Quality Industry Average
7.3.1-IA
Linear (7.3.1-IA)

65
Support Service Caused Outages Industry Average
SSO Improving in 2014
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
%SupportServiceCausedOutages
Month
7.3.1. Support Service Caused Outages Industry Average
7.3.1-IA
Linear (7.3.1-IA)

Wireless Study
66
Three Wireless Product Categories
• 3.3.2.1-Basic (2G) Base Transceiver Systems
• 3.3.2.2-Advanced (3G) Base Transceiver Systems
• 3.3.2.3-(4G) LTE Base Transceiver Systems
Measurements studied:
• YRR (One Year Return Rate)
• SPR2 (Major Software Problem Reports)
• FRT2 (Major Fix Response Time)
• OTI (On Time Item Delivery)

67
Wireless Study
50% Decrease in 3G Return Rate
GOOD
0
0.5
1
1.5
2
2.5
%ReturnRate
Month
YRR Industry Average
3.3.2.1 (2G) 3.3.2.2 (3G) 3.3.2.3 (4G)
YRR Industry Average

68
Wireless Study
2G and 3G at a mature level. 4G Trend Improving
0
0.0001
0.0002
0.0003
0.0004
0.0005
0.0006
0.0007
0.0008
MajorSoftwareProblemReports
PerNetworkElementperYear
Month
SPR2 Industry Average
3.3.2.1 (2G) 3.3.2.2 (3G) 3.3.2.3 (4G)
Major Software Problem Report Industry Average

69
Wireless Study
4G SPR2 IA Improving; 2G and 3G have mature IA
0
0.0001
0.0002
0.0003
0.0004
0.0005
0.0006
0.0007
0.0008
0.0009
MajorSoftwareProblemReports
PerNetworkElementperYear
Month
SPR2 Industry Average Trends
Linear (3.3.2.1 (2G)) Linear (3.3.2.2 (3G)) Linear (3.3.2.3 (4G))
Major Software Problem Report Industry Average

70
Wireless Study
4G Approaching 2G and 3G Major Fix Response Time
GOOD
0
10
20
30
40
50
60
70
80
90
100
%FixResponseTime
Month
FRT2 Industry Average
3.3.2.1 (2G) 3.3.2.2 (3G) 3.3.2.3 (4G)
Major Software Fix Response Time Industry Average

71
Wireless Study
OTI Trend lines at about 70%. 3G & 4G Degrading
0
10
20
30
40
50
60
70
80
90
100
%OnTimeItems
Month
OTI Industry Average Trends
Linear (3.3.2.1 (2G)) Linear (3.3.2.2 (3G)) Linear (3.3.2.3 (4G))
On Time Item Delivery Industry Average

72
Wireless Study
4G YRR, SPR2, FRT2 maturing rivaling 2G and 3G;
OTI degrading for 3G and 4G and not in line with customer expectations
• 2G – Shows most mature performance
as expected
• 3G and 4G:
•SPR2 improving
•FRT2 improving
•OTI degrading

Investigation Letter Trial Example (PC 1.2.2)
73
Dear TL 9000 Registration Administrator,
It has been brought to the attention of the QuEST Forum Sub-team charged with monitoring and improving the
TL 9000 data reports that there is a problem with the return rate reporting for Product Category 1.2.2 Access
Multi-Service. The specific issue is there is at least one organization reporting return rates that are extremely
high. This is evident in the Worst-In-Class values for all four of the return rates, but is most evident for ERI
where the value is running over 200% per year. This would mean that on average each new unit is returned
twice within six months of shipment. Since it is highly unlikely this could be true, we are asking all
organizations with TL 9000 Certification in this product category to verify the FR data that they are submitting
before your next data submittal.
Specifically, please verify the following:
1. Only returns received during the month being reported are being counted
2. Each return is included in one and only one basis population
3. The is no factor applied to the number of returns, the actual count is being reported
4. The denominator includes all units shipped that belong to the basis population
If you find there has been an error in your data reporting, you are required to correct and resubmit your data
for up to the last 24 months.
This letter is being sent to all organizations certified in Product Category 1.2.2 and to your Certification
Bodies. The Certification Bodies are asked to confirm that the organization has verified their TL 9000 return
rate data during their next audit. In the interim, the organization shall confirm to QuEST Forum that this
verification has been performed within 30 days of receipt of this e-mail. Failure to do so will result in the
suspension of your TL 9000 Certification until your confirmation has been received.
If you have any questions, please use the “Contact Us” function on the tl9000.org web site.
Thank you for your co-operation in this matter. Correcting this issue is essential in order to provide useful
comparable return rate data for the organizations in this product category.
QuEST Forum TL 9000 Administrator

1.2.2 Example Continued –
Investigation Letters Resulted in ERI
Rates Returning to Typical Levels
74
24 letters sent, plus reminders as needed.
A few acknowledged issues and resubmitted data.
0
20
40
60
80
100
120
5/2011 8/2011 11/2011 2/2012 5/2012 8/2012 11/2012
1.2.2 ERI
ERI Moved to typical
levels after
resubmissions

2014 Investigations Conducted
75
Product
Category Name Issue # of Certs. # Responded Suspended
Dropped
Cert
Still
Suspended
Reported
Corrected
Data
3.2.5
Fiber to the
User FR - WIC 18 17 0 1 0 1
7.6.1
Procurement
Services OTI - WIC 16 15 0 1 0 1
3.1.1.1.1
Metallic
Conductor
Cable NPRs 19 17 4 0 2 4
3.1.1.2.1
Fiber Optic
Cable NPRs 28 26 6 0 2 7
Tracking of Investigation Responses is Currently Done Manually.
The Team Is Considering Mechanized Methods Going Forward.
No Response Has
Resulted in
Suspensions!

Asking Companies to Check Their Data When
Anomalies Occur, Can Improve Data Accuracy
76
0
10
20
30
40
50
60
70
80
90
100
4/2012
5/2012
6/2012
7/2012
8/2012
9/2012
10/2012
11/2012
12/2012
1/2013
2/2013
3/2013
4/2013
5/2013
6/2013
7/2013
8/2013
9/2013
10/2013
11/2013
12/2013
1/2014
2/2014
3/2014
Change in 7.6.1 OTI Worst in Class
NEW WorstInClass
OLD WorstInClass
While Anomalies can occur for valid reasons, if one seems completely
unreasonable, report it via “Contact Us” and the PDR Team will investigate.

77
Correct?
Category added
to profile
QF approval
noted in profile
Pop-up asks for
info
Info
Complete?
No
Yes
Verification
team notified
Info
Sufficient?
No
Yes
Request for
more info
No
Yes
Feedback to
organization
Info received
1) Description
2) Product name and/or model number
(hardware/software products –
Families 1-6 and 8)
3) Primary function (hardware/software
products – Families 1-6 and 8)
4) Competitors products name or
model numbers (hardware/software
products– Families 1-6 and 8)
5) Link(s) to material on web
6) Attach brochure(s)
A New Product
Category Approval
Process Has Been
Approved to Improve
Product Category
Selection Accuracy –

Useful PDR Whitepapers
• PDR Usage
• PDR Measurements Outputs and
Calculations
• Product Category Selection
• TL 9000 Data Submission
Methods
78

QuEST FORUM
Member Value Study
2014 Results

© 2014 Walker Information - proprietary and confidential 80
2014 MEMBER VALUE SURVEY RESULTS
Response Rate
Invitations
Sent1871
Completed
Surveys273
Response
Rate15%
Companies
Identified205
Companies
Participated125
Company
Response61%
2014 INDIVIDUAL RESPONSE
2014 COMPANY RESPONSE
1720 Invitations Sent 258 Completed Surveys 15% Response Rate
187 Companies Identified 112 Companies Participated 60% Company Response
2013 COMPARISON
2013 COMPARISON

Overall Membership Satisfaction
The higher the impact number, the more impact that measure
has on the overall question shown at the top of the graph.

Performance Data Reports
The higher the impact number, the more impact that measure
has on the overall question shown at the top of the graph.

Performance Data Reports

QuEST Forum Member Value Survey
PDR Specific Data Trends
2008 2009 2010 2011 2012 2013 2014
Overall PDR Satisfaction 49 60 63 70 67 71 70
Useful/Relevant PDR Data 48 54 62 68 66 69 69
Identifying Improvement Areas 45 52 56 58 55 60 62
Easy To Use 54 58 60 71 63 68 69
Providing Benchmarking Data 53 56 64 73 67 67 65
Providing Accurate Data 47 52 61 67 67 64 71
Timely Data 65 66 73 84 79 79 82
Supporting Continual Improvement 70
Aiding in Supplier Management 51
Top 2 Box Results
2014 n=152
2013 n=161
2012 n=156
2011 n=172
Significant Improvement in “Providing Accurate Data” while the
2014 PDR Overall Satisfaction Score holds steady near all time highs.
Efforts are paying off - but more needs to be done

Contact Information:
Ken Koffman
JDSU
+1 (240) 404-1249
ken.koffman@jdsu.com
John Wronka
Alcatel-Lucent
+1 (727) 204-9508
john.wronka@alcatel-lucent.com

TL 9000 Measurements and Requirements Interactive Workshop

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to TL 9000 Measurements and Requirements Interactive Workshop

Similar to TL 9000 Measurements and Requirements Interactive Workshop (20)

More from QuEST Forum

More from QuEST Forum (20)

Recently uploaded

Recently uploaded (20)

TL 9000 Measurements and Requirements Interactive Workshop