SlideShare a Scribd company logo
1 of 20
Test What Matters Most
Johnny Willemsen, Remedy IT
Keri Sprinkle, Coverity
Jon Jarboe, Coverity
Agenda
2 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
• Part 1: Remedy IT and ACE
• Introduction
• ACE and Coverity Scan
• Testing challenges
• Creating an ENFORCEABLE testing policy
• 4 Step policy with a big impact
• Part 2: Coverity
• Overview of Coverity Test Advisor
• Coverity suite of development testing solutions
• Development Testing Maturity Model
• Q & A
Remedy IT
ACE
3For Coverity and Partner use only. Copyright Coverity, Inc., 2013
Remedy IT
4 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
• Company of specialists focused on middleware and
component technologies
• Dedicated to open standards and
preferable open source
• Actively involved in the development of
ACE/TAO/CIAO/DAnCE/OpenDDS
open source software suite
What is ACE?
5 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
ADAPTIVE Communication Environment
• Open-source object-oriented (OO) framework that
implements many core patterns for concurrent
communication software
• Strong adoption in the telecom, medical, aerospace,
defense and financial services industries
ACE and Coverity Scan
6 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
• What is Coverity Scan?
• Free static analysis for open source projects
• Find and fix defects in C/C++ or Java
• Tests every line of code and potential execution path
• Explains root cause of each defect making it easy to fix bugs
• ACE and Coverity Scan
• Joined in April 2011
• Fixed 3243 defects as of June 2013
• 1827 defects outstanding
• Defect density of 0.24
ACE testing challenges
7 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
Goal: reduce risk through actionable, efficient testing
• Large, decentralized project with distributed development teams
• 7.8 million lines of C++ code
• 20 primary developers, 100 developers committing code to the repository,
about 2375 developers who have contributed additions/patches
• Last release on May 28, 2013
• Testing a priority, but not enforceable, so most code remains untested
• Current overall code coverage is 27% (57% for critical components)
• Over 275,000 functions not fully tested including examples and other code
that does not require testing
Creating an ENFORCEABLE test policy
8 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
• Many concerns before the project began
• Where do I start, and how do I prioritize, with so much untested code?
• How much real risk is there that I need to address ASAP?
• How do I hold developers accountable for testing their own code?
• How do I ensure code is tested as new features are developed?
• ACE developed a basic Coverity Test Advisor policy to
address these concerns
• Filtering rules to determine which code is important to test
• Analyzing untested code against those rules to identify missing tests
and prioritize them
4 Step policy with a big impact
9 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
Step Remaining
Violations
Testing Policy
0 275,313 Simple line coverage
All insufficiently‐ or un‐tested functions
1 12,833 Focus on core components
Premise: certain components are more important to test 
than others
2 2,967 Only called code (ignore uncalled)
Premise: we only write tests for important functions
3 2,757 Exclude debug, logging code
Premise: diagnostic messages are not important to test
4 2,588
Since 6.1:  605
Exclude error cases, prioritize violations by release
Premise: testing error cases provides minimal value and it is 
often difficult to trigger this code
Note: These numbers are for the exact same code
Using Coverity
10 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
• ACE performs a weekly build dedicated to Coverity scan
• Coverity scan analyzes the code and publishes the Scan
results
• New issues are emailed to all developers
• The ACE build czar will enforce that any new issue gets
resolved by the original developer
• Supports our goal that quality increases weekly and never
decreases
Addressing the issues
11 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
• Test violations in the core code are most important
• Try to resolve high priority violations with funding through
maintenance and support
• When resolving bugs, test violations are used to focus the
new unit tests on what is really needed
• When adding a feature to ACE, Test Advisor is used to
determine that the feature is well tested
• After a decrease in number of test violations, the basic Test
Advisor policy can be relaxed
Coverity
12For Coverity and Partner use only. Copyright Coverity, Inc., 2013
Automated Testing Is Not Effectively
Measured
13 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
Effort to Develop Tests
%ofCodeTest
1. Diminishing return for
increased test effort
2. Not all code is testable:
• Unreachable statements
• Dead code
3. Not all tested code adds
equal value:
• Non critical code
• Debug code
• Exception handling code
100%
Coverity Test Advisor
14 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
Improves unit testing effectiveness and efficiency
Focus testing
time where
it matters
And
don’t waste
time writing tests
you don’t need
High
Risk
Code
Creating an enforceable process
The high priority code that must
be thoroughly tested
Define
Analyze
Remediate
Govern
Code that has changed
and been impacted by changes
Code that has been insufficiently
tested
Manage progress to improve test
coverage
15
What code needs to be tested based on
your high risk criteriaDefine
Analyze
Remediate
Govern
With patent-pending techniques based
on code behavior and change impact
Surface issues in your workflow and
efficiently manage to closure
Create a testing stage gate and enforce
developer accountability
Generic example:
• All code changes for next release …
• And code impacted by those changes …
• Must have 100% coverage …
• Not counting exception handling and debug
code
Policy definition example
16 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
ACE:
• Focus on core components
• Only called code (ignore uncalled)
• Exclude debug, logging code
• Exclude error cases, prioritize violations by
release
• Exclude error code
• Prioritize violations by release
Remediate Testing Violations
Unified view
of test
violations and
defects
Unified view
of test
violations and
defects
Clear
description of
the violation
Clear
description of
the violation
Automatically
assign
violations to
owners
Automatically
assign
violations to
owners
Identify where
the violation
occurs in the
code
Identify where
the violation
occurs in the
code
For Coverity and Partner use only. Copyright Coverity, Inc., 2013
17
Coverity Connect
Coverity SAVE™
Static Analysis Verification Engine
Test
Advisor
SDLC
Integrations
Test
Execution
Third Party
Metrics
Code
Coverage
SCM
Coverity Development Testing Platform
18 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
Security
Advisor
Test
Advisor
Analysis Packs
Coverity SAVE™
Static Analysis Verification Engine
Policy Manager
Quality
Advisor
Architecture
Analysis
Dynamic
Analysis
Analysis
Integration
FindBugs™ | FxCop
Analysis
Integration
Toolkit
Coverity Connect
Test
Execution
Third Party
Metrics
Build/
Continuous
Integration
ALM
HP | IBM
IDE
Code
Coverage
Defect
Tracking
SCM
Proprietary Code | Open Source Code
(Scan)
Development Testing Maturity Model
19 For Coverity and Partner use only. Copyright Coverity, Inc., 2013
Development Testing Adoption
IntegrationintoSDLC
Level 1
Detection of
critical quality and
security defects as
part of SW build
process.
No new defects
introduced.
Level 3
Integration into
the existing SDLC
using a common
workflow for all
defects and test
effectiveness
issues.
Level 4
Establish and
enforce consistent
source code
quality and
security policies.
Establish source
code acceptance
criteria.
Level 5
All legacy defects
eliminated, build
fails if new defects
are introduced.
All critical code
and code impacted
by change is
tested.
High
High
Level 2
Identification of
areas of risk
caused by
insufficient
automated testing.
Ensure critical
code is prioritized
and tested.
Q & A
20 For Coverity and Partner use only. Copyright Coverity, Inc., 2012
• Remedy IT:
www.remedy.nl
• ACE:
www.cs.wustl.edu/~schmidt/ACE.html
• Coverity:
coverity.com
• Coverity Scan:
scan.coverity.com

More Related Content

What's hot

Mickiel Vroon - Test Environment, The Future Achilles’ Heel
Mickiel Vroon - Test Environment, The Future Achilles’ HeelMickiel Vroon - Test Environment, The Future Achilles’ Heel
Mickiel Vroon - Test Environment, The Future Achilles’ HeelTEST Huddle
 
Agile Testing Transformation is as Easy as 1, 2, 3 by Michael Buening
Agile Testing Transformation is as Easy as 1, 2, 3 by Michael BueningAgile Testing Transformation is as Easy as 1, 2, 3 by Michael Buening
Agile Testing Transformation is as Easy as 1, 2, 3 by Michael BueningQA or the Highway
 
DevOps the Big Picture for Testers by Joseph Ours
DevOps the Big Picture for Testers by Joseph OursDevOps the Big Picture for Testers by Joseph Ours
DevOps the Big Picture for Testers by Joseph OursQA or the Highway
 
Quality for DevOps teams - Quality engineering in the DevOps culture
Quality for DevOps teams - Quality engineering in the DevOps cultureQuality for DevOps teams - Quality engineering in the DevOps culture
Quality for DevOps teams - Quality engineering in the DevOps cultureRik Marselis
 
EXTENT-2016: The Future of Software Testing
EXTENT-2016:	 The Future of Software TestingEXTENT-2016:	 The Future of Software Testing
EXTENT-2016: The Future of Software TestingIosif Itkin
 
The 10 commandments of an agile tester in a legacy world
The 10 commandments of an agile tester in a legacy worldThe 10 commandments of an agile tester in a legacy world
The 10 commandments of an agile tester in a legacy worldAgilePractitionersIL
 
'Customer Testing & Quality In Outsourced Development - A Story From An Insur...
'Customer Testing & Quality In Outsourced Development - A Story From An Insur...'Customer Testing & Quality In Outsourced Development - A Story From An Insur...
'Customer Testing & Quality In Outsourced Development - A Story From An Insur...TEST Huddle
 
Creating a successful continuous testing environment by Eran Kinsbruner
Creating a successful continuous testing environment by Eran KinsbrunerCreating a successful continuous testing environment by Eran Kinsbruner
Creating a successful continuous testing environment by Eran KinsbrunerQA or the Highway
 
Synthesizing Continuous Deployment Practices in Software Development
Synthesizing Continuous Deployment Practices in Software DevelopmentSynthesizing Continuous Deployment Practices in Software Development
Synthesizing Continuous Deployment Practices in Software DevelopmentAkond Rahman
 
Continuous Quality: DevOps with Quality by Josh Eastman
Continuous Quality: DevOps with Quality by Josh EastmanContinuous Quality: DevOps with Quality by Josh Eastman
Continuous Quality: DevOps with Quality by Josh EastmanQA or the Highway
 
Quality Assurance to Test Engineering – Insights From our Journey by Oksana S...
Quality Assurance to Test Engineering – Insights From our Journey by Oksana S...Quality Assurance to Test Engineering – Insights From our Journey by Oksana S...
Quality Assurance to Test Engineering – Insights From our Journey by Oksana S...QA or the Highway
 
IoT Software Testing Challenges: The IoT World Is Really Different
IoT Software Testing Challenges: The IoT World Is Really DifferentIoT Software Testing Challenges: The IoT World Is Really Different
IoT Software Testing Challenges: The IoT World Is Really DifferentTechWell
 
Achieving CI Excellence with Quality Engineering
Achieving CI Excellence with Quality EngineeringAchieving CI Excellence with Quality Engineering
Achieving CI Excellence with Quality EngineeringGreg Sypolt
 
AppsSec In a DevOps World
AppsSec In a DevOps WorldAppsSec In a DevOps World
AppsSec In a DevOps WorldParasoft
 
The Legend of Software Hollow: Defeating the Headless Horseman of Faulty Appl...
The Legend of Software Hollow: Defeating the Headless Horseman of Faulty Appl...The Legend of Software Hollow: Defeating the Headless Horseman of Faulty Appl...
The Legend of Software Hollow: Defeating the Headless Horseman of Faulty Appl...Parasoft
 
Let's focus more on Quality and less on Testing by Joel Montvelisky
Let's focus more on Quality and less on Testing by Joel MontveliskyLet's focus more on Quality and less on Testing by Joel Montvelisky
Let's focus more on Quality and less on Testing by Joel MontveliskyQA or the Highway
 
[Vu Van Nguyen] Value-based Software Testing an Approach to Prioritizing Tests
[Vu Van Nguyen]  Value-based Software Testing an Approach to Prioritizing Tests[Vu Van Nguyen]  Value-based Software Testing an Approach to Prioritizing Tests
[Vu Van Nguyen] Value-based Software Testing an Approach to Prioritizing TestsHo Chi Minh City Software Testing Club
 
Let the adventure begin the 80-20 testing - ingenuity
Let the adventure begin   the 80-20 testing - ingenuityLet the adventure begin   the 80-20 testing - ingenuity
Let the adventure begin the 80-20 testing - ingenuityIndium Software
 
The digital shakeout in quality assurance and testing by Shiva Agolla and Sat...
The digital shakeout in quality assurance and testing by Shiva Agolla and Sat...The digital shakeout in quality assurance and testing by Shiva Agolla and Sat...
The digital shakeout in quality assurance and testing by Shiva Agolla and Sat...QA or the Highway
 
Instill a DevOps Testing Culture in Your Team and Organization
Instill a DevOps Testing Culture in Your Team and Organization Instill a DevOps Testing Culture in Your Team and Organization
Instill a DevOps Testing Culture in Your Team and Organization TechWell
 

What's hot (20)

Mickiel Vroon - Test Environment, The Future Achilles’ Heel
Mickiel Vroon - Test Environment, The Future Achilles’ HeelMickiel Vroon - Test Environment, The Future Achilles’ Heel
Mickiel Vroon - Test Environment, The Future Achilles’ Heel
 
Agile Testing Transformation is as Easy as 1, 2, 3 by Michael Buening
Agile Testing Transformation is as Easy as 1, 2, 3 by Michael BueningAgile Testing Transformation is as Easy as 1, 2, 3 by Michael Buening
Agile Testing Transformation is as Easy as 1, 2, 3 by Michael Buening
 
DevOps the Big Picture for Testers by Joseph Ours
DevOps the Big Picture for Testers by Joseph OursDevOps the Big Picture for Testers by Joseph Ours
DevOps the Big Picture for Testers by Joseph Ours
 
Quality for DevOps teams - Quality engineering in the DevOps culture
Quality for DevOps teams - Quality engineering in the DevOps cultureQuality for DevOps teams - Quality engineering in the DevOps culture
Quality for DevOps teams - Quality engineering in the DevOps culture
 
EXTENT-2016: The Future of Software Testing
EXTENT-2016:	 The Future of Software TestingEXTENT-2016:	 The Future of Software Testing
EXTENT-2016: The Future of Software Testing
 
The 10 commandments of an agile tester in a legacy world
The 10 commandments of an agile tester in a legacy worldThe 10 commandments of an agile tester in a legacy world
The 10 commandments of an agile tester in a legacy world
 
'Customer Testing & Quality In Outsourced Development - A Story From An Insur...
'Customer Testing & Quality In Outsourced Development - A Story From An Insur...'Customer Testing & Quality In Outsourced Development - A Story From An Insur...
'Customer Testing & Quality In Outsourced Development - A Story From An Insur...
 
Creating a successful continuous testing environment by Eran Kinsbruner
Creating a successful continuous testing environment by Eran KinsbrunerCreating a successful continuous testing environment by Eran Kinsbruner
Creating a successful continuous testing environment by Eran Kinsbruner
 
Synthesizing Continuous Deployment Practices in Software Development
Synthesizing Continuous Deployment Practices in Software DevelopmentSynthesizing Continuous Deployment Practices in Software Development
Synthesizing Continuous Deployment Practices in Software Development
 
Continuous Quality: DevOps with Quality by Josh Eastman
Continuous Quality: DevOps with Quality by Josh EastmanContinuous Quality: DevOps with Quality by Josh Eastman
Continuous Quality: DevOps with Quality by Josh Eastman
 
Quality Assurance to Test Engineering – Insights From our Journey by Oksana S...
Quality Assurance to Test Engineering – Insights From our Journey by Oksana S...Quality Assurance to Test Engineering – Insights From our Journey by Oksana S...
Quality Assurance to Test Engineering – Insights From our Journey by Oksana S...
 
IoT Software Testing Challenges: The IoT World Is Really Different
IoT Software Testing Challenges: The IoT World Is Really DifferentIoT Software Testing Challenges: The IoT World Is Really Different
IoT Software Testing Challenges: The IoT World Is Really Different
 
Achieving CI Excellence with Quality Engineering
Achieving CI Excellence with Quality EngineeringAchieving CI Excellence with Quality Engineering
Achieving CI Excellence with Quality Engineering
 
AppsSec In a DevOps World
AppsSec In a DevOps WorldAppsSec In a DevOps World
AppsSec In a DevOps World
 
The Legend of Software Hollow: Defeating the Headless Horseman of Faulty Appl...
The Legend of Software Hollow: Defeating the Headless Horseman of Faulty Appl...The Legend of Software Hollow: Defeating the Headless Horseman of Faulty Appl...
The Legend of Software Hollow: Defeating the Headless Horseman of Faulty Appl...
 
Let's focus more on Quality and less on Testing by Joel Montvelisky
Let's focus more on Quality and less on Testing by Joel MontveliskyLet's focus more on Quality and less on Testing by Joel Montvelisky
Let's focus more on Quality and less on Testing by Joel Montvelisky
 
[Vu Van Nguyen] Value-based Software Testing an Approach to Prioritizing Tests
[Vu Van Nguyen]  Value-based Software Testing an Approach to Prioritizing Tests[Vu Van Nguyen]  Value-based Software Testing an Approach to Prioritizing Tests
[Vu Van Nguyen] Value-based Software Testing an Approach to Prioritizing Tests
 
Let the adventure begin the 80-20 testing - ingenuity
Let the adventure begin   the 80-20 testing - ingenuityLet the adventure begin   the 80-20 testing - ingenuity
Let the adventure begin the 80-20 testing - ingenuity
 
The digital shakeout in quality assurance and testing by Shiva Agolla and Sat...
The digital shakeout in quality assurance and testing by Shiva Agolla and Sat...The digital shakeout in quality assurance and testing by Shiva Agolla and Sat...
The digital shakeout in quality assurance and testing by Shiva Agolla and Sat...
 
Instill a DevOps Testing Culture in Your Team and Organization
Instill a DevOps Testing Culture in Your Team and Organization Instill a DevOps Testing Culture in Your Team and Organization
Instill a DevOps Testing Culture in Your Team and Organization
 

Viewers also liked

How To Write Good (Nice + Tidy For The Interwebs!)
How To Write Good (Nice + Tidy For The Interwebs!)How To Write Good (Nice + Tidy For The Interwebs!)
How To Write Good (Nice + Tidy For The Interwebs!)ReferralCandy
 
Innovations in Grid Computing with Oracle Coherence
Innovations in Grid Computing with Oracle CoherenceInnovations in Grid Computing with Oracle Coherence
Innovations in Grid Computing with Oracle CoherenceBob Rhubart
 
인터랙티브미디어디자인 l 1514599 서은성
인터랙티브미디어디자인 l 1514599 서은성인터랙티브미디어디자인 l 1514599 서은성
인터랙티브미디어디자인 l 1514599 서은성은성 서
 
DeNAにおける機械学習・深層学習活用
DeNAにおける機械学習・深層学習活用DeNAにおける機械学習・深層学習活用
DeNAにおける機械学習・深層学習活用Kazuki Fujikawa
 
Facebook's Growth Hacker on how they put Facebook on the Path to 1 Billion Users
Facebook's Growth Hacker on how they put Facebook on the Path to 1 Billion UsersFacebook's Growth Hacker on how they put Facebook on the Path to 1 Billion Users
Facebook's Growth Hacker on how they put Facebook on the Path to 1 Billion Usersgrowthhackersconference
 
Artificial Intelligence and Law - 
A Primer
Artificial Intelligence and Law - 
A Primer Artificial Intelligence and Law - 
A Primer
Artificial Intelligence and Law - 
A Primer Daniel Katz
 
Exome sequencing for disease gene identification and patient diagnostics, Gen...
Exome sequencing for disease gene identification and patient diagnostics, Gen...Exome sequencing for disease gene identification and patient diagnostics, Gen...
Exome sequencing for disease gene identification and patient diagnostics, Gen...Copenhagenomics
 
ハードディスクの正しい消去(2015.7)
ハードディスクの正しい消去(2015.7)ハードディスクの正しい消去(2015.7)
ハードディスクの正しい消去(2015.7)UEHARA, Tetsutaro
 
Son los sofás modulares iguales que los sofás seccionales
Son los sofás modulares iguales que los sofás seccionalesSon los sofás modulares iguales que los sofás seccionales
Son los sofás modulares iguales que los sofás seccionalesJacob Michael
 

Viewers also liked (10)

How To Write Good (Nice + Tidy For The Interwebs!)
How To Write Good (Nice + Tidy For The Interwebs!)How To Write Good (Nice + Tidy For The Interwebs!)
How To Write Good (Nice + Tidy For The Interwebs!)
 
Innovations in Grid Computing with Oracle Coherence
Innovations in Grid Computing with Oracle CoherenceInnovations in Grid Computing with Oracle Coherence
Innovations in Grid Computing with Oracle Coherence
 
인터랙티브미디어디자인 l 1514599 서은성
인터랙티브미디어디자인 l 1514599 서은성인터랙티브미디어디자인 l 1514599 서은성
인터랙티브미디어디자인 l 1514599 서은성
 
DeNAにおける機械学習・深層学習活用
DeNAにおける機械学習・深層学習活用DeNAにおける機械学習・深層学習活用
DeNAにおける機械学習・深層学習活用
 
Facebook's Growth Hacker on how they put Facebook on the Path to 1 Billion Users
Facebook's Growth Hacker on how they put Facebook on the Path to 1 Billion UsersFacebook's Growth Hacker on how they put Facebook on the Path to 1 Billion Users
Facebook's Growth Hacker on how they put Facebook on the Path to 1 Billion Users
 
RESTful API Design, Second Edition
RESTful API Design, Second EditionRESTful API Design, Second Edition
RESTful API Design, Second Edition
 
Artificial Intelligence and Law - 
A Primer
Artificial Intelligence and Law - 
A Primer Artificial Intelligence and Law - 
A Primer
Artificial Intelligence and Law - 
A Primer
 
Exome sequencing for disease gene identification and patient diagnostics, Gen...
Exome sequencing for disease gene identification and patient diagnostics, Gen...Exome sequencing for disease gene identification and patient diagnostics, Gen...
Exome sequencing for disease gene identification and patient diagnostics, Gen...
 
ハードディスクの正しい消去(2015.7)
ハードディスクの正しい消去(2015.7)ハードディスクの正しい消去(2015.7)
ハードディスクの正しい消去(2015.7)
 
Son los sofás modulares iguales que los sofás seccionales
Son los sofás modulares iguales que los sofás seccionalesSon los sofás modulares iguales que los sofás seccionales
Son los sofás modulares iguales que los sofás seccionales
 

Similar to Test What Matters Most

Test What Matters Most
Test What Matters MostTest What Matters Most
Test What Matters MostRemedy IT
 
Adopting Agile
Adopting AgileAdopting Agile
Adopting AgileCoverity
 
Backward thinking design qa system for quality goals
Backward thinking   design qa system for quality goalsBackward thinking   design qa system for quality goals
Backward thinking design qa system for quality goalsgaoliang641
 
[India Merge World Tour] Coverity
[India Merge World Tour] Coverity[India Merge World Tour] Coverity
[India Merge World Tour] CoverityPerforce
 
Test driven development
Test driven developmentTest driven development
Test driven developmentJames Wong
 
Test driven development
Test driven developmentTest driven development
Test driven developmentFraboni Ec
 
Test driven development
Test driven developmentTest driven development
Test driven developmentHarry Potter
 
Test driven development
Test driven developmentTest driven development
Test driven developmentYoung Alista
 
Test driven development
Test driven developmentTest driven development
Test driven developmentTony Nguyen
 
Test driven development
Test driven developmentTest driven development
Test driven developmentLuis Goldster
 
How to build confidence in your release cycle
How to build confidence in your release cycleHow to build confidence in your release cycle
How to build confidence in your release cycleDiUS
 
Webcast Presentation: Accelerate Continuous Delivery with Development Testing...
Webcast Presentation: Accelerate Continuous Delivery with Development Testing...Webcast Presentation: Accelerate Continuous Delivery with Development Testing...
Webcast Presentation: Accelerate Continuous Delivery with Development Testing...GRUC
 
Effective Testing Practices in an Agile Environment
Effective Testing Practices in an Agile EnvironmentEffective Testing Practices in an Agile Environment
Effective Testing Practices in an Agile EnvironmentRaj Indugula
 
Chapter 13 software testing strategies
Chapter 13 software testing strategiesChapter 13 software testing strategies
Chapter 13 software testing strategiesSHREEHARI WADAWADAGI
 
Curiosity and Infuse Consulting Present: Sustainable Test Automation Strategi...
Curiosity and Infuse Consulting Present: Sustainable Test Automation Strategi...Curiosity and Infuse Consulting Present: Sustainable Test Automation Strategi...
Curiosity and Infuse Consulting Present: Sustainable Test Automation Strategi...Curiosity Software Ireland
 
How BDD enables True CI/CD
How BDD enables True CI/CDHow BDD enables True CI/CD
How BDD enables True CI/CDRoger Turnau
 

Similar to Test What Matters Most (20)

Test What Matters Most
Test What Matters MostTest What Matters Most
Test What Matters Most
 
Test Policy and Practices
Test Policy and PracticesTest Policy and Practices
Test Policy and Practices
 
Adopting Agile
Adopting AgileAdopting Agile
Adopting Agile
 
Backward thinking design qa system for quality goals
Backward thinking   design qa system for quality goalsBackward thinking   design qa system for quality goals
Backward thinking design qa system for quality goals
 
[India Merge World Tour] Coverity
[India Merge World Tour] Coverity[India Merge World Tour] Coverity
[India Merge World Tour] Coverity
 
Test driven development
Test driven developmentTest driven development
Test driven development
 
Test driven development
Test driven developmentTest driven development
Test driven development
 
Test driven development
Test driven developmentTest driven development
Test driven development
 
Test driven development
Test driven developmentTest driven development
Test driven development
 
Test driven development
Test driven developmentTest driven development
Test driven development
 
Test driven development
Test driven developmentTest driven development
Test driven development
 
How to build confidence in your release cycle
How to build confidence in your release cycleHow to build confidence in your release cycle
How to build confidence in your release cycle
 
Code coverage
Code coverageCode coverage
Code coverage
 
Methodology: IT test
Methodology: IT testMethodology: IT test
Methodology: IT test
 
Webcast Presentation: Accelerate Continuous Delivery with Development Testing...
Webcast Presentation: Accelerate Continuous Delivery with Development Testing...Webcast Presentation: Accelerate Continuous Delivery with Development Testing...
Webcast Presentation: Accelerate Continuous Delivery with Development Testing...
 
Effective Testing Practices in an Agile Environment
Effective Testing Practices in an Agile EnvironmentEffective Testing Practices in an Agile Environment
Effective Testing Practices in an Agile Environment
 
Software coding and testing
Software coding and testingSoftware coding and testing
Software coding and testing
 
Chapter 13 software testing strategies
Chapter 13 software testing strategiesChapter 13 software testing strategies
Chapter 13 software testing strategies
 
Curiosity and Infuse Consulting Present: Sustainable Test Automation Strategi...
Curiosity and Infuse Consulting Present: Sustainable Test Automation Strategi...Curiosity and Infuse Consulting Present: Sustainable Test Automation Strategi...
Curiosity and Infuse Consulting Present: Sustainable Test Automation Strategi...
 
How BDD enables True CI/CD
How BDD enables True CI/CDHow BDD enables True CI/CD
How BDD enables True CI/CD
 

More from Remedy IT

Integrating DDS into AXCIOMA, the component approach
Integrating DDS into AXCIOMA, the component approachIntegrating DDS into AXCIOMA, the component approach
Integrating DDS into AXCIOMA, the component approachRemedy IT
 
AXCIOMA, the component framework for distributed, real-time and embedded systems
AXCIOMA, the component framework for distributed, real-time and embedded systemsAXCIOMA, the component framework for distributed, real-time and embedded systems
AXCIOMA, the component framework for distributed, real-time and embedded systemsRemedy IT
 
AXCIOMA, the internals, the component framework for distributed, real-time, a...
AXCIOMA, the internals, the component framework for distributed, real-time, a...AXCIOMA, the internals, the component framework for distributed, real-time, a...
AXCIOMA, the internals, the component framework for distributed, real-time, a...Remedy IT
 
Remedy IT Company presentation
Remedy IT Company presentationRemedy IT Company presentation
Remedy IT Company presentationRemedy IT
 
CORBA Programming with TAOX11/C++11 tutorial
CORBA Programming with TAOX11/C++11 tutorialCORBA Programming with TAOX11/C++11 tutorial
CORBA Programming with TAOX11/C++11 tutorialRemedy IT
 
Comparing IDL to C++ with IDL to C++11
Comparing IDL to C++ with IDL to C++11Comparing IDL to C++ with IDL to C++11
Comparing IDL to C++ with IDL to C++11Remedy IT
 
Integrating DDS into AXCIOMA, the component approach
Integrating DDS into AXCIOMA, the component approachIntegrating DDS into AXCIOMA, the component approach
Integrating DDS into AXCIOMA, the component approachRemedy IT
 
Modernizing SCA through new Object Management Group (OMG) standards
Modernizing SCA through new Object Management Group (OMG) standardsModernizing SCA through new Object Management Group (OMG) standards
Modernizing SCA through new Object Management Group (OMG) standardsRemedy IT
 
AXCIOMA, the internals, the component framework for distributed, real-time, a...
AXCIOMA, the internals, the component framework for distributed, real-time, a...AXCIOMA, the internals, the component framework for distributed, real-time, a...
AXCIOMA, the internals, the component framework for distributed, real-time, a...Remedy IT
 
Modernizing SCA through new Object Management Group (OMG) standards
Modernizing SCA through new Object Management Group (OMG) standardsModernizing SCA through new Object Management Group (OMG) standards
Modernizing SCA through new Object Management Group (OMG) standardsRemedy IT
 
ACE/TAO/CIAO/DAnCE Maintenance overview
ACE/TAO/CIAO/DAnCE Maintenance overviewACE/TAO/CIAO/DAnCE Maintenance overview
ACE/TAO/CIAO/DAnCE Maintenance overviewRemedy IT
 
Remedy IT Revised Submission Presentation for the Unified Component Model (UC...
Remedy IT Revised Submission Presentation for the Unified Component Model (UC...Remedy IT Revised Submission Presentation for the Unified Component Model (UC...
Remedy IT Revised Submission Presentation for the Unified Component Model (UC...Remedy IT
 
Revised submission for Unified Component Model (UCM) for Distributed, Real-Ti...
Revised submission for Unified Component Model (UCM) for Distributed, Real-Ti...Revised submission for Unified Component Model (UCM) for Distributed, Real-Ti...
Revised submission for Unified Component Model (UCM) for Distributed, Real-Ti...Remedy IT
 
DDS Programming with IDL to C++11 tutorial
DDS Programming with IDL to C++11 tutorialDDS Programming with IDL to C++11 tutorial
DDS Programming with IDL to C++11 tutorialRemedy IT
 
Component Based DDS with C++11 and R2DDS
Component Based DDS with C++11 and R2DDSComponent Based DDS with C++11 and R2DDS
Component Based DDS with C++11 and R2DDSRemedy IT
 
AXCIOMA, the component framework for distributed, real-time and embedded systems
AXCIOMA, the component framework for distributed, real-time and embedded systemsAXCIOMA, the component framework for distributed, real-time and embedded systems
AXCIOMA, the component framework for distributed, real-time and embedded systemsRemedy IT
 
Component Technologies for Fractionated Satellites
Component Technologies for Fractionated SatellitesComponent Technologies for Fractionated Satellites
Component Technologies for Fractionated SatellitesRemedy IT
 
UCM Initial Submission presentation
UCM Initial Submission presentationUCM Initial Submission presentation
UCM Initial Submission presentationRemedy IT
 
Remedy IT Initial Submission for the Unified Component Model (UCM) for Distri...
Remedy IT Initial Submission for the Unified Component Model (UCM) for Distri...Remedy IT Initial Submission for the Unified Component Model (UCM) for Distri...
Remedy IT Initial Submission for the Unified Component Model (UCM) for Distri...Remedy IT
 
Unified Component Model for Distributed, Real- Time and Embedded Systems Requ...
Unified Component Model for Distributed, Real- Time and Embedded Systems Requ...Unified Component Model for Distributed, Real- Time and Embedded Systems Requ...
Unified Component Model for Distributed, Real- Time and Embedded Systems Requ...Remedy IT
 

More from Remedy IT (20)

Integrating DDS into AXCIOMA, the component approach
Integrating DDS into AXCIOMA, the component approachIntegrating DDS into AXCIOMA, the component approach
Integrating DDS into AXCIOMA, the component approach
 
AXCIOMA, the component framework for distributed, real-time and embedded systems
AXCIOMA, the component framework for distributed, real-time and embedded systemsAXCIOMA, the component framework for distributed, real-time and embedded systems
AXCIOMA, the component framework for distributed, real-time and embedded systems
 
AXCIOMA, the internals, the component framework for distributed, real-time, a...
AXCIOMA, the internals, the component framework for distributed, real-time, a...AXCIOMA, the internals, the component framework for distributed, real-time, a...
AXCIOMA, the internals, the component framework for distributed, real-time, a...
 
Remedy IT Company presentation
Remedy IT Company presentationRemedy IT Company presentation
Remedy IT Company presentation
 
CORBA Programming with TAOX11/C++11 tutorial
CORBA Programming with TAOX11/C++11 tutorialCORBA Programming with TAOX11/C++11 tutorial
CORBA Programming with TAOX11/C++11 tutorial
 
Comparing IDL to C++ with IDL to C++11
Comparing IDL to C++ with IDL to C++11Comparing IDL to C++ with IDL to C++11
Comparing IDL to C++ with IDL to C++11
 
Integrating DDS into AXCIOMA, the component approach
Integrating DDS into AXCIOMA, the component approachIntegrating DDS into AXCIOMA, the component approach
Integrating DDS into AXCIOMA, the component approach
 
Modernizing SCA through new Object Management Group (OMG) standards
Modernizing SCA through new Object Management Group (OMG) standardsModernizing SCA through new Object Management Group (OMG) standards
Modernizing SCA through new Object Management Group (OMG) standards
 
AXCIOMA, the internals, the component framework for distributed, real-time, a...
AXCIOMA, the internals, the component framework for distributed, real-time, a...AXCIOMA, the internals, the component framework for distributed, real-time, a...
AXCIOMA, the internals, the component framework for distributed, real-time, a...
 
Modernizing SCA through new Object Management Group (OMG) standards
Modernizing SCA through new Object Management Group (OMG) standardsModernizing SCA through new Object Management Group (OMG) standards
Modernizing SCA through new Object Management Group (OMG) standards
 
ACE/TAO/CIAO/DAnCE Maintenance overview
ACE/TAO/CIAO/DAnCE Maintenance overviewACE/TAO/CIAO/DAnCE Maintenance overview
ACE/TAO/CIAO/DAnCE Maintenance overview
 
Remedy IT Revised Submission Presentation for the Unified Component Model (UC...
Remedy IT Revised Submission Presentation for the Unified Component Model (UC...Remedy IT Revised Submission Presentation for the Unified Component Model (UC...
Remedy IT Revised Submission Presentation for the Unified Component Model (UC...
 
Revised submission for Unified Component Model (UCM) for Distributed, Real-Ti...
Revised submission for Unified Component Model (UCM) for Distributed, Real-Ti...Revised submission for Unified Component Model (UCM) for Distributed, Real-Ti...
Revised submission for Unified Component Model (UCM) for Distributed, Real-Ti...
 
DDS Programming with IDL to C++11 tutorial
DDS Programming with IDL to C++11 tutorialDDS Programming with IDL to C++11 tutorial
DDS Programming with IDL to C++11 tutorial
 
Component Based DDS with C++11 and R2DDS
Component Based DDS with C++11 and R2DDSComponent Based DDS with C++11 and R2DDS
Component Based DDS with C++11 and R2DDS
 
AXCIOMA, the component framework for distributed, real-time and embedded systems
AXCIOMA, the component framework for distributed, real-time and embedded systemsAXCIOMA, the component framework for distributed, real-time and embedded systems
AXCIOMA, the component framework for distributed, real-time and embedded systems
 
Component Technologies for Fractionated Satellites
Component Technologies for Fractionated SatellitesComponent Technologies for Fractionated Satellites
Component Technologies for Fractionated Satellites
 
UCM Initial Submission presentation
UCM Initial Submission presentationUCM Initial Submission presentation
UCM Initial Submission presentation
 
Remedy IT Initial Submission for the Unified Component Model (UCM) for Distri...
Remedy IT Initial Submission for the Unified Component Model (UCM) for Distri...Remedy IT Initial Submission for the Unified Component Model (UCM) for Distri...
Remedy IT Initial Submission for the Unified Component Model (UCM) for Distri...
 
Unified Component Model for Distributed, Real- Time and Embedded Systems Requ...
Unified Component Model for Distributed, Real- Time and Embedded Systems Requ...Unified Component Model for Distributed, Real- Time and Embedded Systems Requ...
Unified Component Model for Distributed, Real- Time and Embedded Systems Requ...
 

Recently uploaded

Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Brian Pichman
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesDavid Newbury
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxUdaiappa Ramachandran
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.YounusS2
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024D Cloud Solutions
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024SkyPlanner
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IES VE
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Will Schroeder
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...DianaGray10
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXTarek Kalaji
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URLRuncy Oommen
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfJamie (Taka) Wang
 

Recently uploaded (20)

Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )Building Your Own AI Instance (TBLC AI )
Building Your Own AI Instance (TBLC AI )
 
Linked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond OntologiesLinked Data in Production: Moving Beyond Ontologies
Linked Data in Production: Moving Beyond Ontologies
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Building AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptxBuilding AI-Driven Apps Using Semantic Kernel.pptx
Building AI-Driven Apps Using Semantic Kernel.pptx
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.Basic Building Blocks of Internet of Things.
Basic Building Blocks of Internet of Things.
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024Artificial Intelligence & SEO Trends for 2024
Artificial Intelligence & SEO Trends for 2024
 
Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024Salesforce Miami User Group Event - 1st Quarter 2024
Salesforce Miami User Group Event - 1st Quarter 2024
 
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
IESVE Software for Florida Code Compliance Using ASHRAE 90.1-2019
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
Apres-Cyber - The Data Dilemma: Bridging Offensive Operations and Machine Lea...
 
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
Connector Corner: Extending LLM automation use cases with UiPath GenAI connec...
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
VoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBXVoIP Service and Marketing using Odoo and Asterisk PBX
VoIP Service and Marketing using Odoo and Asterisk PBX
 
Designing A Time bound resource download URL
Designing A Time bound resource download URLDesigning A Time bound resource download URL
Designing A Time bound resource download URL
 
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
activity_diagram_combine_v4_20190827.pdfactivity_diagram_combine_v4_20190827.pdf
 

Test What Matters Most

  • 1. Test What Matters Most Johnny Willemsen, Remedy IT Keri Sprinkle, Coverity Jon Jarboe, Coverity
  • 2. Agenda 2 For Coverity and Partner use only. Copyright Coverity, Inc., 2013 • Part 1: Remedy IT and ACE • Introduction • ACE and Coverity Scan • Testing challenges • Creating an ENFORCEABLE testing policy • 4 Step policy with a big impact • Part 2: Coverity • Overview of Coverity Test Advisor • Coverity suite of development testing solutions • Development Testing Maturity Model • Q & A
  • 3. Remedy IT ACE 3For Coverity and Partner use only. Copyright Coverity, Inc., 2013
  • 4. Remedy IT 4 For Coverity and Partner use only. Copyright Coverity, Inc., 2013 • Company of specialists focused on middleware and component technologies • Dedicated to open standards and preferable open source • Actively involved in the development of ACE/TAO/CIAO/DAnCE/OpenDDS open source software suite
  • 5. What is ACE? 5 For Coverity and Partner use only. Copyright Coverity, Inc., 2013 ADAPTIVE Communication Environment • Open-source object-oriented (OO) framework that implements many core patterns for concurrent communication software • Strong adoption in the telecom, medical, aerospace, defense and financial services industries
  • 6. ACE and Coverity Scan 6 For Coverity and Partner use only. Copyright Coverity, Inc., 2013 • What is Coverity Scan? • Free static analysis for open source projects • Find and fix defects in C/C++ or Java • Tests every line of code and potential execution path • Explains root cause of each defect making it easy to fix bugs • ACE and Coverity Scan • Joined in April 2011 • Fixed 3243 defects as of June 2013 • 1827 defects outstanding • Defect density of 0.24
  • 7. ACE testing challenges 7 For Coverity and Partner use only. Copyright Coverity, Inc., 2013 Goal: reduce risk through actionable, efficient testing • Large, decentralized project with distributed development teams • 7.8 million lines of C++ code • 20 primary developers, 100 developers committing code to the repository, about 2375 developers who have contributed additions/patches • Last release on May 28, 2013 • Testing a priority, but not enforceable, so most code remains untested • Current overall code coverage is 27% (57% for critical components) • Over 275,000 functions not fully tested including examples and other code that does not require testing
  • 8. Creating an ENFORCEABLE test policy 8 For Coverity and Partner use only. Copyright Coverity, Inc., 2013 • Many concerns before the project began • Where do I start, and how do I prioritize, with so much untested code? • How much real risk is there that I need to address ASAP? • How do I hold developers accountable for testing their own code? • How do I ensure code is tested as new features are developed? • ACE developed a basic Coverity Test Advisor policy to address these concerns • Filtering rules to determine which code is important to test • Analyzing untested code against those rules to identify missing tests and prioritize them
  • 9. 4 Step policy with a big impact 9 For Coverity and Partner use only. Copyright Coverity, Inc., 2013 Step Remaining Violations Testing Policy 0 275,313 Simple line coverage All insufficiently‐ or un‐tested functions 1 12,833 Focus on core components Premise: certain components are more important to test  than others 2 2,967 Only called code (ignore uncalled) Premise: we only write tests for important functions 3 2,757 Exclude debug, logging code Premise: diagnostic messages are not important to test 4 2,588 Since 6.1:  605 Exclude error cases, prioritize violations by release Premise: testing error cases provides minimal value and it is  often difficult to trigger this code Note: These numbers are for the exact same code
  • 10. Using Coverity 10 For Coverity and Partner use only. Copyright Coverity, Inc., 2013 • ACE performs a weekly build dedicated to Coverity scan • Coverity scan analyzes the code and publishes the Scan results • New issues are emailed to all developers • The ACE build czar will enforce that any new issue gets resolved by the original developer • Supports our goal that quality increases weekly and never decreases
  • 11. Addressing the issues 11 For Coverity and Partner use only. Copyright Coverity, Inc., 2013 • Test violations in the core code are most important • Try to resolve high priority violations with funding through maintenance and support • When resolving bugs, test violations are used to focus the new unit tests on what is really needed • When adding a feature to ACE, Test Advisor is used to determine that the feature is well tested • After a decrease in number of test violations, the basic Test Advisor policy can be relaxed
  • 12. Coverity 12For Coverity and Partner use only. Copyright Coverity, Inc., 2013
  • 13. Automated Testing Is Not Effectively Measured 13 For Coverity and Partner use only. Copyright Coverity, Inc., 2013 Effort to Develop Tests %ofCodeTest 1. Diminishing return for increased test effort 2. Not all code is testable: • Unreachable statements • Dead code 3. Not all tested code adds equal value: • Non critical code • Debug code • Exception handling code 100%
  • 14. Coverity Test Advisor 14 For Coverity and Partner use only. Copyright Coverity, Inc., 2013 Improves unit testing effectiveness and efficiency Focus testing time where it matters And don’t waste time writing tests you don’t need High Risk Code
  • 15. Creating an enforceable process The high priority code that must be thoroughly tested Define Analyze Remediate Govern Code that has changed and been impacted by changes Code that has been insufficiently tested Manage progress to improve test coverage 15
  • 16. What code needs to be tested based on your high risk criteriaDefine Analyze Remediate Govern With patent-pending techniques based on code behavior and change impact Surface issues in your workflow and efficiently manage to closure Create a testing stage gate and enforce developer accountability Generic example: • All code changes for next release … • And code impacted by those changes … • Must have 100% coverage … • Not counting exception handling and debug code Policy definition example 16 For Coverity and Partner use only. Copyright Coverity, Inc., 2013 ACE: • Focus on core components • Only called code (ignore uncalled) • Exclude debug, logging code • Exclude error cases, prioritize violations by release • Exclude error code • Prioritize violations by release
  • 17. Remediate Testing Violations Unified view of test violations and defects Unified view of test violations and defects Clear description of the violation Clear description of the violation Automatically assign violations to owners Automatically assign violations to owners Identify where the violation occurs in the code Identify where the violation occurs in the code For Coverity and Partner use only. Copyright Coverity, Inc., 2013 17
  • 18. Coverity Connect Coverity SAVE™ Static Analysis Verification Engine Test Advisor SDLC Integrations Test Execution Third Party Metrics Code Coverage SCM Coverity Development Testing Platform 18 For Coverity and Partner use only. Copyright Coverity, Inc., 2013 Security Advisor Test Advisor Analysis Packs Coverity SAVE™ Static Analysis Verification Engine Policy Manager Quality Advisor Architecture Analysis Dynamic Analysis Analysis Integration FindBugs™ | FxCop Analysis Integration Toolkit Coverity Connect Test Execution Third Party Metrics Build/ Continuous Integration ALM HP | IBM IDE Code Coverage Defect Tracking SCM Proprietary Code | Open Source Code (Scan)
  • 19. Development Testing Maturity Model 19 For Coverity and Partner use only. Copyright Coverity, Inc., 2013 Development Testing Adoption IntegrationintoSDLC Level 1 Detection of critical quality and security defects as part of SW build process. No new defects introduced. Level 3 Integration into the existing SDLC using a common workflow for all defects and test effectiveness issues. Level 4 Establish and enforce consistent source code quality and security policies. Establish source code acceptance criteria. Level 5 All legacy defects eliminated, build fails if new defects are introduced. All critical code and code impacted by change is tested. High High Level 2 Identification of areas of risk caused by insufficient automated testing. Ensure critical code is prioritized and tested.
  • 20. Q & A 20 For Coverity and Partner use only. Copyright Coverity, Inc., 2012 • Remedy IT: www.remedy.nl • ACE: www.cs.wustl.edu/~schmidt/ACE.html • Coverity: coverity.com • Coverity Scan: scan.coverity.com