Presentation on the "Business Value of Agile Testing: Using Test Driven Development, Continuous Integration, Continuous Delivery, & DevOps," which are highly-disciplined contemporary new product development (NPD) approaches for rapidly building high-quality information technology-intensive systems. Identifies the motivation for agile methods, provide a brief introduction to agile methods, describe the fundamental mechanics of agile methods, and a brief survey of the benefits of agile methods as reported by major industry studies (including rarely seen, late-breaking economic data and results from the top consulting firms). Defines agile testing and introduce basic and advanced agile testing practices, strategies, metrics, outcomes, costs & benefits, cost of quality, and statistical performance data. Introduces basic and advanced agile scaling practices, case studies of enterprise-level agile testing, Continuous Delivery, and DevOps at major Internet firms, and common agile testing tools and automation suites. Closes with a summary of agile testing adoption rates, common barriers to agile testing, organizational change models for agile testing, and a summary of the benefits of agile testing.
Business Value of Agile Testing: Using TDD, CI, CD, & DevOps
1. Business Value of
Agile Testing
Using TDD, CI, CD & DevOps
Dr. David F. Rico, PMP, CSEP, FCP, FCT, ACP, CSM, SAFe
Twitter: @dr_david_f_rico
Website: http://www.davidfrico.com
LinkedIn: http://www.linkedin.com/in/davidfrico
Agile Capabilities: http://davidfrico.com/rico-capability-agile.pdf
Agile Resources: http://www.davidfrico.com/daves-agile-resources.htm
Agile Cheat Sheet: http://davidfrico.com/key-agile-theories-ideas-and-principles.pdf
2. Author Background
ī¯ Govât contractor with 32+ years of IT experience
ī¯ B.S. Comp. Sci., M.S. Soft. Eng., & D.M. Info. Sys.
ī¯ Large govât projects in U.S., Far/Mid-East, & Europe
2
ī
ī Career systems & software engineering methodologist
ī Lean-Agile, Six Sigma, CMMI, ISO 9001, DoD 5000
ī NASA, USAF, Navy, Army, DISA, & DARPA projects
ī Published seven books & numerous journal articles
ī Intnâl keynote speaker, 130 talks to 12,000+ people
ī Specializes in metrics, models, & cost engineering
ī Cloud Computing, SOA, Web Services, FOSS, etc.
ī Adjunct at five Washington, DC-area universities
4. Software in U.S. DoD Systems
Kennedy, M. P., & Umphress, D. A. (2011). An agile systems engineering process: The missing link. Crosstalk, 24(3), 16-20.
ī¯ No. of software-intensive systems is growing
ī¯ 80% of US DoD functions performed in software
ī¯ Major driver of cost, schedule, & tech. performance
4
ī
5. Software in U.S. DoD Avionics
Blackburn, M. R. (2014). Transforming systems engineering through a holistic approach to model centric engineering. Washington, DC: Stevens Institute of Technology.
ī¯ Software in U.S. DoD avionics growing exponentially
ī¯ 10x growth from F-16 to F-22 (& another 10x to F-35)
ī¯ Productivity must grow by 10x for next gen systems
5
ī
6. Traditional Projects
6
ī¯ Big projects result in poor quality and scope changes
ī¯ Productivity declines with long queues/wait times
ī¯ Large projects are unsuccessful or canceled
Jones, C. (1991). Applied software measurement: Assuring productivity and quality. New York, NY: McGraw-Hill.
Size vs. Quality
DEFECTS
0.00
3.20
6.40
9.60
12.80
16.00
0 2 6 25 100 400
SIZE
Size vs. Productivity
PRODUCTIVITY
0.00
1.00
2.00
3.00
4.00
5.00
0 2 6 25 100 400
SIZE
Size vs. Change
CHANGE
0%
8%
16%
24%
32%
40%
0 2 6 25 100 400
SIZE
Size vs. Success
SUCCESS
0%
12%
24%
36%
48%
60%
0 2 6 25 100 400
SIZE
ī
7. Global Project Failures
7
Standish Group. (2015). Chaos summary 2015. Boston, MA: Author.
Sessions, R. (2009). The IT complexity crisis: Danger and opportunity. Houston, TX: Object Watch.
ī¯ Challenged and failed projects hover at 67%
ī¯ Big projects fail more often, which is 5% to 10%
ī¯ Of $1.7T spent on IT projects, over $858B were lost
$0.0
$0.4
$0.7
$1.1
$1.4
$1.8
2002 2003 2004 2005 2006 2007 2008 2009 2010
Trillions(USDollars)
Expenditures Failed Investments
ī
0% 20% 40% 60% 80% 100%
28%
34%
29%
35%
32%
33%
27%
28%
29%
49%
51%
53%
46%
44%
41%
56%
55%
52%
23%
15%
18%
19%
24%
26%
17%
17%
19%
2000
2002
2004
2006
2008
2010
2012
2014
2015
Year
Successful Challenged Failed
8. Requirements Defects & Waste
8
Sheldon, F. T. et al. (1992). Reliability measurement: From theory to practice. IEEE Software, 9(4), 13-20
Johnson, J. (2002). ROI: It's your job. Extreme Programming 2002 Conference, Alghero, Sardinia, Italy.
ī¯ Requirements defects are #1 reason projects fail
ī¯ Traditional projects specify too many requirements
ī¯ More than 65% of requirements are never used at all
Other 7%
Requirements
47%
Design
28%
Implementation
18%
Defects
Always 7%
Often 13%
Sometimes
16%
Rarely
19%
Never
45%
Waste
ī
9. What is Agility?
ī¯ A-gil-i-ty (Ķ-'ji-lĶ-tÄ) Property consisting of quickness,
lightness, and ease of movement; To be very nimble
īŽ The ability to create and respond to change in order to
profit in a turbulent global business environment
īŽ The ability to quickly reprioritize use of resources when
requirements, technology, and knowledge shift
īŽ A very fast response to sudden market changes and
emerging threats by intensive customer interaction
īŽ Use of evolutionary, incremental, and iterative delivery
to converge on an optimal customer solution
īŽ Maximizing BUSINESS VALUE with right sized, just-
enough, and just-in-time processes and documentation
Highsmith, J. A. (2002). Agile software development ecosystems. Boston, MA: Addison-Wesley.
9
ī ī
10. What are Agile Methods?
10
ī¯ People-centric way to create innovative solutions
ī¯ Product-centric alternative to documents/process
ī¯ Market-centric model to maximize business value
Agile Manifesto. (2001). Manifesto for agile software development. Retrieved September 3, 2008, from http://www.agilemanifesto.org
Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods. Ft. Lauderdale, FL: J. Ross Publishing.
Rico, D. F. (2012). Agile conceptual model. Retrieved February 6, 2012, from http://davidfrico.com/agile-concept-model-1.pdf
Customer Collaboration
Working Systems & Software
Individuals & Interactions
Responding to Change
valued
more than
valued
more than
valued
more than
valued
more than
Contracts
Documentation
Processes
Project Plans
īˇ Frequent comm.
īˇ Close proximity
īˇ Regular meetings
īˇ Multiple comm. channels
īˇ Frequent feedback
īˇ Relationship strength
īˇ Leadership
īˇ Boundaries
īˇ Empowerment
īˇ Competence
īˇ Structure
īˇ Manageability/Motivation
īˇ Clear objectives
īˇ Small/feasible scope
īˇ Acceptance criteria
īˇ Timeboxed iterations
īˇ Valid operational results
īˇ Regular cadence/intervals
īˇ Org. flexibility
īˇ Mgt. flexibility
īˇ Process flexibility
īˇ System flexibility
īˇ Technology flexibility
īˇ Infrastructure flexibility
īˇ Contract compliance
īˇ Contract deliverables
īˇ Contract change orders
īˇ Lifecycle compliance
īˇ Process Maturity Level
īˇ Regulatory compliance
īˇ Document deliveries
īˇ Document comments
īˇ Document compliance
īˇ Cost Compliance
īˇ Scope Compliance
īˇ Schedule Compliance
ī
ī
ī
ī
Courage
ī
11. Agile World View
ī¯ âAgilityâ has many dimensions other than IT
ī¯ It ranges from leadership to technological agility
ī¯ Todayâs focus is on organizational & enterprise agility
ī ī
Agile Leaders
Agile Organization Change
Agile Acquisition & Contracting
Agile Strategic Planning
Agile Capability Analysis
Agile Program Management
Agile Tech.
Agile Information Systems
Agile Tools
Agile Processes & Practices
Agile Systems Development
Agile Project Management
11
ī
12. Network
Computer
Operating System
Middleware
Applications
APIs
GUI
How Agile Works
ī¯ Agile requirements implemented in slices vs. layers
ī¯ User needs with higher business value are done first
ī¯ Reduces cost & risk while increasing business success
12Shore, J. (2011). Evolutionary design illustrated. Norwegian Developers Conference, Oslo, Norway.
Agile Traditional
1 2 3īˇ Faster
īˇ Early ROI
īˇ Lower Costs
īˇ Fewer Defects
īˇ Manageable Risk
īˇ Better Performance
īˇ Smaller Attack Surface
Late īˇ
No Value īˇ
Cost Overruns īˇ
Very Poor Quality īˇ
Uncontrollable Risk īˇ
Slowest Performance īˇ
More Security Incidents īˇSeven Wastes
1. Rework
2. Motion
3. Waiting
4. Inventory
5. Transportation
6. Overprocessing
7. Overproduction
MINIMIZES MAXIMIZES
īˇ JIT, Just-enough architecture
īˇ Early, in-process system V&V
īˇ Fast continuous improvement
īˇ Scalable to systems of systems
īˇ Maximizes successful outcomes
īˇ Myth of perfect architecture
īˇ Late big-bang integration tests
īˇ Year long improvement cycles
īˇ Breaks down on large projects
īˇ Undermines business success
ī
13. Thousands of Tests
Continuously Executed
No More Late Big
Bang Integration
ī¯ User needs designed & developed one-at-a-time
ī¯ Changes automatically detected, built, and tested
ī¯ System fully tested and deployed as changes occur
13Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.
Build
Integration
Server
Version
Control
Server
Build
Scripts
UsesWatches
Build
Status
ProvidesDeveloper A
Developer B
Developer C
Commits
Changes
Commits
Changes
Commits
Changes
Builds
Database
Analysis
Testing
Reporting
Documentation
Deployment
Early, Automated, Fast,
Efficient, & Repeatable
Constant Readiness
State & CM Control
Lean, Waste Free, Low WIP,
No Deadlocked Test Queues
Rapidly & Successfully
Dev. Complex Systems
ī
Basic Agile Mechanics
14. 14
Capability/MMF #1
â Feature 1
â Feature 2
â Feature 3
â Feature 4
â Feature 5
â Feature 6
â Feature 7
Capability/MMF #2
â Feature 8
â Feature 9
â Feature 10
â Feature 11
â Feature 12
â Feature 13
â Feature 14
Capability/MMF #3
â Feature 15
â Feature 16
â Feature 17
â Feature 18
â Feature 19
â Feature 20
â Feature 21
Capability/MMF #4
â Feature 22
â Feature 23
â Feature 24
â Feature 25
â Feature 26
â Feature 27
â Feature 28
Capability/MMF #5
â Feature 29
â Feature 30
â Feature 31
â Feature 32
â Feature 33
â Feature 34
â Feature 35
Capability/MMF #6
â Feature 36
â Feature 37
â Feature 38
â Feature 39
â Feature 40
â Feature 41
â Feature 42
Capability/MMF #7
â Feature 43
â Feature 44
â Feature 45
â Feature 46
â Feature 47
â Feature 48
â Feature 49
1
2 3
4
5 6
7
8 9
10
11 12
13
14 15
16
17 18
19
20 21
Evolving âUnified/Integratedâ Enterprise Data Model
âDisparateâ LEGACY SYSTEM DATABASES (AND DATA MODELS)
ETL
A A
B C
D E F
G H I J K
A
B C
D E F
A
B C
D E
A
B C
D
A
B C
A
B
âLegacyâ MS SQL Server Stovepipes âInter-Departmentalâ Linux Blade/Oracle/Java/WebSphere Server
âLeasedâ DWA/HPC/Cloud Services
Sprint 1 Sprint 2 Sprint 3 Sprint 4 Sprint 5 Sprint 6 Sprint 7
Release
Release
Release
Release
ETL ETL ETL ETL ETL ETL
Bente, S., Bombosch, U., & Langade, S. (2012). Collaborative enterprise architecture: Enriching EA with lean, agile, and enterprise 2.0 practices. Waltham, MA: Elsevier.
(for example, assume 25 user stories per feature, 175 user stories per capability/MMF, and 1,225 user stories total)
ī¯ Organize needs into capabilities, features, and stories
ī¯ Prioritize features, group releases, and initiate sprints
ī¯ Develop minimum set of features with highest valueī
Agile Systems Development
15. Models of AGILE DEVELOPMENT
15
ī¯ Agile methods spunoff flexible manufacturing 1990s
ī¯ Extreme Programming (XP) swept the globe by 2002
ī¯ Today, over 90% of IT projects use Scrum/XP hybrid
īˇUse Cases
īˇDomain Model
īˇObject Oriented
īˇIterative Dev.
īˇRisk Planning
īˇInfo. Radiators
īˇPlanning Poker
īˇProduct Backlog
īˇSprint Backlog
īˇ2-4 Week Spring
īˇDaily Standup
īˇSprint Demo
īˇFeasibility
īˇBusiness Study
īˇFunc. Iteration
īˇDesign Iteration
īˇImplementation
īˇTesting
īˇDomain Model
īˇFeature List
īˇObject Oriented
īˇIterative Dev.
īˇCode Inspection
īˇTesting
īˇRelease Plans
īˇUser Stories
īˇPair Programmer
īˇIterative Dev.
īˇTest First Dev.
īˇOnsite Customer
Cockburn, A. (2002). Agile software development. Boston, MA: Addison-Wesley.
Schwaber, K., & Beedle, M. (2001). Agile software development with scrum. Upper Saddle River, NJ: Prentice-Hall.
Stapleton, J. (1997). DSDM: A framework for business centered development. Harlow, England: Addison-Wesley.
Palmer, S. R., & Felsing, J. M. (2002). A practical guide to feature driven development. Upper Saddle River, NJ: Prentice-Hall.
Beck, K. (2000). Extreme programming explained: Embrace change. Reading, MA: Addison-Wesley.
CRYSTAL METHODS
- 1991 -
SCRUM
- 1993 -
DSDM
- 1993 -
FDD
- 1997 -
XP
- 1998 -
īˇReflection W/S īˇRetrospective īˇQuality Control īˇQuality Control īˇContinuous Del.
ī
16. Basic SCRUM Framework
Schwaber, K., & Beedle, M. (2001). Agile software development with scrum. Upper Saddle River, NJ: Prentice-Hall.
ī¯ Created by Jeff Sutherland at Easel in 1993
ī¯ Product backlog comprised of prioritized features
ī¯ Iterative sprint-to-sprint, adaptive & emergent model
16
17. Models of AGILE PROJECT MGT.
17
ī¯ Dozens of Agile project management models emerged
ī¯ Many stem from principles of Extreme Programming
ī¯ Vision, releases, & iterative development common
īˇPrioritization
īˇFeasibility
īˇPlanning
īˇTracking
īˇReporting
īˇReview
īˇVisionate
īˇSpeculate
īˇInnovate
īˇRe-Evaluate
īˇDisseminate
īˇTerminate
īˇScoping
īˇPlanning
īˇFeasibility
īˇCyclical Dev.
īˇCheckpoint
īˇReview
īˇEnvision
īˇSpeculate
īˇExplore
īˇIterate
īˇLaunch
īˇClose
īˇVision
īˇRoadmap
īˇRelease Plan
īˇSprint Plan
īˇDaily Scrum
īˇRetrospective
Thomsett, R. (2002). Radical project management. Upper Saddle River, NJ: Prentice-Hall.
DeCarlo, D. (2004). Extreme project management: Using leadership, principles, and tools to deliver value in the face of volatility. San Francisco, CA: Jossey-Bass.
Wysocki, R.F. (2010). Adaptive project framework: Managing complexity in the face of uncertainty. Boston, MA: Pearson Education.
Highsmith, J. A. (2010). Agile project management: Creating innovative products. Boston, MA: Pearson Education.
Layton, M. C., & Maurer, R. (2011). Agile project management for dummies. Hoboken, NJ: Wiley Publishing.
RADICAL
- 2002 -
EXTREME
- 2004 -
ADAPTIVE
- 2010 -
AGILE
- 2010-
SIMPLIFIED
- 2011 -
ī
18. Layton, M. C., & Maurer, R. (2011). Agile project management for dummies. Hoboken, NJ: Wiley Publishing.
ī¯ Created by Mark Layton at PlatinumEdge in 2012
ī¯ Mix of new product development, XP, and Scrum
ī¯ Simplified codification of XP and Scrum hybrid
18
Simplified AGILE PROJECT MGT.
19. 19
ī¯ Numerous models of agile portfolio mgt. emerging
ī¯ Based on lean-kanban, release planning, and Scrum
ī¯ Include organization, program, & project management
Schwaber, K. (2007). The enterprise and scrum. Redmond, WA: Microsoft Press.
Leffingwell, D. (2007). Scaling software agility: Best practices for large enterprises. Boston, MA: Pearson Education.
Larman, C., & Vodde, B. (2008). Scaling lean and agile development: Thinking and organizational tools for large-scale scrum. Boston, MA: Addison-Wesley.
Ambler, S. W., & Lines, M. (2012). Disciplined agile delivery: A practitioner's guide to agile software delivery in the enterprise. Boston, MA: Pearson Education.
Thompson, K. (2013). cPrimeâs R.A.G.E. is unleashed: Agile leaders rejoice! Retrieved March 28, 2014, from http://www.cprime.com/tag/agile-governance
Schwaber, K. (2015). The definitive guide to nexus: The exoskeleton of scaled scrum development. Lexington, MA: Scrum.Org
ī
Models of AGILE PORTFOLIO MGT.
ESCRUM
- 2007 -
SAFe
- 2007 -
LESS
- 2007 -
DAD
- 2012 -
RAGE
- 2013 -
SPS
- 2015 -
īˇProduct Mgt
īˇProgram Mgt
īˇProject Mgt
īˇProcess Mgt
īˇBusiness Mgt
īˇMarket Mgt
īˇStrategic Mgt
īˇPortfolio Mgt
īˇProgram Mgt
īˇTeam Mgt
īˇQuality Mgt
īˇDelivery Mgt
īˇBusiness Mgt
īˇPortfolio Mgt
īˇProduct Mgt
īˇArea Mgt
īˇSprint Mgt
īˇRelease Mgt
īˇBusiness Mgt
īˇPortfolio Mgt
īˇInception
īˇConstruction
īˇIterations
īˇTransition
īˇBusiness
īˇGovernance
īˇPortfolio
īˇProgram
īˇProject
īˇDelivery
īˇProduct Mgt
īˇProgram Mgt
īˇSprint Mgt
īˇTeam Mgt.
īˇInteg Mgt.
īˇRelease Mgt
20. Scaled Agile Framework (SAFE)
ī¯ Created by Dean Leffingwell of Rally in 2007
ī¯ Knowledge to scale agile practices to enterprise
ī¯ Hybrid of Kanban, XP release planning, and Scrum
20
Leffingwell, D. (2007). Scaling software agility: Best practices for large enterprises. Boston, MA: Pearson Education.
ī
21. 21
Agile Performance MeasurementWork(Story,Point,Task)orEffort(Week,Day,Hour)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Burndown
Work(Story,Point,Task)orEffort(Week,Day,Hour)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Cumulative Flow
Work(Story,Point,Task)orEffort(Week,Day,Hour)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Earned Value Management - EVM CPI
SPI
PPC
APC
Work(Story,Point,Task)orEffort(Week,Day,Hour)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Earned Business Value - EBV
22. What is Agile Testing?
ī¯ Traditional testing is a late, manual process
ī¯ Agile testing is an early and automated process
ī¯ Goal to deliver early & often and V&V components
22
Rico, D. F. (2012). Agile testing resources. Retrieved Sep. 9, 2012, from http://davidfrico.com/agile-testing-resources.txt
Crispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.
Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.
ī
AGILE TESTING
- Early Incremental Testing -
TRADITIONAL TESTING
- Late Big Bang Integration Testing -
Test Criteria Accompany Stories
īˇAutomated Tests Written First
īˇUnits Coded-Tested One at Time
īˇCode is Frequently Checked In
īˇCode Automatically Retrieved
īˇCode Automatically Compiled
īˇTests Automatically Executed
īˇInstant Feedback & Test Reports
Test Criteria Written After Fact
īˇManual Tests Written Much Later
īˇUnits Coded Late All at One Time
īˇCode Checked In Late in Project
īˇCode Manually Submitted to Test
īˇCode Manually Compiled & Built
īˇTests Manually Executed Late
īˇLate Project Feedback & Reports
īˇīˇ
īˇCode Automatically DeployedīˇLate Defects Freeze Projects
23. BASICâTest Driven Development
ī¯ Term coined by Kent Beck in 2003
ī¯ Consists of writing all tests before design
ī¯ Ensures all components are verified and validated
23Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.
24. ADVANCEDâContinuous Integration
ī¯ Term coined by Martin Fowler in 1998
ī¯ Process of automated build/regression testing
ī¯ Evaluates impact of changes against entire system
24Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
īŊ ALL DEVELOPERS RUN PRIVATE BUILDS
īŊ DEVELOPERS COMMIT CODE TO VERSION CONTROL
īŊ INTEGRATION BUILDS OCCUR SEVERAL TIMES PER DAY
īŊ 100% OF SYSTEM TESTS MUST PASS FOR EVERY BUILD
īŊ A SHIPPABLE PRODUCT RESULTS FROM EVERY BUILD
īŊ FIXING BROKEN BUILDS IS OF THE HIGHEST PRIORITY
īŊ REPORTS AUTOMATICALLY GENERATED & REVIEWED
25. ī¯ Agile testing consists of seven broad practices
ī¯ Automated build, database, inspection, tests, etc.
ī¯ Include reporting, documentation, deployment, etc.
25
Practice
Building
Database
Inspections
Testing
Feedback
Documentation
Deployment
Description
Frequently assembling products and services to ensure delivery readiness
Frequently generating/analyzing database schemas, queries, and forms
Frequently performing automated static analysis of product/service quality
Frequently performing automated dynamic product and service evaluation
Frequently generating automated status reports/messages for all stakeholders
Frequently performing automated technical/customer document generation
Frequently performing automated delivery of products/services to end users
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.
ī
PRACTICESâContinuous Integration
26. ī¯ Created by Jez Humble of ThoughtWorks in 2011
ī¯ Includes CM, build, testing, integration, release, etc.
ī¯ Goal is one-touch automation of deployment pipeline
26
Humble, J., & Farley, D. (2011). Continuous delivery. Boston, MA: Pearson Education.
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration. Boston, MA: Addison-Wesley.
Ohara, D. (2012). Continuous delivery and the world of devops. San Francisco, CA: GigaOM Pro.
ī
ī
CoQ
âĸ 80% MS Tst
âĸ 8/10 No Val
âĸ $24B in 90s
âĸ Rep by CD
âĸ Not Add MLK
ENTERPRISEâContinuous Delivery
27. ī¯ Created by Patrick Debois of Jedi BVBA in 2007
ī¯ Collaboration of developers & infrastructure people
ī¯ Goal to automate the deployment to end-user devices
27
Bass, L., Weber, I., & Zhu, L. (2015). Devops: A software architect's perspective. Old Tappan, NJ: Pearson Education.
Gruver, G., & Mouser, T. (2015). Leading the transformation: Applying agile and devops at scale. Portland, OR: IT Revolution Press.
Humble, J., Molesky, J., & O'Reilly, B. (2015). Lean enterprise: How high performance organizations innovate at scale. Sebastopol, CA: O'Reilly Media.
ī
GLOBALâDevelopment Operations
ī
28. ī¯ Agile methods are based on traditional measures
ī¯ Story points, velocity, and burndown basic metrics
ī¯ Experts use Agile EVM, test, ROI & portfolio metrics
28Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods. Ft. Lauderdale, FL: J. Ross Publishing.
AGILE METRICS
1. Agile CODE Metrics
2. Agile PROJECT Metrics
3. Agile TRACKING Metrics
4. Agile TESTING Metrics
5. Agile VALUE Metrics
6. Agile HEALTH Metrics
7. Agile PORTFOLIO Metrics
1. Agile CODE Metrics
īˇ Code Size
īˇ Code Complexity
īˇ Object Oriented
īˇ Code Coverage
īˇ Code Defects
īˇ Relational Design
2. Agile PROJECT Metrics
īˇ Software Size
īˇ Software Productivity
īˇ Software Effort
īˇ Software Quality
īˇ Software Schedule
īˇ Software Success
3. Agile TRACKING Metrics
īˇ Story Points
īˇ Sprint Burndown
īˇ Release Burndown
īˇ Velocity
īˇ Feature Progress
īˇ Agile Earned Value
4. Agile TESTING Metrics
īˇ Test Coverage
īˇ Test Automation
īˇ Integration Builds
īˇ Running Tested Features
īˇ DevOps Automation
īˇ Deployment Frequency
7. Agile PORTFOLIO Metrics
īˇ Portfolio Kanban
īˇ Epic Progress
īˇ Portfolio Radar
īˇ Release Train Radar
īˇ Lean Portfolio Metrics
īˇ Enterprise Scorecard
6. Agile HEALTH Metrics
īˇ Teamwork Quality
īˇ Collaboration Quality
īˇ Agile Process Maturity
īˇ Agile Adoption Rate
īˇ Degree of Agility
īˇ Product Flexibility
5. Agile VALUE Metrics
īˇ Total Lifecycle Costs
īˇ Total Lifecycle Benefits
īˇ Benefit to Cost Ratio
īˇ Return on Investment
īˇ Net Present Value
īˇ Real Options Analysis
Agile Testing MetricsâTaxonomy
29. 29
METRIC DESCRIPTION
TEST COVERAGE Percent or degree to which software source code is tested
TEST AUTOMATION Ratio or degree to which software tests are automated
INTEGRATION BUILDS Frequency of automated software builds and integrations
RUNNING TESTED FEATURES Number of completed and tested features or user stories
DEVOPS AUTOMATION Ratio or degree to which deployments are automated
DEPLOYMENT FREQUENCY Frequency of automated software deployments or deliveries
ī¯ Software test automation emerged during the 1970s
ī¯ Reached their height in personal computer (PC) era
ī¯ Most are FOSS and used by successful agile teams
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
Agile Testing MetricsâDefinitions
31. 31
Traditional vs. Agile Cumulative Flow
Work(Story,Point,Task)orEffort(Week,Day,Hour)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Work(Story,Point,Task)orEffort(Week,Day,Hour)
Time Unit (Roadmap, Release, Iteration, Month, Week, Day, Hour, etc.)
Traditional Cumulative Flow Agile Cumulative Flow
ī¯ Late big bang integration increases WIP backlog
ī¯ Agile testing early and often reduces WIP backlog
ī¯ Improves workflow and reduces WIP & lead times
Anderson, D. J. (2004). Agile management for software engineering. Upper Saddle River, NJ: Pearson Education.
Anderson, D. J. (2010). Kanban: Successful evolutionary change for your technology business. Sequim, WA: Blue Hole Press.
ī
Agile TestingâWorkflow
32. ī¯ Fewer integrations leave in higher bug counts
ī¯ Frequent, early integrations eliminate most defects
ī¯ Goal is to have as many early integrations as possible
32
Lacoste, F. J. (2009). Killing the gatekeeper: Introducing a continuous integration system. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA, 387-392.
ī ī
Number of
Integrations
Less Defects
âĸMore Integrations
âĸEarly IntegrationsMore Defects
âĸFew Integrations
âĸLate Integrations
ī
Agile TestingâEconomic Drivers
33. ī¯ Traditional testing finds a defect in about 10 hours
ī¯ Manual code inspections find a defect in 1 hour
ī¯ Agile testing finds a defect every 6 minutes
33
Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.
ī
Agile TestingâEconomics
34. ī¯ Agile testing is 10x better than code inspections
ī¯ Agile testing is 100x better than traditional testing
ī¯ Agile testing is done earlier âandâ 1,000x more often
34
Rico, D. F. (2012). The Cost of Quality (CoQ) for Agile vs. Traditional Project Management. Fairfax, VA: Gantthead.Com.
ī
Agile TestingâCost of Quality
35. Agile Cost & Benefit Analysis
ī¯ Costs based on avg. productivity and quality
ī¯ Productivity ranged from 4.7 to 5.9 LOC an hour
ī¯ Costs were $588,202 and benefits were $3,930,631
35
Rico, D. F., Sayani, H. H., & Sone, S. (2009). The business value of agile software methods: Maximizing ROI with just-in-time processes and documentation.
Ft. Lauderdale, FL: J. Ross Publishing.
d1 = [ln(Benefits ī¸ Costs) + (Rate + 0.5 ī´ Risk2) ī´ Years] ī¸ Risk ī´ ī Years, d2 = d1 ī Risk ī´ ī Years
īĨ īŊ
5
1i
ī
36. Benefits of Agile Methods
ī¯ Analysis of 23 agile vs. 7,500 traditional projects
ī¯ Agile projects are 54% better than traditional ones
ī¯ Agile has lower costs (61%) and fewer defects (93%)
Mah, M. (2008). Measuring agile in the enterprise: Proceedings of the Agile 2008 Conference, Toronto, Canada.
Project Cost in Millions $
0.75
1.50
2.25
3.00
2.8
1.1
Before Agile
After Agile
61%
Lower
Cost
Total Staffing
18
11
Before Agile
After Agile
39%
Less
Staff
5
10
15
20
Delivery Time in Months
5
10
15
20
18
13.5
Before Agile
After Agile
24%
Faster
Cumulative Defects
625
1250
1875
2500
2270
381
Before Agile
After Agile
93%
Less
Defects
36
ī
ī
ī
ī
ī
37. Agile vs. Traditional Success
ī¯ Traditional projects succeed at 50% industry avg.
ī¯ Traditional projects are challenged 20% more often
ī¯ Agile projects succeed 3x more and fail 3x less often
Standish Group. (2012). Chaos manifesto. Boston, MA: Author.
37
Agile Traditional
Success
42%
Failed
9%
Challenged
49%
Success
14%
Failed
29%
Challenged
57%
ī
38. Grant, T. (2005). Continuous integration using cruise control. Northern Virginia Java Users Group (Novajug), Reston, Virginia, USA.
Fredrick, J. (2008). Accelerate software delivery with continuous integration and testing. Japanese Symposium on Software Testing, Tokyo, Japan.
ī¯ Most agile testing tools are âfreeâ open source
ī¯ Build server costs no more than a commodity PC
ī¯ 10x more efficient/effective than traditional testing
38
ī
Agile TestingâCI Statistics
39. 39
ī¯ Hewlett-Packard is a major user of CI, CD, & DevOps
ī¯ 400 engineers developed 10 million LOC in 4 years
ī¯ Major gains in testing, deployment, & innovation
Gruver, G., Young, M. & Fulghum, P. (2013). A practical approach to large-scale agile development. Upper Saddle River, NJ: Pearson Education.
ī
TYPE METRIC MANUAL DEVOPS MAJOR GAINS
CYCLE TIME
IMPROVEMENTS
Build Time 40 Hours 3 Hours 13 x
No. Builds 1-2 per Day 10-15 per Day 8 x
Feedback 1 per Day 100 per Day 100 x
Regression Testing 240 Hours 24 Hours 10 x
DEVELOPMENT
COST EFFORT
DISTRIBUTION
Integration 10% 2% 5 x
Planning 20% 5% 4 x
Porting 25% 15% 2 x
Support 25% 5% 5 x
Testing 15% 5% 3 x
Innovation 5% 40% 8 x
Agile TestingâCD Statistics
40. ī¯ Assembla went from 2 to 45 releases every month
ī¯ 15K Google developers run 120 million tests per day
ī¯ 30K+ Amazon developers deliver 8,600 releases a day
40Singleton, A. (2014). Unblock: A guide to the new continuous agile. Needham, MA: Assembla, Inc.
ī
62x Faster
U.S. DoD
IT Project
3,645x Faster
U.S. DoD
IT Project
ī
Agile TestingâDevOps Statistics
41. ī¯ Google early adopter of agile methods and Scrum
ī¯ Google also uses agile testing at enterprise scale
ī¯ 15,000 developers run 120 million tests per day
41
Micco, J. (2013). Continuous integration at google scale. Eclipse Con, Boston, MA.
Whittaker, J., Arbon, J., & Carollo, J. (2012). How google tests software. Upper Saddle River, NJ: Pearson Education.
īˇ 440 billion unique users run 37 trillion searches each year
īˇ Single monolithic code tree with mixed language code
īˇ Submissions at head â One branch â All from source
īˇ 20+ code changes/minute â 50% code change/month
īˇ 5,500+ submissions/day â 120 million tests per day
īˇ 80,000 builds per day â 20 million builds per year
īˇ Auto code inspections â For low defect density
īˇ 10X programming productivity improvement
īˇ $150 million in annual labor savings (ROI as a result)
ī
ī
ī
Agile TestingâGoogle Statistics
42. ī¯ Amazon adopted agile in 1999 and Scrum in 2004
ī¯ Using enterprise-scale continuous delivery by 2010
ī¯ 30,000+ developers deploy over 8,600 releases a day
42
Atlas, A. (2009). Accidental adoption: The story of scrum at amazon.com. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA, 135-140.
Jenkins, J. (2011). Velocity culture at amazon.com. Proceedings of the Velocity 2011 Conference, Santa Clara, California, USA.
Elisha, S. (2013). Continuous deployment with amazon web services. Proceedings of the AWS Summit 2013, Sydney, New South Wales, Australia.
īˇ Software deployment every 11.6 seconds (as of 2011)
īŽ 24,828 to 86,320 releases per Iteration
īŽ 161,379 to 561,080 releases per Quarter
īŽ 645,517 to 2,244,320 releases per Year
īˇ Automatic, split-second roll-forward & backward
īˇ 75-90% reduction in release-caused outages (0.001%)
īˇ Millions of times faster (than traditional methods)
īŽ 4,357,241 to 15,149,160 per traditional release
īˇ Thousands of times faster (than manual agility)
īŽ 161,379 to 561,080 per Scrum/SAFe release
īˇ Used agile methods long before U.S. government (1999)
ī
ī
ī
ī
ī
Agile TestingâAmazon Statistics
43. ī¯ Enables enterprises to be flexible but disciplined
ī¯ Allows enterprises to distribute project work teams
ī¯ Ensures distributed project teams are collaborating
43
Agile Tools
âAcross the Life Cycleâ
Project Management
Requirements
DOORS
Requisite Pro
SLATE
Design
Rhapsody
Telelogic System Architect
Rational System Architect
Coding
Eclipse
Visual Studio
Sun Studio
Testing
JUnit
NUnit
Xunit
CPPUnit
Gtest
Fit
Fitnesse
Selenium
Quality Assurance
CheckStyle
PMD
EMMA
Jdepend
Cobertura
Gcov
Configuration Mgt
Subversion (SVN)
Concurrent Versions Sys.
ClearCase
Build Automation
Ant
NAnt
Maven
Make
Continuous Integ.
Cruise Control
Hudson
BuildBot
Collaboration
WebEx
Skype
MeetMe
Wimba
Wiki
MediaWiki
TracWiki
PhpWiki
Documentation
NDoc
Javadoc
Doxygen
iText
Version One
Rally
Scrum Works
VSTS
Agile Team
Agile Enterprise
Scope Manager
Story Studio
XP Plan It
Iterate
XP Tracker
Agilo
XP CGI
XP Web
Xplanner
Ice Scrum
Project Cards
Target Process
Xtreme Planner
Team System
Community
Enterprise
Mingle
Hansoft
ī
44. ī¯ There are literally hundreds of agile testing tools
ī¯ There are tools for building, testing, and deployment
ī¯ Integration tools monitor repositories and initiate tests
44
Agile Tools
âIn-Depth Test Automationâ
Smart, J. (2009). Automated deployment with maven and friends: Going the whole nine yards. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.
ī
45. ī¯ Simple example of a DevOps reference architecture
ī¯ Includes CM, continuous integration, & deployment
ī¯ Code automatically built/tested/deployed to users
45
Agile Tools
âSimple DevOps Automationâ
Morris, B., & Cassatt, C. (2015). Devops for the rest of us. Proceedings of the Agile DC Conference, Washington, DC, USA.
Weeks, D. E. (2014). Devops and continuous delivery reference architectures (volume 1 & 2). Fulton, MD: Sonatype.
ī
46. 46
Agile Tools
âPeriodic Table of DevOps Automationâ
XeniaLabs. (2016). Periodic table of devops tools. Retrieved April 11, 2016, from https://xebialabs.com/periodic-table-of-devops-tools.
47. 47
Holler, R. (2015). Ninth annual state of agile survey: State of agile development. Atlanta, GA: VersionOne.
ī¯ VersionOne found 94% using agile methods today
ī¯ Most are using Scrum with several key XP practices
ī¯ Lean-Kanban is a rising practice with a 31% adoption
ī
ī
Continuous
Integration
â
â
â
â
â
â
â
â
â
â
â
ī
â
ī
â
Agile TestingâAdoption Statistics
48. ī¯ Agile test use is low in spite of its age, i.e., 15 years
ī¯ Many do not understand its utter simplicity and power
ī¯ Failure to use agile testing undermines project success
48
Kim, D. (2013). The state of scrum: Benchmarks and guidelines. Indianapolis, IN: Scrum Alliance.
Agile Practices
Retrospectives
Refactoring
Done Definition
Test Tools
Test Driven Dev.
CM Tools
Simplicity
Pair Programming
Technical Debt
Agile Testing 13%
Continuous Integrations
Weekly
Daily
2-3 Times
Per Day
Never
2-3
Times
Per
Iteration
ī
ī
ī
Agile TestingâUsage Statistics
49. ī¯ Agile teams donât often use TDD, CI, CD & DevOps
ī¯ Implement independent test teams after Sprints done
ī¯ Sprint Waterfalling, Scrummerfalling, & Wagile result
49
Heusser, M. (2015). 12 years of agile testing: What do we know now. Proceedings of the Agile Gathering, Grand Rapids, Michigan, USA.
ī
ī
ī
ī
Incorrect
âĸ Phased Testing
âĸ Separate Teams
âĸ Delayed Testing
Correct
âĸ Integrated Testing
âĸ Integrated Teams
âĸ Continuous Testing
Agile TestingâAnti-Patterns
50. ī¯ Agile testing slows down with very large systems
ī¯ Slow testing slows integration and increases bugs
ī¯ Agile testing can speed back up with more attention
50
Kokko, H. (2009). Increase productivity with large scale continuous integration. Proceedings of the Agile 2009 Conference, Chicago, Illinois, USA.
ī
MICRO ADJUSTMENTS
- Focused Impact Tuning-
MACRO ADJUSTMENTS
- Wide Impact Tuning-
Add More CPUs & Memory
īˇParallelize System Builds
īˇReplace 3rd Party Test Libraries
īˇReduce or Remove Test Timeouts
īˇSelect Different Tests
īˇRefactor Code & Components
īˇTune Network & Software
īˇTune Database & Middleware
In-Memory Compilation
īˇParallelize Test Runs
īˇPre-Install Test Libraries
īˇRemove Process Randomness
īˇUse Faster Code & Test Tools
īˇIncremental vs. Big Bang Tests
īˇParallelize Build & Install
īˇTune & Optimize Build Process
īˇīˇ
Agile TestingâScaling Practices
51. ī¯ Industry very slow in adopting agile testing model
ī¯ Cost, difficulty, and territorialism are common issues
ī¯ Developers must take initiative for disciplined testing
51
Technical BarriersOrganizational Barriers
Developers donât want to test
¡ Infrequently committing code
¡ Committing broken code
¡ Failing to immediately fix builds
¡ Not writing automated tests
¡ Not ensuring 100% of tests pass
¡ Not running private builds
¡ Resorting to traditional testing
Resistance to change
¡ Fear of investment costs
¡ Fear of learning new skills
¡ Test group territorialism
¡ Organizational policy conflicts
¡ Overhead of maintaining CI
¡ Complexity and scaling
¡ Not developing a quality culture
¡¡
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
ī
ī
ī
ī
ī
ī
Agile TestingâCommon Barriers
52. ī¯ Eliminates big-bang integration in the 11th hour
ī¯ Creates a repeatable and reliable testing process
ī¯ Evaluates system-wide changes throughout project
52Maeda, M. K. (2009). Agile testing: Early, often, and smart. Arlington, MA: Cutter Consortium.
Whatâs the Bottom Line?
âAgile Testing Done Early & Oftenâ
Agile TestingTraditional Testing
Dramatically reduces risks
¡ Automates manual processes
¡ Instant verification & validation
¡ High project visibility
¡ Greater confidence and morale
¡ Incremental business value
¡ 24x7 deployability to users
¡ Highly quality and reliability
Late defect discovery
¡ Low quality software
¡ Poor project visibility
¡ Lack of deployability
¡ Late big-bang integration
¡ Testing is a bottleneck
¡ Poor customer satisfaction
¡ Outright project failure
¡¡
ī
53. Conclusion
ī¯ Agile methods DONâT mean deliver it now & fix it later
ī¯ Lightweight, yet disciplined approach to development
ī¯ Reduced cost, risk, & waste while improving quality
53
Rico, D. F. (2012). Whatâs really happening in agile methods: Its principles revisited? Retrieved June 6, 2012, from http://davidfrico.com/agile-principles.pdf
Rico, D. F. (2012). The promises and pitfalls of agile methods. Retrieved February 6, 2013 from, http://davidfrico.com/agile-pros-cons.pdf
Rico, D. F. (2012). How do lean & agile intersect? Retrieved February 6, 2013, from http://davidfrico.com/agile-concept-model-3.pdf
What How Result
Flexibility Use lightweight, yet disciplined processes and artifacts Low work-in-process
Customer Involve customers early and often throughout development Early feedback
Prioritize Identify highest-priority, value-adding business needs Focus resources
Descope Descope complex programs by an order of magnitude Simplify problem
Decompose Divide the remaining scope into smaller batches Manageable pieces
Iterate Implement pieces one at a time over long periods of time Diffuse risk
Leanness Architect and design the system one iteration at a time JIT waste-free design
Swarm Implement each component in small cross-functional teams Knowledge transfer
Collaborate Use frequent informal communications as often as possible Efficient data transfer
Test Early Incrementally test each component as it is developed Early verification
Test Often Perform system-level regression testing every few minutes Early validation
Adapt Frequently identify optimal process and product solutions Improve performance
ī
ī
ī
ī
ī
ī
ī
ī
ī
ī
ī
ī
ī
ī
ī
ī
ī
54. Daveâs PROFESSIONAL CAPABILITIES
54
Software
Quality
Mgt.
Technical
Project
Mgt.
Software
Development
Methods
Organization
Change
Systems
Engineering
Cost
Estimating
Government
Contracting
Government
Acquisitions
Lean
Kanban
Big Data,
Cloud, NoSQL
Workflow
Automation
Metrics,
Models, & SPC
Six
Sigma
BPR, IDEF0,
& DoDAF
DoD 5000,
TRA, & SRA
PSP, TSP, &
Code Reviews
CMMI &
ISO 9001
Innovation
Management
Statistics, CFA,
EFA, & SEM
Research
Methods
Evolutionary
Design
Valuation â Cost-Benefit Analysis, B/CR, ROI, NPV, BEP, Real Options, etc.
Lean-Agile â Scrum, SAFe, Continuous Integration & Delivery, DevOps, etc.
STRENGTHS â Data Mining īˇ Gathering & Reporting Performance Data īˇ Strategic Planning īˇ Executive & Manage-
ment Briefs īˇ Brownbags & Webinars īˇ White Papers īˇ Tiger-Teams īˇ Short-Fuse Tasking īˇ Audits & Reviews īˇ Etc.
â Data mining. Metrics, benchmarks, & performance.
â Simplification. Refactoring, refinement, & streamlining.
â Assessments. Audits, reviews, appraisals, & risk analysis.
â Coaching. Diagnosing, debugging, & restarting stalled projects.
â Business cases. Cost, benefit, & return-on-investment (ROI) analysis.
â Communications. Executive summaries, white papers, & lightning talks.
â Strategy & tactics. Program, project, task, & activity scoping, charters, & plans.
PMP, CSEP,
FCP, FCT
ACP, CSM,
& SAFE
32 YEARS
IN IT
INDUSTRY
55. Books on Agile Testing
ī¯ Thousands of textbooks on agile methods
ī¯ Include requirements, design, coding, test, etc.
ī¯ Continuous Integration, Delivery, & DevOps best
55
ī
Beck, K. (2003). Test-driven development: By example. Boston, MA: Addison-Wesley.
Crispin, L., & Gregory, J. (2009). Agile testing: A practical guide for testers and agile teams. Boston, MA: Addison-Wesley.
Gregory, J., & Crispin, L. (2015). More agile testing: Learning journeys for the whole team. Upper Saddle River, NJ: Pearson Education.
Duvall, P., Matyas, S., & Glover, A. (2006). Continuous integration: Improving software quality and reducing risk. Boston, MA: Addison-Wesley.
Humble, J., & Farley, D. (2011). Continuous delivery: Reliable software releases through build, test, and deployment automation. Boston, MA: Pearson Education.
56. Books on ROI of SW Methods
ī¯ Guides to software methods for business leaders
ī¯ Communicates the business value of IT approaches
ī¯ Rosetta stones to unlocking ROI of software methods
īŽ http://davidfrico.com/agile-book.htm (Description)
īŽ http://davidfrico.com/roi-book.htm (Description)
56
ī