Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Testing Object-Oriented Systems: Lessons Learned

Software Quality Week. San Francisco, June 15, 2000
Report on the state of the art and practice of testing object-oriented software.

  • Login to see the comments

Testing Object-Oriented Systems: Lessons Learned

  1. 1. Testing Object-OrientedSystems:Lessons LearnedRobert V. BinderRBSC Corporation www.rbsc.comJune 15, 2000
  2. 2. Overview • Lessons Learned – Design, automation, and process • The State of the Art – Design, automation, and process • The State of the Practice – Three levels – Testing can achieve world class quality© 2000 Robert V. Binder, all rights reserved 2
  3. 3. Lessons Learned • Class/Cluster test design – Super/subclass interaction must be tested: test at flattened scope. – Design subclass test suites to re-run on superclasses. – Design superclass test suites to re-run on subclasses. – Test polymorphic servers for LSP compliance.© 2000 Robert V. Binder, all rights reserved 3
  4. 4. Lessons Learned • Class/Cluster test design – Exercise each binding of a polymorphic server message – Test all parameters for generics – Test interface data flow of non-modal classes© 2000 Robert V. Binder, all rights reserved 4
  5. 5. Lessons Learned • Subsystem/system test design – Control easily obscured or accidental • Complex dependencies between concrete state and message sequence • Hierarchic control in state-based subclasses • Mosaic modularity at larger scope – Model behavior with state machines; achieve transition cover or better© 2000 Robert V. Binder, all rights reserved 5
  6. 6. Lessons Learned • Subsystem/system test design – Objects don’t compose (easily) – Producer’s framework should not be in consumer’s test scope – Minimum system/subsystem test includes • Testing exceptions • Testing class associations • Testing use cases (requires testable content)© 2000 Robert V. Binder, all rights reserved 6
  7. 7. Lessons Learned • Test Automation – Encapsulation and mosaic modularity decrease controllability and observability – Design-by-contract/assertions is the only practical counter-measure for inherent non- determinism and loss of testability© 2000 Robert V. Binder, all rights reserved 7
  8. 8. Lessons Learned • Test Automation – Avoid stubs: increase scope of the IUT or test in bottom-up order – Design test harness to exploit the structure and particulars of the system under test – Complete app = app components + test components under CM control© 2000 Robert V. Binder, all rights reserved 8
  9. 9. Lessons Learned • Process – Inspect for omissions and inconsistencies, test for everything else – Design for testability • Implement hierarchic architecture patterns • Eliminate or encapsulate cyclic dependencies • Assert class invariants, at least – Support reuse with complementary producer/consumer testing strategies© 2000 Robert V. Binder, all rights reserved 9
  10. 10. Lessons Learned • Process – 3 to 6 development increments • Developer class/cluster test -- Run tests locally, design tests globally: “unigration” • XP: “Continuous integration, relentless testing” • Independent build/integration group tests completed increment • Test suites must be regress-able – System testing on final increment© 2000 Robert V. Binder, all rights reserved 10
  11. 11. State of the Art • Representation • Design for Testability • Test Design • Test Automation© 2000 Robert V. Binder, all rights reserved 11
  12. 12. SOA: Representation • Best Practices – Syntropy, Design by Contract – UML/OCL 1.0 – Design Patterns • Challenges – Architecture – Limits of cartoons – Test design as software engineering© 2000 Robert V. Binder, all rights reserved 12
  13. 13. SOA: Design for Testability • Best Practices – Frameworks/libraries with assertions – Lakos’ levelizable architecture – Percolation pattern – OS/400 test framework© 2000 Robert V. Binder, all rights reserved 13
  14. 14. SOA: Design for Testability • Challenges – Seamless language support – OO testability an oxymoron? – Entropy horizon about 24 months© 2000 Robert V. Binder, all rights reserved 14
  15. 15. SOA: Test Design • Best Practices – Test design patterns • Challenges – Intra-class coverage – Polymorphic paths – Validated failure metrics/fault models© 2000 Robert V. Binder, all rights reserved 15
  16. 16. Test Design Pattern • New pattern schema for test design Name/Intent Context Fault Model Test Model Strategy Test Procedure Entry Criteria Oracle Exit Criteria Automation Consequences Known Uses Testing Object-Oriented Systems: Models, Patterns, and Tools. Addison-Wesley.© 2000 Robert V. Binder, all rights reserved 16
  17. 17. Test Design Patterns • Method Scope • Class/Cluster Scope – Category-Partition – Invariant Boundaries – Combinational Function – Modal Class – Recursive Function – Quasi-Modal Class – Polymorphic Message – Polymorphic Server – Modal Hierarchy© 2000 Robert V. Binder, all rights reserved 17
  18. 18. Test Design Patterns • Subsystem Scope • Reusable Components – Class Associations – Abstract Class – Round-Trip Scenarios – Generic Class – Mode Machine – New Framework – Controlled Exceptions – Popular Framework© 2000 Robert V. Binder, all rights reserved 18
  19. 19. Test Design Patterns • Intra-class Integration • Integration Strategy – Small Pop – Big Bang – Alpha-Omega Cycle – Bottom up – Top Down – Collaborations – Backbone – Layers – Client/Server – Distributed Services – High Frequency© 2000 Robert V. Binder, all rights reserved 19
  20. 20. Test Design Patterns • System Scope • Regression Testing – Extended Use Cases – Retest All – Covered in CRUD – Retest Risky Use Cases – Allocate by Profile – Retest Profile – Retest Changed Code – Retest Within Firewall© 2000 Robert V. Binder, all rights reserved 20
  21. 21. SOA: Test Automation • Best Practices – Design patterns for test automation – Automatic driver generation – Simple coverage analyzers© 2000 Robert V. Binder, all rights reserved 21
  22. 22. SOA: Test Harness Patterns • Test Case • Test Drivers Implementation – TestDriver Super Class – Test Case/Test Suite – Percolate the Object Method Under Test – Test Case /Test Suite – Symmetric Driver Class – Subclass Driver – Catch All Exceptions – Private Access Driver • Test Control – Test Control Interface – Server Stub – Drone – Server Proxy – Built-in Test Driver© 2000 Robert V. Binder, all rights reserved 22
  23. 23. SOA: Test Harness Patterns • Test Execution • Built-in Test – Command Line Test – Coherence idiom Bundle – Percolation – Incremental Testing – Built-in Test Driver Framework (e.g. Junit) – Fresh Objects© 2000 Robert V. Binder, all rights reserved 23
  24. 24. SOA: Test Automation • Challenges – Validated failure metrics/fault models – Tool capability gaps • No support for OO-specific coverage • Very weak specification-based test generation • Weak support for test harness generation© 2000 Robert V. Binder, all rights reserved 24
  25. 25. State of the Practice • Best Practices – Testing by scope (about 10%) – Many embedded/real-time shops – Extreme Programming • Challenges – High-frequency/short cycle development – Naïve test design – Tool capability gaps© 2000 Robert V. Binder, all rights reserved 25
  26. 26. SOP: Testing by Poking Around • About 70% of all organizations • Characteristics – Testing done at developer discretion – No test entry/exit criteria – High tolerance for low quality© 2000 Robert V. Binder, all rights reserved 26
  27. 27. SOP: Testing by Poking Around • Improvement Strategy – Assess limits of improvability – Train developers in basic test design – Install basic tool set: • Coverage analyzer • Memory leak detector • Test harness framework/generator (e.g. Junit)© 2000 Robert V. Binder, all rights reserved 27
  28. 28. SOP: Testing by Use Cases • About 20% of all organizations • Complies with “Unified Process” test approach • Characteristics – Assumes objects “just work” – System test from use cases – Frustrated with chronic bugginess© 2000 Robert V. Binder, all rights reserved 28
  29. 29. SOP: Testing by Use Cases • Improvement Strategy – Achieve exit criteria for indicated class/cluster test patterns – Use appropriate component/subsystem test design patterns. – Develop testable use cases – Implement test automation to support regression testing© 2000 Robert V. Binder, all rights reserved 29
  30. 30. SOP: Testing by Scope • About one in ten • Characteristics – Test design corresponds to scope – Scope-specific test entry/exit criteria – Appropriate testing at all scopes – Effective test automation – Stable, repeatable process© 2000 Robert V. Binder, all rights reserved 30
  31. 31. SOP: Testing by Scope • Improvement Strategy – Internal test design pattern-mining – Design for testability – Advanced test automation – Quantified closed loop feedback© 2000 Robert V. Binder, all rights reserved 31
  32. 32. Best Practice Examples • Stepstone Corporation • Ericsson CEE Project • Testing was the primary quality technique© 2000 Robert V. Binder, all rights reserved 32
  33. 33. Stepstone Corporation • ICpack 201 -- Objective-C class library • Inspections for all classes • Extensive automated test harness developed for each complex class • No systematic test design© 2000 Robert V. Binder, all rights reserved 33
  34. 34. Ericsson CEE • 75 KLOC C++ cellular support application • Systematic testing at class, cluster, and system scope • No other verification techniques used© 2000 Robert V. Binder, all rights reserved 34
  35. 35. Achieving World Class OO Quality • Best-in-Class level: – An average of “less than 0.025 user-reported defects per function point” in the first year after release • World Class = 10x Best in Class Capers Jones. Software Quality: Analysis and Guidelines for Success. (London: International Thompson Computer Press, 1997) p. 44© 2000 Robert V. Binder, all rights reserved 35
  36. 36. Achieving World Class OO Quality Organization/ KLOC FP Major Bugs/FP Language Post Release Bugs Stepstone 12 414 5 0.0121 Objective-C Ericsson 75 1364 7 0.0051 C++© 2000 Robert V. Binder, all rights reserved 36
  37. 37. Summary • Lessons learned: OO testing requires unique test design approaches • State of the art: expressed in patterns • State of the practice: world class quality can be achieved through testing© 2000 Robert V. Binder, all rights reserved 37

×