2. The Problem
Failuresguide our work
The product is ever-changing
Developers do not communicate
changes proactively (or at all)
40% of our time is spent accounting for
these changes
Who can analyze 300 test failures every
night?
3. What Changes?
Htmlid (i.e. “Locator”)
Element type
Major navigation path
Each will break most automated tests
4. Html id
Ifthe id changes (e.g. to signinField), the
test will break
The element will no longer exist
5. Element Type
Thelink is changed from a Link to Button
WebDriver will still look for a link
6. Navigation Path
Pages are added
Popups are introduced
Buttons and forms are split up across
pages
UI Look and feel “reset”
WebDriver scripts don’t stand a chance—
you just start over
8. Wish List for a Solution
Build in a mechanism for change
communication
Account for the changes AS THEY
HAPPEN, not reactively
Tighter integration with development
Pass rate needs to be 97% or better, with
all failures accounted for within 24 hours
9. Page Object Generation
Generate the pages on every build
Web Controls are mapped to specific
WebElement types
If the type of an object changes in a way
that breaks automation, the whole build
fails
11. What you get
A page, containing every element in the
UI
Each element is aware of its Tab
Including every Rich WebElement
A Fields object, usable by those writing
Test Scripts
13. The New Change Cycle
•Significant underlying type (Text Input to Radio Button)
•Insignificant underlying type (Button to Link)
Developer
Change •id changes for localization
•Compiler flags type change—breakage
•Compiler ignores type change (Both are IClickable)
Regenerate
Model •Page auto-updates id change—no breakage
•Tester doesn’t know—fixed before dev check-in
•Tester doesn’t know—Framework “absorbs” the change
Test •Tester doesn’t know—id is s String, stored in one place
14. When change happens
Html id changes
Who cares?
Generation process normalizes names
Unit test enforces uniqueness
Element type changes
Developer fixes prior to check-in
Navigation path changes
More rare, but failures show up within 24 hours
Without all the noise, issues are easier to spot,
analyze, and fix
15. Where do you go now?
You have a model of your system
Use it to “analyze itself”
The all-table test
Role-based security tests
Algorithms to
Iterate through every page looking for
standards compliance
500 errors
Forms
Security issues (code injection, XSS etc)
16. Bottom Line
We found 7 major regressions in 2008, the
last year of the “old” platform
12 in 2009 with >500 test cases
20 in 2010 with >1100 test cases
More than 50 in 2011 with 1700 test cases
Customer reported defects did NOT go
down, because…
Velocity increased so much, features
were added at a much faster clip
