Slides from my talk on automated visual testing at the 2015 Selenium Conference

1. Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
adam.carmi@applitools.com
ADVANCED AUTOMATED
VISUAL TESTING WITH
SELENIUM

2. YOU CAN AND SHOULD
AUTOMATE YOUR
VISUAL TESTS!

4. AGENDA
 Why automatedvisualtesting?
 Tools& Technology
 Where does it fit?
 Q & A

5. WHAT IS VISUAL TESTING?
A qualityassurance activity aimed to verify that a
Graphical User Interface appears correctly to users

6. A VISUAL BUG

7. AND ANOTHER…

8. AMAZON PRIME DAY – JULY 15th, 2015

9. WHY SHOULD IT BE AUTOMATED?
THE TEST MATRIX IS TOO BIG TO COVER
MANUALLY
 Webbrowsers
 Devices
 Operatingsystems
 Screen resolutions
 Responsivedesign
 L10n
 3rd Partyupgrades

10. WHY SHOULD IT BE AUTOMATED?

11. WHY SHOULD IT BE AUTOMATED?
NATIVE / HYBRID MOBILE APPS
 Harder to roll back changes
 Can’t pushdaily
 Updates take battery and data
 Higherquality bar

12. WHY SHOULD IT BE AUTOMATED?
Many are already doing it…
PhantomCSS
Fighting Layout Bugs
CSS Critc Wraith
Needle
Grunt PhotoBox
dpdxt
WebdriverCSS
Eyes
Huxley
FBSnapshotTestCase
Gemini
Selenium Visual Diff
VisualCeption
Specter
Snap And Compare
kobold

13. AGENDA
 Why automatedvisualtesting?
 Tools & Technology
 Where does it fit?
 Q & A

14. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

15. DEMOhttps://github.com/webdriverio/webdrivercss

16. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

17. SCREENSHOTS
 Full page?
 Regions?
 Frames?
 Device pixel ratio?
 Rotation?
 Viewport size?
 Page preparation?
 Page stabilization?

18. QUICK FEEDBACK TOOLS
MOTIVATION
 Get fast feedback on code
changes
 Run tests locally in the
background
SETUP
 Render screenshots with a
headless browser
 PhatomJS, SlimerJS
 Configuration file driven tests
CONS
 Partial coverage
 Chrome, IE?
 Limited navigation
 Tests stale browser versions

19. VISUAL COVERAGE TOOLS
MOTIVATION
 Verify that the app / site looks right
in all real execution environments
 Test as many UI states as possible
 Execute many tests in parallel
SETUP
 Render screenshots using real
browsers on a variety of
operating systems and devices
in parallel
 WebDriver, Grid
 Code driven tests
 WebDriver, DSLCONS
 Requires test infrastructure
 Implement and maintain test
code

20. SELENIUM BASED TOOLS
CODE / SCRIPT
 Needle (Python WD)
 WebDriverCss (JS WebDriverIO)
 Gemini (JS DSL)
 Selenium Visual Diff (Java WD)
 VisualCeption (PHP
CodeCeption)
 Pix-Diff (JS Protractor)
 Shoov (JS WebDriverIO)
 Vizregress (.NET WD)
 Rspec Page Regression
(Capibara)
 Applitools Eyes (All WD +
Appium)
 Fighting Layout Bugs (Java
WebDriver)
CONFIGURATION
 Viff
 CSS Visual Test
 GreenOnion
 Wraith-Selenium

21. NON-SELENIUM BASED TOOLS
CODE / SCRIPT
 PhantomCSS (JS CasperJS)
 Specter (JS DSL)
 FBSnapshotTestCase (XCTest)
CONFIGURATION
 Wraith
 Dpdxt
 Grunt PhotoBox
 Grunt-Vigo
 Snap And Compare
 BackstopJS
 CSSCritic
 Kobold
 DiffCop
 SUCCSS
MANUAL
 Eyes Express (Chrome)

22. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

24. WHY NOT?
FALSE POSITIVES

25. ANTI-ALIASING?

26. ANTI ALIASING 1/2

27. ANTI ALIASING 2/2

28. BRIGHTNESS 1/2

29. BRIGHTNESS 2/2

30. TEXT POSIONING 1/2

31. TEXT POSIONING 2/2

32. IMAGE SCALING 1/2

33. IMAGE SCALING 2/2

34. ANDMORE…
1 pixeloffsets in elementpositioning
Dynamic content
Movingelements
Images ofdifferentsize
Performance

36. Image Comparison APIs

37. ImageMagick
 A powerful command linetool for image processing.
 APIs are available for most programming languages.
 Fuzzing is used to eliminateslight color differences
 An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246

38. The Javascript Engines
 Resemble.js
Pixel bypixel + errorratio + anti-aliasing
http://huddle.github.io/Resemble.js
 Blink-Diff
Pixel by pixel + errorratio + anti-aliasing + perceptual colordistance computation
https://github.com/yahoo/blink-diff
 Looks-Same
Pixel by pixel + perceptual color distance computation + ignore Caret
https://github.com/gemini-testing/looks-same

39. Applitools Eyes
 A specialized image processing stackdesigned to compare computer
generated UI images
 Handles anti-aliasing,pixel offsets, color similarity, and image scaling
 Dynamic and moving content
 Compare images of different sizes
 No error ratio configuration required
 Validates full UI pages
 Fast!
 Supports layout matching

40. DEMO

41. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

42. REPORT DIFFERENCES
As files on the file system (combined with source control)

43. REPORT DIFFERENCES
As a Gallery (example from Selenium Visual Diff)

44. THE WORKFLOW
Drive the AUT and take screenshots
Compare screenshots with baseline
images
Report differences
Update the baseline

45. UPDATE THE BASELINE
 Rename or commit individualimage files
 GUI (GeminiGUI)

46. UPDATE THE BASELINE
 Overwrite mode
 Automatic maintenance (ApplitoolsEyes)

47. AGENDA
 Why automatedvisualtesting?
 Tools& Technology
 Where does it fit?
 Q & A

48. WHERE DOES IT FIT?
• Component
s
• Code
review
• Developers
• Designers
• QA
Visual testing of frontend components by
frontend developers

49. WHERE DOES IT FIT?
• Pages
• Page
sections
• Developers
• Designers
• QA
• Others
Full or partial validation of application
screens
A collaboration tool

50. WHERE DOES IT FIT?
• Staging vs.
Production
• Ops
• QA
Validate your staging deployment using your
production deployment as a baseline

51. WHERE DOES IT FIT?
• Monitoring
• Ops
• QA
No missing resources in production
No corruption due to 3rd party data
No corruption due to browser / OS upgrades

52. QUESTIONS?

53. Adam Carmi (@carmiadam)
Co-Founder and VP R&D at Applitools
adam.carmi@applitools.com
THANK YOU