Unit testing is mostly a solved problem, but how do you write tests for the visual side of your app—the part that your users actually see and interact with? How do you stop visual bugs from reaching your users? We will dive deep into visual regression testing, a fast-growing technique for testing apps pixel-by-pixel. We will integrate perceptual diffs in Rails feature specs, and learn how to visually test even complex UI states. We will show tools and techniques for continuous visual integration on every commit, and learn how to introduce team visual reviews right alongside code reviews.

1. Continuous visual integration

2. Mike Fotinakis
@mikefotinakis

3. Percy
https://percy.io
@percy_io
Mike Fotinakis
@mikefotinakis

4. ● jsonapi-serializers
● swagger-blocks
Mike Fotinakis
@mikefotinakis

5. Continuous visual integration

6. - The Problem
- The Solution
- How it works

7. - The Problem
- The Solution
- How it works

15. “Regression”

16. “Visual Regression”

17. How do product teams fix this
today?

18. QA

19. QA is necessary,
but complicated, slow, and manual.

20. QA is necessary,
but complicated, slow, and manual.
And impossible to catch everything...
Medium sized app
60+ models
Hundreds of flows
Thousands of possible page states and permutations
Constant feature churn

21. Also, expen$ive.

23. Regression tests?

24. describe 'posting a project', type: :feature do
it 'creates a new project' do
visit '/'
fill_in 'Title', with: 'My project'
click_button 'Post Project'
expect(page).to have_content('Project created!')
expect(page).to have_content('My project')
end
end

25. describe 'posting a project', type: :feature do
it 'creates a new project' do
visit '/'
fill_in 'Title', with: 'My project'
click_button 'Post Project'
expect(page).to have_content('Project created!')
expect(page).to have_content('My project')
end
end
Doesn’t fail! The button still technically “works”.

26. describe 'posting a project', type: :feature do
it 'creates a new project' do
visit '/'
fill_in 'Title', with: 'My project'
click_button 'Post Project'
# ???
expect(page).to have_content('Project created!')
expect(page).to have_content('My project')
end
end

27. describe 'posting a project', type: :feature do
it 'creates a new project' do
visit '/'
fill_in 'Title', with: 'My project'
click_button 'Post Project'
# Assert that…CSS computed style…color of…
# is not equal… to… ???
expect(page).to have_content('Project created!')
expect(page).to have_content('My project')
end
end

28. xkcd.com/55/

29. The Problem:
Pixels are changing,
but we only test what’s underneath.

30. The Problem:
Even with all our current tests,
we still lack confidence in deploys.

31. - The Problem
- The Solution
- How it works

32. What if we could see
every pixel changed
in any UI state
in every PR?

33. What if we could
test our apps
pixel-by-pixel?

34. “Perceptual diffs”
“pdiffs”
“Visual diffs”

35. Perceptual diffs

36. Perceptual diffs

37. Perceptual diffs

38. Perceptual diffs

41. pdiffs in 30 seconds

50. Let’s write a
visual regression test
in 2 minutes!

51. Awesome right?

52. Simple uses: Catching visual regressions!

53. Simple uses: Catching visual regressions!
Advanced uses:
- CSS refactors / deletions.
- Testing style guides.
- Safe dependency upgrades.
- Visual regression tests for emails.
- Testing D3 visualizations.

54. Simple uses: Catching visual regressions!
Advanced uses:
- CSS refactors / deletions.
- Testing style guides.
- Safe dependency upgrades.
- Visual regression tests for emails.
- Testing D3 visualizations.
What I really want:
- Visual review alongside code review.

55. So if this is all so easy,
why aren’t we all doing
visual testing right now?

56. If it was easy
it wouldn’t be hard.
- Definitely somebody

57. 1. Tooling & workflows
2. Performance
3. Non-deterministic
rendering

58. Non-deterministic rendering

59. Non-deterministic rendering
Animations

60. Non-deterministic rendering
Animations

61. Non-deterministic rendering
Animations

62. Non-deterministic rendering
Animations in Percy (see blog.percy.io)

63. Non-deterministic rendering
Dynamic data

64. Non-deterministic rendering
Old test browsers

65. Non-deterministic rendering
http://alienryderflex.com/sub_pixel/
Sub-pixel antialiasing

66. Whew.

67. So...
pdiffs are only half the battle

68. Back to our goal:
What if we could see
every pixel changed
in any UI state
in every PR?

69. That would require:
- Fast. Really fast. As fast as your test suite.
- Handle complex UI states
- Continuously integrated with your
workflow on any commit.

70. - The Problem
- The Solution
- How it works

71. describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
end
end

72. describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
end
end

73. describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page, name: 'homepage')
end
end

74. describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end
describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end
Percy
describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end

75. describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end
describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end Images??
Percy
describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end

76. describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end
describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end Noooooo...
Percy
describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end

77. describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end
describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end DOM / HTML
snapshots!
Percy
describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end

78. DOM / HTML
snapshots!
(SHA256
fingerprinted)
describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end
describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end
Percy
describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end

79. percy-worker a
mitmproxy
percy-worker a
mitmproxy
percy-worker a
mitmproxy
describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end
describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end
describe 'app features', type: :feature do
it 'shows the dropdown menu when clicked' do
visit '/'
first('.dropdown-toggle').click
Percy::Capybara.snapshot(page)
end
end DOM / HTML
snapshots
percy-hub
percy-api
percy-worker a
mitmproxy

80. 1,000,000 visual diffs rendered in Percy
as of yesterday!

81. Quick Percy examples...

82. Visual testing is:
- Possible.
- A new way to think about testing.
- Helps give you deployment confidence.

83. Visual testing is:
- Possible.
- A new way to think about testing.
- Helps give you deployment confidence.
Much more work to be done to make it a
mainstream engineering practice!

84. One. More. Thing...

85. describe('groups list page', function() {
it('shows correct groups', function() {
visit('/groups');
click('.show-groups')
expect($('.group-list')).to.be.visible;
percySnapshot();
});
});
Sneak peak: percy-js & ember-percy

86. @percy_io
@mikefotinakis
Thanks!
@fotinakis