The Holy Trinity of UI Testing by Diego Molina
•Download as PPTX, PDF•
2 likes•251 views
The first step before testing is defining what we want to test. This may sound trivial, but in reality this not done often properly. We tend to oversee the obvious and we start testing without knowing what we want to accomplish. Do we want to validate the user behavior? Do we need to check that the page design is responsive on different devices? Knowing what is important and what needs to be validated can help us enormously to have a clear purpose. When we know the purpose of our test, we can start planning, coding, executing and improving our tests. But overall, we will know what approach we can use to develop the test. Functional, layout and visual testing are the three pillars of the UI testing trinity. These are three approaches we can use to develop focused tests, tests that are asserting a specific aspect of a web application. But how can we identify what approach to use? When should we combine them? This SauceCon 2018 session will help attendees to define what they want to test and what approach to use when developing the test. It will go deeper through scenarios and code examples that show how to create tests with great assertions and clear purpose, tests that give value to the team. It will also discuss scenarios where a functional test is not enough, or where a visual test is better than a layout test. This talk’s main goal is to offer a different perspective when testing a web application through the UI testing trinity.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to The Holy Trinity of UI Testing by Diego Molina
Similar to The Holy Trinity of UI Testing by Diego Molina (20)
More from Sauce Labs
More from Sauce Labs (20)
Recently uploaded
Recently uploaded (20)
The Holy Trinity of UI Testing by Diego Molina
- 1. The Holy Trinity of UI Testing Diego Molina
- 2. ¡Hola! I am Diego Molina Software Engineer who likes Testing @Zalando and @Element34.com diegofmolina diemol 2
- 3. European Platform for Fashion > 22M Active users > 1.9K Tech (>200 teams) > 250K articles > 10 languages Provide testing infrastructure Help teams to test better Co-developed Zalenium Selenium Box On- Premise solution Trainings & Consulting Test automation solutions around Selenium Co-maintain docker- selenium Contribute to Grid Workshop IRC/Slack to answer questions 3
- 4. Why? ?
- 5. One step back What do we want to test? What do we want to achieve?
- 6. Automation will always need people. Great tools do not replace a good testing strategy Photo by Sneaky Elbow on Unsplash
- 7. Functional Tests Examples Prototypes Story Tests Simulations SupportingtheTeam Business Facing Unit Tests Component Tests Technology Facing CritiqueProduct Q1 Q2 Q3 Q4 Exploratory Testing Scenarios Usability Testing User Acceptance Testing Alpha / Beta Performance, Load, Security... Testing Automated & Manual Manual Automated Tools Agile Testing Quadrants Lisa Crispin - http://lisacrispin.com/2011/11/08/using-the-agile-testing-quadrants/ & “More Agile Testing” by Janet Gregory and Lisa Crispin
- 8. Functional Testing Testing that things work based on a set of specifications. 8Photo by lovestruck. on Foter.com / CC BY-NC
- 9. Layout Testing Validate that a responsive web application changes its layout based on the viewport size and on our expectations. 9Photo by Ricardo Gomez Angel on Unsplash
- 10. Visual Testing Testing that a web application’s UI is displayed to the users based on our expectations. 10Photo by Nonsap Visuals on Unsplash
- 11. When do I use them? When do I mix them? Or not? 11
- 12. Application Characteristics What to consider? Stage in the Development Process 12
- 14. Decision table* Application Type Functional Layout Visual Dynamic (CMS) Yes Maybe Nice to have Internal Tool Yes Not needed Not needed eCommerce Yes Nice to have Yes Animated/Graphics Yes Not needed Yes 14*apply common sense and use context, these are suggestions. Maybe: Consider user base size, analytics info, feature criticality, risk exposure, etc... Nice to have: If time and resources permit, risk exposure, etc...
- 18. 18 Normal ~8 sec. None ~4 sec. Page Load Strategy
- 19. 19 Page Load Strategy, switch from NORMAL to NONE Important: Make sure your waits are working well.
- 23. 23 Galen Framework ● Well maintained ● Open source ● Highly configurable ● Works with Selenium ● Own DSL ● Partly reuse current tests ● Java & JS
- 24. 24 ● Still many things left to check ● Add more specs ● Increase level of detail ● Bit slow with big DOMs
- 29. 29
- 30. 30
- 31. PROS ● Customize all the way ● Full control ● Reuse existing tests ● Execution time Rolling your own CONS ● Maintain more code ● Handle comparison logic ● Handle false positives ● Create a flow to handle baselines ● Maybe in a different language 31 https://github.com/Huddle/Resemble.js https://github.com/mapbox/pixelmatch https://github.com/americanexpress/jest-image-snapshot
- 32. Using open source - Gemini 32 https://github.com/gemini-testing/gemini https://github.com/gemini-testing/gemini-gui https://github.com/gemini-testing/html-reporter
- 34. PROS ● That’s the tool’s purpose ● Configurable ● Useful plugins ● Execution time Using open source - Gemini CONS ● Own DSL in JS ● Rewrite tests 34 https://github.com/gemini-testing/gemini https://github.com/gemini-testing/gemini-gui https://github.com/gemini-testing/html-reporter
- 38. PROS ● That’s the tool’s purpose ● Configurable ● Reuse existing tests ● Many languages supported ● Different validation levels Using Applitools CONS ● Test execution time ● Paid ● Validations limit 38 https://applitools.com/
- 39. Recapping the story so far... 39
- 40. ● It is the cornerstone, the first step ● It is what gives us a sense of security ● Combine it with a testing strategy 40 Functional Testing
- 41. ● Useful when content is stable ● Validates only what we specify to be validated ● Helps to check design and CSS related changes ● Good option for a smoke test in big applications 41 Layout Testing
- 42. ● More validations with less code ○ Very tempting! ● Makes us feel more secure ● Increases coverage ● Combine with functional tests as much as possible ● It is just one more item in your current strategy 42 Visual Testing
- 43. 43 Visual Testing ● Information overload ● Invest time to incorporate it in your workflow ○ It is not plug and play ○ Produce valuable reports - reduce review time ● Define criterias to establish new baselines ○ Don’t say yes to everything ● Give time, there is a learning curve ○ One browser + one viewport size, then iterate
- 44. Credits ● @elgalu - Leo Gallucci ○ Initial ideas over this topic ● @JAVI_GOMEZ - Javier Gómez ○ Ideas exchange and validation ● @lisacrispin - Lisa Crispin ○ Agile Testing Quadrants ● @katrina_tester - Katrina Clokie ○ http://katrinatester.blogspot.com/2017/10/strategies-for-automated-visual.html ● @jcolantonio - Joe Colantonio ○ https://www.joecolantonio.com/2017/03/16/applitools-galen/ ○ https://www.joecolantonio.com/2017/02/02/top-21-free-visual-validation-tools-testers/ 44
- 45. 45 Thanks! Any questions? Find me at @diegofmolina & diemol@gmail.com Photo by Trent Erwin on Unsplash
Editor's Notes
- How to give back to Open Source? Help to improve the documentation Open spaces at your company for meetups and knowledge sharing events Report bugs that are reproducible (and help to fix them if possible)
- A large amount of new tools, popping up everywhere and every week, providing great solutions. Who is kind of active in Twitter? Reads blog posts? Don’t you feel that you need to catch up? So many buzzwords. All this information overload makes us lose focus on the value we want to add to our team, and to our web application. We spend more time choosing tools instead of planning our tests. Here are my two cents, hopefully this talk can help you to have focused tests and achieve clear goals.
- Let’s have a coffee/tea, sit down for a moment and relax… Before thinking about tools that will solve all our problems… Let’s think about what we want to achieve and what we want to test.
- To have a sustainable and maintainable test suite, you need to plan a bit first. Think about a testing strategy and what you want to achieve. When that is clear, choosing the right tools is much easier. This does not mean that I don’t want to use Sauce Labs, Applitools, Selenium Box… The tools are there. I just need to know how to use them efficiently. We test to add value to the product, not to only find bugs.
- A test strategy should include, unit testing, TDD, BDD, functional testing, exploratory testing, non-functional testing, etc… But when do do each one? Depending where you are, they help to identify the types of testing needed. The numbers do not imply an order! Left side verifies, right side validates. Goals: Q1: Grained business logic works, isolation (problems), confidence when making changes. Fast feedback. Facilitates other quadrants. Q2: Guide dev through business-facing tests. Build examples and then tests. Find out if what we expect from the product works or not. Q3: Reality check, feedback from real use. Are things working in different scenarios. User Acceptance and Exploratory Testing. Helps Q2 to improve tests. Q4: Memory leaks? Disaster recovery? How stable under high load, hacker attacks… Cross browser testing (page rendering). Planning: Scope -> Priorities + Risks -> Tools
- Basically, we want to validate that the application works according to its functional specification. If A takes me to B when I do C, and as a result I will get D. If logging in takes me to my account when I enter my credentials, and I will see my personal data.
- Focused on testing responsive web applications in different devices and resolutions (in a relative position). Validates if the elements are present or not depending on the device resolution and our expectations.
- Focuses more into how the web application looks like, its colors, shapes, presence of elements, elements overlapping, sizes… Web applications can render differently in different browsers and devices. It works by first establishing a baseline (the current look), and future executions get compare against the baseline (and update it when needed).
- Taking into account the Agile Testing Quadrants and the types of testing we have mentioned… When does it make sense to use and mix them? Always use the 3 of them? Maybe not? How do we put all this information together?
- Depending on two main decision points, we could decide what type (or types) of testing we can use.
- What type of application do we have? eCommerce? Mostly static content? (Government Website) CMS backed application? Generates images dynamically (charts, graphics)? Internal tool website? Supported browsers?
- This decision table should serve as a guide when we are planning which testing type to use. This is not a silver bullet, common sense and context are needed to make the final decision. Get as much information you can: Analytics, risk assessment (how bad is it if we screw up), application behavior (does it work the same way in different browsers, resolution, languages, payment types?)
- After deciding what type of testing we are going to use, we also need to decide when to run each type of tests. Does it make sense to run functional, layout and visual tests on every commit? We need to remember that the main role of tests during development is to provide fast feedback,
- Zalando Normal: ~15 None: ~9
- GitHub Normal: ~8 None: ~4
- This is how the capabilities look like, it is just a simple change in the PageLoadStrategy, but double check that the waits are well tuned.
- A few commits have been done in the feature branch, now there is a potentially working feature to be merged and released. What tests do we run now? Let’s start with the Dynamic (CMS) web applications.
- Currently, the only well supported and up-to-date tool is the Galen Framework. It focuses on validating that the relative positioning of elements is correct based on a set of specifications. Like checking that the button is below the textbox, or that an element contains X or Y. It can also validate images, and then the configuration detail gets quite complex, therefore the maintenance cost gets higher. Go to IntelliJ, show original test, modified test, specs and reports.
- Currently, the only well supported and up-to-date tool is the Galen Framework. It focuses on validating that the relative positioning of elements is correct based on a set of specifications. Like checking that the button is below the textbox, or that an element contains X or Y. It can also validate images, and then the configuration detail gets quite complex, therefore the maintenance cost gets higher. Go to IntelliJ, show original test, modified test, specs and reports.
- Currently, the only well supported and up-to-date tool is the Galen Framework. It focuses on validating that the relative positioning of elements is correct based on a set of specifications. Like checking that the button is below the textbox, or that an element contains X or Y. It can also validate images, and then the configuration detail gets quite complex, therefore the maintenance cost gets higher. Go to IntelliJ, show original test, modified test, specs and reports.
- Still many things need to be checked. Too much content that can change a lot. We would need to have a huge list of specs to reach a certain level of security, but we would incur into high maintenance costs (in the end, it is a tradeoff). More suitable for web applications where there is a smaller amount of important elements to check and the content is in general more stable.
- Now, for eCommerce and Animated web applications, we will use Visual testing. Learning curve, dealing with false positives.
- Test execution time stays fairly the same, since all the comparison is done locally. But this saved time needs to be used implementing an adaptable workflow (what happens when there are differences? How do we approve them and establish a new baseline? How do I ignore regions?)
- Test execution time stays fairly the same, since all the comparison is done locally. But this saved time needs to be used implementing an adaptable workflow (what happens when there are differences? How do we approve them and establish a new baseline? How do I ignore regions?)
- Test execution time stays fairly the same, since all the comparison is done locally. But this saved time needs to be used implementing an adaptable workflow (what happens when there are differences? How do we approve them and establish a new baseline? How do I ignore regions?) Documentation? Handling baselines for different resolutions.
- Go to IDE and show how the setup looks like.
- Go to IDE and show how the setup looks like.
- Go to IDE and show how the setup looks like.
- Go to IDE and show how the setup looks like.
- If the layout is OK then we assume the content is also OK. Image checking can be done with Galen, but you need to configure each validation, increasing the level of detail and impacting the code maintainability.
- New blog posts the whole time, new tools. Don’t do it because everyone is doing it, see how it will add value first. More tests with less code, then create meaningful reports, so people don’t need to spend lots of time analysing them -- rejecting or approving changes. Invest time establishing baselines, and check them often to be sure they are actually ok. Also in different viewport sizes, and browsers (go slowly, step by step, don’t put all browser and screen resolutions - everything from the beginning, you’ll get a lot to review. Usually works well when you have already a good functional test suite that shows you what is being tested. Crossbrowser? First you need to make your functional tests work in different browsers.
- New blog posts the whole time, new tools. Don’t do it because everyone is doing it, see how it will add value first. More tests with less code, then create meaningful reports, so people don’t need to spend lots of time analysing them -- rejecting or approving changes. Invest time establishing baselines, and check them often to be sure they are actually ok. Also in different viewport sizes, and browsers (go slowly, step by step, don’t put all browser and screen resolutions - everything from the beginning, you’ll get a lot to review. Usually works well when you have already a good functional test suite that shows you what is being tested. Crossbrowser? First you need to make your functional tests work in different browsers.
- Lisa Crispeta