Another way to monitor performance of your site with Selenium and log analyzer

1. Synthetic performance monitoring
with Selenium
Andriy Samilyak
OpsWay
CEO

2. Who is talking?
Working in IT since 2005
● Since 2008 - Smile Ukraine CTO
● Since 2013 – OpsWay CEO
● Coaching with SmartMe.com.ua
●

3. You know this...
“Metrics show that a 0.1 second delay in page
rendering can translate into a 1% drop in
customer activity”
Amazon research papers

4. Projects are changed daily
●
Application code
●
Product database
●
Marketing tags
●
Server configuration
●
External services
●
etc
… and no QA engineers at all...

5. User checkout cycle under control
Add to cart
Select product
Success page!
Login
~30-50 sec
Billing/Shipping
options
Payment

6. Is it still running smoothly?
●
Passive checks – analyzing end-user activity
–
–
Webserver log analysis
–
●
NewRelic
Web analytics (real user monitoring)
Active checks – run test and record results
–
Gomez GPN
–
Uptrends
–
Pingdom

7. NewRelic
Cool, sexy, not so cheap and not really what we need

8. GoogleAnalytics with UserTiming
https://github.com/opsway/magento-mod-rum
… but we need immediate alerts!

9. Server-side log analysis
●
Complex models:
–
2 add to cart requests timing
–
1 login step timing
–
5 checkout steps timings
●
not comparable side by side
●
not representative when user activity is low

10. Server-side logs: false-positive
3 mins to checkout? Are you serious?

11. Pingdom/Uptrends
●
Transaction monitoring via scripting
●
Fancy scripting language
●
●
Robust, popular but not so cheap for multiple
projects
Fine for public sites, but not for intranet

12. We liked Compuware Gomez!

13. Compuware Gomez is cool

14. But we have already Selenium!
Selenium
Splunk
1. Load product page
Send timestamp via UDP
2. Add to cart
Send timestamp via UDP
3. Shopping cart
Send timestamp via UDP
4. Address
5. Payment
6. Success page
...

15. It looks like this in code...

16. Splunk combines to transaction...

17. We can chart transactions

18. … and alert on abnormal behavior
Slow transaction
Failed
Transaction!

19. Why did transaction fail?
●
Selenium reported timeout – not always
sufficient
●
Rarely reproducible (many servers)
●
We needed screenshots!

20. Capturing a screenshot

21. Saving screenshot with Jenkins

22. Jenkins build with errors
We got it!

23. Why it took so long?
●
●
We don't know always what http request
blocked transaction
Inspired by Gomez we integrated HAR
recording of test sessions

24. Inspired by Gomez ...

25. HAR is not perfect, but ...

26. HAR proxy
●
Based on BrowserMob proxy server
http://bmp.lightbody.net/
HAR file
Launching WebDriver
Selenium Grid
Proxy on port 64001
Request passes
through proxy
www.mysite.com
Selenium node
(Firefox)

27. Jenkins build with errors
Here we have HAR

28. Issues we have
●
Supporting test scripts takes a lot of time
●
Support test infrastructure is tedious
●
False-positives
●
Still not enough debug information

29. Andriy Samilyak
samilyak@gmail.com
skype: samilyaka
http://opsway.com