2. Agenda Goal is to show how you can enhance your automation tests to gather performance metrics Keep the short presentation as practical as possible with enough code snippets to give you a starting point
3. One more thing… HTTP Archive (HAR) What? UTF8 Text based JSON encoding to represent performance data Why? Many libraries to convert from JSON to objects More tools allowing import/export/view HAR Becoming the standard format
6. Which Metrics? Overall page load time DOM loading/interactive/complete, browser 1st render, … Per-item timings DNS, SSL connect, time to first byte, total time to download bytes, … Headers, status codes, and content For a more detailed picture of the reasons for the page performance Can help with debugging performance, e.g. were items compressed, or not being loaded dynamically, or certain items not being prioritized for above-the-fold rendering
7. Methods for gathering metrics Setting your own timings in test code Using the new ‘Navigation.Timings’ or Web Timings standard built into browsers Using browser plugins that report back the timings to you, e.g. WebPageTest Per HTTP request logs via Selenium’s built-in proxy Routing the browser traffic through a local proxy and gathering the statistics from there. Network traffic capture
8. Method 1: Own Timings WebDriver driver = newChromeDriver(); Date start = new Date(); driver.get("http://www.webmetrics.com"); Date end = new Date();
9. Method 2: Web Timings Currently Chrome and IE9 supported, coming soon for Firefox http://w3c-test.org/webperf/specs/NavigationTiming/ WebDriver driver = newChromeDriver(); // A "base url", used by selenium to resolve relative URLs StringbaseUrl = "http://www.webmetrics.com"; driver.get(baseUrl); JavascriptExecutorjs = (JavascriptExecutor)driver; LongloadEventEnd= (Long) js.executeScript("return window.performance.timing.loadEventEnd"); LongnavigationStart= (Long) js.executeScript("returnwindow.performance.timing.navigationStart"); Long answerToAllProblems =loadEventEnd - navigationStart;
10. Method 2: Web Timings Could also modify this by using your own JavaScript loaded into the page to set timings Also would work in combination with the already set Web Timings E.g. Set a Date().getTime() variable when some dynamically loaded content is loaded into the DOM LongperceivedLoadTime = (Long) js.executeScript("return elementInsertTime – window.performance.timing.navigationStart");
12. Method 3: Browser Plugins Typically can give compute time metrics Javascript execution time CPU usage Render times IE8 - WebPageTest CPU usage Video capture Firefox - Firebug Net Panel + NetExport https://github.com/lightbody/browsermob-page-perf https://github.com/AutomatedTester/AutomatedTester.PagePerf.git DynaTrace browser plugin (FireFox/IE on Windows) Good breakdown of stats (Javascript execution times, CSS times, render, 'first impression' time). http://ajax.dynatrace.com/ajax/en/ Chrome – some export from the Developer Tools Network Panel?
15. Method 4: Use a Proxy Use built in Selenium 1 proxy Use a 3rd party proxy library Many available, few capture metrics in a convenient way Two good ones: BrowserMob Proxy Fiddler
17. Method 4: Selenium built-in Proxy Not in HTTP Archive (HAR) format, but does report: HTTP status code, URL and HTTP method Request & response headers Overall request->response timing Bytes downloaded Works with Chrome and Firefox Doesn’t work for WebDriver tests. Still work with Selenium 2 RC, but only if send jobs via Selenese API. Not much control over the proxy
18. Method 4: Advantages of using a Proxy Testing benefits beyond performance monitoring Blacklisting/whitelisting URLs Comparing page load times of site with/without external content Not hitting analytics/advertising content Simulating external content being offline URL rewrites Redirect production URLs back to staging/test environment Pretend to be production as far as the browser is concerned (e.g. for cookies) Make sure ad/metrics requests are being made
19. Method 4: Advantages of using a Proxy Header changes Set the user agent manually to test different browser behavior Auto authentication Wait until all content is downloaded HTTP idle for X seconds Limit bandwidth
20. Method 4: BrowserMob Proxy Open source cross platform proxy written in Java. HTTP Archive support Friendly API Source code available: https://github.com/lightbody/browsermob-proxy
21. Method 4: BrowserMob Proxy Export HAR Sample import org.browsermob.proxy.ProxyServer; ProxyServer proxy = new ProxyServer(9090); proxy.start(); Proxy mobProxy = new Proxy().setHttpProxy("localhost:9090"); DesiredCapabilities caps = new DesiredCapabilities(); caps.setCapability("proxy", mobProxy); FirefoxDriver driver = new FirefoxDriver(caps);
22. Method 4: BrowserMob Proxy Run test, denoting pages in HAR proxy.newHar("Yahoo"); driver.get("http://yahoo.com"); proxy.endPage(); proxy.newPage("CNN"); driver.get("http://cnn.com"); proxy.endPage();
25. Method 4: BrowserMob Proxy When to use Cross platform Java Can set the browser proxy
26. Method 4: Proxy - FiddlerCore Fiddler is an application for viewing HTTP traffic on Windows. Works with Chrome, FireFox, Internet Explorer. Fiddler Application is built on FiddlerCore .NET library Allows extensive programmatic control over the proxy Captures HTTP timings Allows request/responses to be intercepted and modified Configures itself as default Windows proxy (supports proxy chaining) Can decrypt SSL traffic
27. To start the proxy and register it as the default system wide proxy. Each HTTP transaction can then be recorded as follows: Method 4: Proxy - FiddlerCore // Initialize the proxy Fiddler.FiddlerApplication.Startup( 8877, FiddlerCoreStartupFlags.Default); var items = new List<Fiddler.Session>(); Fiddler.FiddlerApplication.AfterSessionComplete += delegate(Fiddler.SessionoS) { items.Add(oS); };
28. Method 4: Proxy - FiddlerCore Run Selenium test as normal As each item is downloaded it will be added to the ’items’ var in previous slide. string baseUrl = "http://www.webmetrics.com"; varwebDriver = newOpenQA.Selenium.Firefox.FirefoxDriver(); var selenium = newSelenium.WebDriverBackedSelenium (webDriver, baseUrl); selenium.Start(); selenium.Open(baseUrl); selenium.WaitForPageToLoad("30000"); selenium.Stop();
29. Method 4: Proxy (Fiddler -> HAR) Using the HAR Exporter to convert the sessions to HAR Add FiddlerCore-BasicFormats.dll to the project references and load the assembly: https://www.fiddler2.com/dl/FiddlerCore-BasicFormats.zip String exePath = Assembly.GetExecutingAssembly().Location; String path = Path.Combine( Path.GetDirectoryName(exePath), @"FiddlerCore-BasicFormats.dll"); FiddlerApplication.oTranscoders.ImportTranscoders(path);
30. Method 4: Proxy (Fiddler -> HAR) Finally, export the HAR to a file: varoExportOptions = new Dictionary<string, object>(); string filename = @"output.har”; oExportOptions.Add("Filename", filename); Fiddler.FiddlerApplication.DoExport( "HTTPArchive v1.2", sessions, oExportOptions, null);
31.
32. Method 4: Proxy – FiddlerCore When to use Windows Only Cross browser .NET
33. Method 5: Wire HTTP capture Captured network traffic can be converted to HAR On Unix/OSX use tcpdump tcpdump -i en0 -n -s 0 -wtraffic.pcap On Windows use WinPCap winpcap -i 0 -n -s 0 -wtraffic.pcap Then use pcap2har to do the conversion: main.pytraffic.pcap http-requests.har Pcap2har https://github.com/andrewf/pcap2har
34. Method 5: Wire HTTP capture Captures allHTTP traffic Timings based on actual network traffic No interference from proxy No extra network delays/context switches talking to the proxy Browsers sometimes behave differently when talking to a proxy
35. Summary Discussed 5 methods for recording performance metrics Use timers around selenium commands Use WebTimings JavaScript API Firebug & NetExport Proxy Use the built-in Selenium proxy Use a 3rd party proxy Sniff network traffic
36. Links BrowserMob proxy https://github.com/lightbody/browsermob-proxy Fiddler Cookbook http://www.fiddler2.com/fiddler/dev/ScriptSamples.asp Examples from this talk https://github.com/watsonmw/selenium-pageloadmetrics
Editor's Notes
Welcome to our talk about getting performance metrics in a day
Outline the talk, what is the takeawayRun your standard unit tests (Junit, minitest, ….) but put in performance testing as well
Becoming the standard mechanism to store/dump performance data
Lots of text on this slide, we’ll keep it more entetaining laterOverall page load time= The obvious starting point and usually the easiest to acquire
Setting your own timings – obvious, but is not always very accurate if you are in a sequence of steps and capture page transitions and other delays Very coarse, incorporates latency/setup/teardown delays in the times reported, e.g. it sometimes takes a second or 2 for an IE window to become responsiveWeb Timings == Very simple at the moment, but gives very accurate overall stats, like initial connection time, DOM ready, and total request->response times.Browser plugins == Accurate picture of what the browser is doing, but not so straightforward to incorporate into your code.Proxy == Gives you very good timings for http traffic but can’t give you timings like render time or DOM ready events.Network traffic capture tools like WinPCAP, or wireshark
Doesn't seem to work currently in FF4Will have to parse the HAR format if you just want the basic timing out of it
Going to go over:- How to grab metrics using a proxy How using a proxy can be beneficial for testing in general
(Fiddler refers to thehttp transaction as 'sessions')
Documentation spread out over blogs, fiddler google groups, and fiddler extensions pageAPI is little difficult
