4. Automated Acceptance Test
In Continuous Delivery
• “Are we good to go live?”
- Test jobs are your “eyes and ears”
- Optimize for them!
• Test code equals production code
Common problems:
• Costly to maintain test buckets
• Spent much time running and debugging false alarms
9. “ Don’t have a separate Testing/QA team!
Quality is down to everyone - Developer owns Acceptance Tests!! ”
― David Farley, Pipeline Conf. 2015
10.
11. “ Not enough to have acceptance test and acceptance test got to be
created and maintained by developer
will make code more testable and maintainable. ”
― Jez Humble, Pipeline Conf. 2016
12. don’t hire too many dedicated testers
developer relies on them, lazier and write more bugs
hire people who can development and test functions
13. code review occupies a central position
no separate group of tester or QA people
relies on automated testing
14. “Testing is essentially the responsibility of the person
who develops a given feature”
(tens of thousands of regression tests)
developer responsible for writing unit tests, regression
tests and performance tests
15. regulated FX exchange
London & Tokyo
launched 2010
~ $2 trillion traded in 2015
one of UK’s fastest growing technology companies
16. high quality - very low rate of production issues
(order of magnitude below industry average)
“lowest bug count we’ve ever found”
- independent analysis by a well-known tool vendor
17. 2 million lines of code
(50:50 test/production)
half the codebase < 18 months old
26. Test Isolation
• Reliability/Repeatability
- Provide consistent, repeatable results.
• Isolation
- Tests should not depend, or be affected by, the results of
other tests: testContext
- users, accounts, advertiser names etc: alias name
- external services, 3rd-party integrations: stub, simulator
account: ‘Johnny’
==> ‘Johnny_4534031’
book: ‘DevOps 101’
==> ‘DevOps 101_1234567’
27. Parallelisation
• Commit stage < 15 mins
• Acceptance Test stage < 45 mins
• Fail FASTer!
• Radical parallelisation
- throw-away environments (e.g. containers)
28. Separation of Concern
• Use Domain Specific Language (DSL)
- Focus on “What” not “How”
• Ease of Development
- Hide details of how the tests talk to the SUT
• Ease of Maintenance
- When tests break, we Identify the problem and fix it quickly.
33. API Example
@Given("^I create a advertiser with those fields fill : name '(.+)', state '(.+)', currency '(.+)'$")
public void createAdvertiser(final String name, final String state, final String currency) {
advertiserAPI.createDefaultAdvertiser(name, state, currency);
}
package com.vpon.dsp.driver.web.rest.service.adv;
public class AdvertiserServiceImpl {
static { AdvertiserService advService = ServiceGenerator.createService(AdvertiserService.class); }
public static String createAdvertiser(Advertiser obj) throws IOException{
Call<Advertiser> advExec = advService.createAdvertiser(obj);
Response<Advertiser> resp = advExec.execute();
// id should not be null if created successfully
if(resp.isSuccessful() && null != resp.body().id) {
...
}}}
package com.vpon.dsp.dsl.web;
public String createDefaultAdvertiser(String name) {
Advertiser obj = defaultAdvertiser(name);
return AdvertiserServiceImpl.createAdvertiser(obj);
}
DSL layer:
Driver layer:
Test case layer:
39. Goal
@end2end
Feature: LineItem end-to-end system targeting test
Background: An Advertiser, LineItem with different System targeting conditions created
Given There is a 'advTargeting' advertiser and following 'System' targeting lineItems:
| name | category | targeting |
| LI_OS_iOS | OS Family | Apple iOS |
| LI_LANG_Eng | Language | English |
Scenario Outline: Bid accordingly with lineItem targeting setting
Given I enable '<lineItemName>' lineItem
When I send a bid request with '<criteria>' targeting only
And I send a 'default' bid request with no targeting criteria
Then I should receive '1' successful bid response
Examples:
| lineItemName | criteria |
| "LI_OS_iOS" | "iOS" |
| "LI_LANG_Eng" | "English" |
41. GUI (Flaky) Test
Stability is super important for automation in CD
• 1% failure rate with 100 test scripts ( 0.99¹⁰⁰) = 63% chance of failure
• You can’t get away with flaky tests in CI/CD
• Spend much time debugging false alarms
42. Selenium + jQuery Selector
• Implement JQueryBy class
- Tells Selenium how to find element with jQuery locator
• If no jQuery in application page
- Inject one for testing
• jQuery
- Powerful API for navigating and selecting content
- Manipulate UI for easy assertion
- CSS based, a whole lot better than XPath
- Has “qualified” feature (ex: has, contains) for filtering
• No xPath locator
- Slow & tight to DOM structure
- Easily broken with html changes
- Bad readability and hard to maintain
43. Selenium PageObject
• Separate test logic from web implementation
- No html stuff (locator, Selenium code, …) in test case
• Increase maintainability
- No need to update all hundreds of TCs when UI changes
- Object classes are the only place to modify
• Increase stability
- More tuning on the framework, it gets more stable
src: Martin
44. Handling Wait in UI Test
User Perform
Action
if_Async
CheckPageReady()
Check Ajax Request
Complete
Wait Dynamic
Element Presence
Make Assertion
Check DB / Queue
Status Match
Thread.sleep()
45. Handling Wait in UI Test
• Each page loading is following the same ‘wait’ checking procedure
• Each page can define it’s own ‘getPageLoadCondition()’
public T initPage(Class<T> clazz) {
T page = PageFactory.initElements(getDriver(), clazz);
checkPageReady(); //<== check for DOM, JS lib
ExpectedCondition pageLoadCondition =
((AbstractPage) page).getPageLoadCondition();
// for extended page does not want to check loading condition ...
if (null != pageLoadCondition) {
waitForCondition(pageLoadCondition); //<== check for page’s load condition
}
return page;
}
46. Handling Wait in UI Test
private static void checkPageReady() {
/* for generic loading, but not all browsers compatible...
final String chk1 =
“(document.readyState==='complete'|| document.readyState===‘interactive')";
*/
final String chk1 =
"(typeof jQuery != 'undefined') && ($(window).load(function() {return true;}))";
//for jQuery loaded and no ajax requests pending
final String chk2 = "($.active === 0)";
final String condition = "return " + chk1 + " && " + chk2 + ";";
ExpectedCondition<Boolean> response = new ExpectedCondition<Boolean>() {
public Boolean apply(WebDriver d) {
return (Boolean)((JavascriptExecutor) d).executeScript(condition);
}
};
new WebDriverWait(getDriver(), LOAD_TIMEOUT).until(response);
}
47. //for those with data grid loading
public static ExpectedCondition getDataLoadingCondition() {
LogUtils.printDebugMessage(logger, "Before returning _loading wait condition...");
return new ExpectedCondition<Boolean>() {
public Boolean apply(WebDriver d) {
boolean result = (Boolean)((JavascriptExecutor) d).executeScript(
"return ($("div[id$='_loading'][style*='display: none;']").length===2" +
" && $.active===0 “ +
“ && $("div[id$='_loading'][style$='display: block;']").length===0)"
);
return result;
}
};
}
Wait Ajax Data Loading
(1) $.active === 0
=> No ajax reqs pending
(2) $(”div[id$=‘_loading’]…..”).length===0)
=> loading icon disappear
48. Wait Dynamic Web Element
public String getToolTipText() {
SeleniumDriver.onPage(ByJQuery.jQuerySelector("div#content"));
. . . . .
//mouseover
new Actions(getDriver()).moveToElement(e).build().perform();
String toolTipLocator = "div.tooltip[style$='display: block;']";
LogUtils.printMessage(logger, "trying to get tooltips of a warning ...");
return SeleniumDriver.waitUntilElementPresence(
ByJQuery.jQuerySelector(toolTipLocator)).getText();
}
49. Capacity Testing
• Performance testing for components
• Long run and stress
• Use production traffic - boost confidence level
• Tools: Gor, Gatling
50. Reference
[How google Test Software]
[Continuous Delivery: Reliable Software Releases]
[From research paper “Moving Fast with Software Verification - 2015”]
Others:
https://dzone.com/articles/dev-centric-culture-breaking-down-the-walls
http://blog.xebialabs.com/2015/06/22/guidelines-for-a-successful-test-strategy/
http://www.androidwarriors.com/2015/12/retrofit-20-android-example-web.html
https://gortool.com/
http://gatling.io/#/
[About Vpon]
[About Me]