Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

From zero to one - How we evolved our test automation processes and mindset in Deep Security

Test automation experience sharing from Trend Micro

  • Login to see the comments

From zero to one - How we evolved our test automation processes and mindset in Deep Security

  1. 1. From Zero to One How we evolved our test automation processes and mindset in Deep Security
  2. 2. Copyright 2018 Trend Micro Inc.2 Introduction
  3. 3. Copyright 2018 Trend Micro Inc.4 What is Deep Security?
  4. 4. Copyright 2018 Trend Micro Inc.5 What is Deep Security?
  5. 5. Copyright 2018 Trend Micro Inc.6 Deep Security • Deep Security is a collaborative effort between Taiwan and Ottawa, with about 200 R&D people working on both sides
  6. 6. Copyright 2018 Trend Micro Inc.7 Deep Security • Aside from the 12/13-hour communication gap, we also have to work through the language and cultural barriers • Since we are such a diverse team, we have to be able to work together to release Deep Security
  7. 7. Copyright 2018 Trend Micro Inc.8 Automation in Deep Security • I’ve been in the DS automation team since I joined Trend about x years ago • Over the years, automation has evolved from just a simple test tool, to a team-wide integration of DevOps principles • The revolution has been quite beautiful !
  8. 8. Copyright 2018 Trend Micro Inc.9 Automation in Deep Security Deep Security Automation Framework
  9. 9. Copyright 2018 Trend Micro Inc.10 Automation in Deep Security 13000 test cases
  10. 10. Copyright 2018 Trend Micro Inc.11 Automation in Deep Security release branch feature branch
  11. 11. Copyright 2018 Trend Micro Inc.12 Automation in Deep Security
  12. 12. Copyright 2018 Trend Micro Inc.13 From Zero to One
  13. 13. Copyright 2018 Trend Micro Inc.14 Introducing the Rogue One team • We are a 3-person team responsible for – Install / upgrade automation – Test case development – Innovating automation practices – Provide guidance for feature teams for anything related to automation • In Ottawa, we have a counterpart called the Phoenix team
  14. 14. Copyright 2018 Trend Micro Inc.15 9.5 9.6 10.0 12.0 11.0 Established 2017
  15. 15. Copyright 2018 Trend Micro Inc.16 • Initial goal of our team is to revamp the existing install / upgrade automation to make it more efficient, flexible and user friendly • So we made Lift.
  16. 16. Copyright 2018 Trend Micro Inc.17 Lift
  17. 17. Copyright 2018 Trend Micro Inc.18 Lift
  18. 18. Copyright 2018 Trend Micro Inc.19 Environment Creation • The main parts of install / upgrade automation is to be able to spin up a host on the fly for testing • In the old system, this is done by having a pool of VMs (all supported platforms has one VM), and keeping a CLEAN snapshot of each VM • Whenever a test is started, this VM is started up and reverted to the CLEAN state • Guess what kind of issues we had with this setup?
  19. 19. Copyright 2018 Trend Micro Inc.20 Environment Creation • Issues: – Only one test can run at a time – Long test times – If something fails and needs attention, other tests in sequence cannot run – If something needs updating, all snapshots need to be updated
  20. 20. Copyright 2018 Trend Micro Inc.21 Environment Creation • Ansible is an open-source tool for environment orchestration. • Ansible can help you with config management, deployment and task automation.
  21. 21. Copyright 2018 Trend Micro Inc.22 Ansible • Why use Ansible? – No Agent • SSH / Python • Fast deployment – Use YAML • Easy to understand – Many available modules – Open-source – Small learning curve – Ansible Tower
  22. 22. Copyright 2018 Trend Micro Inc.23 Test Case Frameworks
  23. 23. Copyright 2018 Trend Micro Inc.24 Test Cases Framework • JUnit was the first test framework for DS • It was initially made to create automation tests for the FW / DPI feature • Overtime, it grew to support all features of DS, including AM • Without a proper PR process, it became too big, too dirty (dependencies everywhere! !) and very hard to maintain • So we made DSAF.
  24. 24. Copyright 2018 Trend Micro Inc.25 Test Cases Framework • One of the reasons why we moved from JUnit to DSAF, is because of flaky tests • A test could fail ONCE out of 10x • It’s a big problem especially when you run Nightly tests on multiple combinations of environments • How do you classify that test? – Do you fix it? – Do you remove it? – What do you do with it?
  25. 25. Copyright 2018 Trend Micro Inc.26 Test Cases Framework • Test Case Robustness
  26. 26. Copyright 2018 Trend Micro Inc.27 Test Cases Framework
  27. 27. Copyright 2018 Trend Micro Inc.28 Promote Solution to QA / SDET
  28. 28. Copyright 2018 Trend Micro Inc.29 Promote Solution • The most difficult point is not task, it’s human. – Team member change – Knowledge – I have my own tool, script. That’s enough. – Data analysis – Fix once, solve everywhere
  29. 29. Copyright 2018 Trend Micro Inc.30 Promote Solution - Seed team TaiwanCanada Feature Team Feature Team Feature Team Feature Team Feature Team Feature Team
  30. 30. Copyright 2018 Trend Micro Inc.31 Promote Solution - Continue • Pull – New features – Utility, Tool and Report • Push – End old service – Manager
  31. 31. Copyright 2018 Trend Micro Inc.32 How – Code Review • Code Review – Collaborator – GitHub
  32. 32. Copyright 2018 Trend Micro Inc.33 How – Code Review • Change Size • Review Time < 1 hour • Coding Rule
  33. 33. Copyright 2018 Trend Micro Inc.34 How – Code Review
  34. 34. Copyright 2018 Trend Micro Inc.35 How – Code Review • Comment • Code as Document
  35. 35. Copyright 2018 Trend Micro Inc.36 How – Continue • Wiki – Guide – Document – Help • Jira Cases
  36. 36. Copyright 2018 Trend Micro Inc.37 Resource / Performance / Dashboard for Environment
  37. 37. Copyright 2018 Trend Micro Inc.38 What issues we encountered • Guess and Try • Evidence • Performance
  38. 38. Copyright 2018 Trend Micro Inc.39 Why dashboard • Take aim & Focus • Ignore detail information • 2 dimensions for human • Learning curve • Sporadic
  39. 39. Copyright 2018 Trend Micro Inc.40 Tools • Grafana • InfluxDB • Prometheus • Jenkins • ElasticSearch
  40. 40. Copyright 2018 Trend Micro Inc.41 Targets • Disk • Memory • CPU • Job queue • Garbage
  41. 41. Copyright 2018 Trend Micro Inc.42 Auto Alert to Auto Fix • Disk -> remove some files • Memory, CPU -> record, scale up • Job queue -> record, scale up • Garbage -> remove it!
  42. 42. Copyright 2018 Trend Micro Inc.43 @app.route('/alert', methods=['POST']) def get_alert(): j = request.get_json(force=True) print j return "SUCCESS"
  43. 43. Copyright 2018 Trend Micro Inc.44
  44. 44. Copyright 2018 Trend Micro Inc.45 Dashboard for Test Result
  45. 45. Copyright 2019 Trend Micro Inc.46 Test Result on Testrail…..
  46. 46. Copyright 2019 Trend Micro Inc.47 Test Result on Jenkins……
  47. 47. Copyright 2019 Trend Micro Inc.48 Jenkinis API Python
  48. 48. Copyright 2019 Trend Micro Inc.49 What about the volume of logs?
  49. 49. Copyright 2019 Trend Micro Inc.50 Elasticsearch is a real-time distributed and open source full-text search and analytics engine. It is accessible from RESTful web service interface and uses schema less JSON (JavaScript Object Notation) documents to store data.
  50. 50. Copyright 2019 Trend Micro Inc.51 Push test results and logs to ES
  51. 51. Copyright 2019 Trend Micro Inc.52 DSL Query (Domain Specific Language) POST _search { "query": { "bool": { "should": [ { "match": { "job_name": "Lift/Integration/DSM_Upgrade/win2012R2x64_sql12_10_0" } } ] } } }
  52. 52. Copyright 2019 Trend Micro Inc.53 Visualize data from ES
  53. 53. Copyright 2019 Trend Micro Inc.54 Root Cause Analysis
  54. 54. Copyright 2019 Trend Micro Inc.55
  55. 55. Copyright 2019 Trend Micro Inc.56
  56. 56. Copyright 2019 Trend Micro Inc.57 Data Collection What’s the question we want to ask? A query for answer the question Data Visualization Dashboard for each question How we made them…
  57. 57. Copyright 2018 Trend Micro Inc.58 Our Works
  58. 58. Copyright 2018 Trend Micro Inc.59 • Zero-Click • Mothra • Rose
  59. 59. Copyright 2018 Trend Micro Inc.60 Zero-Click
  60. 60. Copyright 2018 Trend Micro Inc.61 Zero-Click • Have you wondered if your branch is sane enough to be merged to the main branch? • Have you wondered if the branches being merged into main branch are safe? • Have you ever encountered a bug after PR caused by a merge from a buggy or untested branch? • How much do you trust your own code?
  61. 61. Copyright 2018 Trend Micro Inc.62 Zero-Click
  62. 62. Copyright 2018 Trend Micro Inc.63 Zero-Click
  63. 63. Copyright 2018 Trend Micro Inc.64 Zero-Click
  64. 64. Copyright 2018 Trend Micro Inc.65 Zero-Click • Runs smoke tests before merging code to main branch • Auto-triggered for every PR, every commit • Takes about 40 mins to run • Failed 0-click job means a BUG, and will BLOCK your PR • A guarantee that your branch is shippable
  65. 65. Copyright 2018 Trend Micro Inc.66 SonarQube
  66. 66. Copyright 2018 Trend Micro Inc.67 Zero-Click • Zero-Click was the first collaboration between the automation teams in Taiwan and Ottawa • We spent 2 weeks, working closely together in one room in the Ottawa office
  67. 67. Copyright 2018 Trend Micro Inc.68 Consolidated Test Result (Mothra)
  68. 68. Copyright 2018 Trend Micro Inc.69 Multi pipelines across Canada & Taiwan • Inconsistent data on TestRail • Overview on all test environments • The most important factors – Build number – Platform – Feature – More?
  69. 69. Copyright 2018 Trend Micro Inc.70 Issues • Detail and Log • New database or Existed database? • ElasticSearch or PostgreSQL? • How to push test result? – Command line on multi-platform • Dashboard for everyone, by everyone
  70. 70. Copyright 2018 Trend Micro Inc.71 Solution
  71. 71. Copyright 2018 Trend Micro Inc.72 Platform name • TestRail Config ID • Mothra ID • Mapping table on every environment • Mapping table on Mothra – Allow strings
  72. 72. Copyright 2018 Trend Micro Inc.73
  73. 73. Copyright 2018 Trend Micro Inc.74 Big Data
  74. 74. Relationship of Source Code and Test System (ROSE !)
  75. 75. Copyright 2018 Trend Micro Inc.76 What are the problems ROSE is trying to solve?
  76. 76. Copyright 2019 Trend Micro Inc.77 1. Where can I find the automation pipeline and test suite to test my code change? 2. Are the tests from Zero-Click enough to test my code change (PR)? 3. Is it possible to find the relationship between the source code and the existing tests so that we can determine which tests should run for the code changes? 4. What’s the code coverage rate of our automation testing? 5. What are the features we don’t have tests for?
  77. 77. Copyright 2018 Trend Micro Inc.78 Automation Pipelines Run Test Collect Test Coverage Data Mapping Coverage Data with Tests Developer Create a Pull Request Analyze Code Change and Recommend Covered Tests Run Recommended Tests Before the Pull Request Pass the Tests and Finish the Pull Request
  78. 78. Copyright 2018 Trend Micro Inc.79 How do we find the relationship between the source code and testing in DSM?
  79. 79. Copyright 2019 Trend Micro Inc.80
  80. 80. Copyright 2018 Trend Micro Inc.81
  81. 81. Copyright 2018 Trend Micro Inc.82
  82. 82. Copyright 2018 Trend Micro Inc.83 COCO Listener COCO Web Service COCO Portal COCO Reports
  83. 83. Copyright 2018 Trend Micro Inc.84
  84. 84. Copyright 2018 Trend Micro Inc.85 How do we learn from data collected by COCO?
  85. 85. Copyright 2019 Trend Micro Inc.86 MATT (Machine Determined Testing)
  86. 86. Copyright 2019 Trend Micro Inc.87 Multi-label classification DSM Installation Anti-Malware
  87. 87. Copyright 2018 Trend Micro Inc.88 COCO Java File Java File Java File MATT Java File Java File Java File Unknown Java files Predicted Result
  88. 88. Copyright 2019 Trend Micro Inc.89 Result
  89. 89. Copyright 2019 Trend Micro Inc.90 Result (Cont.)
  90. 90. Copyright 2018 Trend Micro Inc.91
  91. 91. Copyright 2018 Trend Micro Inc.92 Data-Driven Decision Making
  92. 92. Copyright 2018 Trend Micro Inc.93 Code Coverage Data Decision/Learning Increase Code Coverage Testing
  93. 93. Copyright 2018 Trend Micro Inc.94 https://code.fb.com/developer-tools/predictive-test-selection/
  94. 94. Copyright 2018 Trend Micro Inc.95
  95. 95. Copyright 2018 Trend Micro Inc.96 Code Coverage
  96. 96. Copyright 2018 Trend Micro Inc.97 https://arxiv.org/abs/1810.05286
  97. 97. Copyright 2018 Trend Micro Inc.98 Product Automation Testing System Automation Data Collection Data Visualization Data Analysis Machine Learning Microservices
  98. 98. Copyright 2018 Trend Micro Inc.99 Q&A
  99. 99. Copyright 2018 Trend Micro Inc.100

×