The document describes the evolution of test automation processes at Trend Micro for their Deep Security product. It discusses moving from manually managed virtual machine environments to using Ansible for automated and scalable environment setup. It also describes transitioning from the JUnit test framework to the custom-built Deep Security Automation Framework (DSAF) to improve test robustness. Additionally, it outlines the development of various automation tools and dashboards including Lift for installation/upgrade testing, Mothra for consolidated test results, and the use of Elasticsearch, Grafana, and Prometheus for data collection, visualization, and monitoring.
5. Copyright 2018 Trend Micro Inc.6
Deep Security
• Deep Security is a collaborative effort between Taiwan and
Ottawa, with about 200 R&D people working on both sides
6. Copyright 2018 Trend Micro Inc.7
Deep Security
• Aside from the 12/13-hour communication gap, we also
have to work through the language and cultural barriers
• Since we are such a diverse team, we have to be able to
work together to release Deep Security
7. Copyright 2018 Trend Micro Inc.8
Automation in Deep Security
• I’ve been in the DS automation team since I joined Trend
about x years ago
• Over the years, automation has evolved from just a simple
test tool, to a team-wide integration of DevOps principles
• The revolution has been quite beautiful !
8. Copyright 2018 Trend Micro Inc.9
Automation in Deep Security
Deep Security Automation Framework
13. Copyright 2018 Trend Micro Inc.14
Introducing the Rogue One team
• We are a 3-person team responsible for
– Install / upgrade automation
– Test case development
– Innovating automation practices
– Provide guidance for feature teams for anything related to automation
• In Ottawa, we have a counterpart called the Phoenix team
15. Copyright 2018 Trend Micro Inc.16
• Initial goal of our team is to revamp the existing install /
upgrade automation to make it more efficient, flexible and
user friendly
• So we made Lift.
18. Copyright 2018 Trend Micro Inc.19
Environment Creation
• The main parts of install / upgrade automation is to be able
to spin up a host on the fly for testing
• In the old system, this is done by having a pool of VMs (all
supported platforms has one VM), and keeping a CLEAN
snapshot of each VM
• Whenever a test is started, this VM is started up and
reverted to the CLEAN state
• Guess what kind of issues we had with this setup?
19. Copyright 2018 Trend Micro Inc.20
Environment Creation
• Issues:
– Only one test can run at a time
– Long test times
– If something fails and needs attention, other tests in sequence
cannot run
– If something needs updating, all snapshots need to be updated
20. Copyright 2018 Trend Micro Inc.21
Environment Creation
• Ansible is an open-source tool for environment
orchestration.
• Ansible can help you with config management, deployment
and task automation.
21. Copyright 2018 Trend Micro Inc.22
Ansible
• Why use Ansible?
– No Agent
• SSH / Python
• Fast deployment
– Use YAML
• Easy to understand
– Many available modules
– Open-source
– Small learning curve
– Ansible Tower
23. Copyright 2018 Trend Micro Inc.24
Test Cases Framework
• JUnit was the first test framework for DS
• It was initially made to create automation tests for the FW /
DPI feature
• Overtime, it grew to support all features of DS, including AM
• Without a proper PR process, it became too big, too dirty
(dependencies everywhere! !) and very hard to maintain
• So we made DSAF.
24. Copyright 2018 Trend Micro Inc.25
Test Cases Framework
• One of the reasons why we moved from JUnit to DSAF, is
because of flaky tests
• A test could fail ONCE out of 10x
• It’s a big problem especially when you run Nightly tests on
multiple combinations of environments
• How do you classify that test?
– Do you fix it?
– Do you remove it?
– What do you do with it?
28. Copyright 2018 Trend Micro Inc.29
Promote Solution
• The most difficult point is not task, it’s
human.
– Team member change
– Knowledge
– I have my own tool, script. That’s enough.
– Data analysis
– Fix once, solve everywhere
29. Copyright 2018 Trend Micro Inc.30
Promote Solution - Seed team
TaiwanCanada
Feature
Team
Feature
Team
Feature
Team
Feature
Team
Feature
Team
Feature
Team
30. Copyright 2018 Trend Micro Inc.31
Promote Solution - Continue
• Pull
– New features
– Utility, Tool and Report
• Push
– End old service
– Manager
40. Copyright 2018 Trend Micro Inc.41
Targets
• Disk
• Memory
• CPU
• Job queue
• Garbage
41. Copyright 2018 Trend Micro Inc.42
Auto Alert to Auto Fix
• Disk -> remove some files
• Memory, CPU -> record, scale up
• Job queue -> record, scale up
• Garbage -> remove it!
49. Copyright 2019 Trend Micro Inc.50
Elasticsearch is a real-time distributed and open
source full-text search and analytics engine. It is
accessible from RESTful web service interface and
uses schema less JSON (JavaScript Object Notation)
documents to store data.
56. Copyright 2019 Trend Micro Inc.57
Data
Collection
What’s the
question we
want to ask?
A query for
answer the
question
Data
Visualization
Dashboard for
each question
How we made them…
60. Copyright 2018 Trend Micro Inc.61
Zero-Click
• Have you wondered if your branch is sane enough to be merged to the
main branch?
• Have you wondered if the branches being merged into main branch are
safe?
• Have you ever encountered a bug after PR caused by a merge from a
buggy or untested branch?
• How much do you trust your own code?
64. Copyright 2018 Trend Micro Inc.65
Zero-Click
• Runs smoke tests before merging code to main branch
• Auto-triggered for every PR, every commit
• Takes about 40 mins to run
• Failed 0-click job means a BUG, and will BLOCK your PR
• A guarantee that your branch is shippable
66. Copyright 2018 Trend Micro Inc.67
Zero-Click
• Zero-Click was the first collaboration between the automation teams in Taiwan
and Ottawa
• We spent 2 weeks, working closely together in one room in the Ottawa office
68. Copyright 2018 Trend Micro Inc.69
Multi pipelines across Canada & Taiwan
• Inconsistent data on TestRail
• Overview on all test environments
• The most important factors
– Build number
– Platform
– Feature
– More?
69. Copyright 2018 Trend Micro Inc.70
Issues
• Detail and Log
• New database or Existed database?
• ElasticSearch or PostgreSQL?
• How to push test result?
– Command line on multi-platform
• Dashboard for everyone, by everyone
71. Copyright 2018 Trend Micro Inc.72
Platform name
• TestRail Config ID
• Mothra ID
• Mapping table on every environment
• Mapping table on Mothra
– Allow strings
75. Copyright 2018 Trend Micro Inc.76
What are the problems ROSE is
trying to solve?
76. Copyright 2019 Trend Micro Inc.77
1. Where can I find the automation pipeline and test suite to test my code
change?
2. Are the tests from Zero-Click enough to test my code change (PR)?
3. Is it possible to find the relationship between the source code and the
existing tests so that we can determine which tests should run for the code
changes?
4. What’s the code coverage rate of our automation testing?
5. What are the features we don’t have tests for?
77. Copyright 2018 Trend Micro Inc.78
Automation
Pipelines
Run Test
Collect Test
Coverage Data
Mapping
Coverage Data
with Tests
Developer
Create a Pull
Request
Analyze Code
Change and
Recommend
Covered Tests
Run
Recommended
Tests Before the
Pull Request
Pass the
Tests and
Finish the
Pull Request
78. Copyright 2018 Trend Micro Inc.79
How do we find the relationship
between the source code and testing
in DSM?
97. Copyright 2018 Trend Micro Inc.98
Product
Automation Testing
System Automation
Data Collection
Data Visualization
Data Analysis
Machine Learning
Microservices