In this advanced session, we will investigate all the ways that you can automate your testing processes with TestBox and many CI and automation tools. From Jenkins integration, Travis CI, Node runners, Grunt watchers and much more. This session will show you the value of continuous integration and how to apply it with modern tools and technologies. Main Points Why we want to automate Continuous Integration ANT/CommandBox Test Runner Setup of a Jenkins CI server Travis CI integration Pipelines CI integration Node TestBox Runners Grunt Watchers and Browser Live Reloads

1. AUTOMATE
THYSELF
Luis Majano @lmajano @ortussolutions

2. WHO AM I?
• Luis Majano
• Computer Engineer
• El Salvador
• CEO of Ortus Solutions
• Creator of all things Box
• Sandals & Beaches -> ESRI ->
Ortus

3. Story of Why People don’t test
TestingTools
ExecutingTests
Automation
Continuous Integration
NOT ABOUT:
BDD
Basics of Testing

4. Where to start?
Test Harness
What to test?
What not to test?

5. WHY PEOPLE DON’T
TEST
COMFORT

6. WHY PEOPLE DON’T
TEST
New
Methodology
New Learned
Behavior

7. BIGGEST LIES IN SOFTWARE
Don’t worry, we will
create the tests and
refactor it later!
Testing will double my
estimates on time

8. HOW DO YOU KNOW YOUR
FEATURE IS DONE?
Manual testing is only a momentary satisfaction!

9. HOW IT WORKS?

10. Bugs cost money
Cost To Fix Time detected
Requirements Design Building Testing Post-Release
Time
Introduced
Requirements 1x 3x 5-10x 10x 10-100x
Design --- 1x 10x 15x 25-100x
Building -- -- 1x 10x 10-25x
^ Kaner, Cem; James Bach, Bret Pettichord (2001). Lessons Learned in Software Testing: A Context-Driven Approach. Wiley. p. 4. ISBN 0-471-08112-4.

11. • Just do it!
• You will get dirty
• It can hurt (a little)
• Learned behavior
NO MORE EXCUSES
It will accelerate your development!

12. Continuous Integration
Continuous Delivery
Continuous Improvement
AUTOMATION

13. AGENDA
• Proven event-driven ColdFusion Framework
• Follows OO principles and best practices
• Conventions Based
• Non-intrusive framework you code in ColdFusion
• More than MVC => ColdBox Platform
• SoftwareToolkit
• Remote Framework
• NOT A MAGIC FRAMEWORK,YOU STILL NEEDTO CODE!!

14. cbVue application
install coldbox-samples/cbVue

15. TestBox Exposed
API Docs
Core
Test Browser
Test Harness
Test Runner
Samples
install testbox —saveDev
install testbox@be —saveDev

16. Test Harness
Automated test
results!
xUnit/BDDTest
Bundles
Harness bootstrap
HTML Runner
ANT Runner

17. Test Bundle CFCs
• Inherits from testbox.system.BaseSpec

18. RUN TESTS
• Execute bundles via the URL
• http://localhost/tests/bundle.cfc
• SOAP Runner
• HTTP/REST Runner
• ANT Runner
• NodeJS Runner
• CommandBox Runner
• Custom Runner

19. HOW IT WORKS?
TESTBOX REPORTERS
•ANTJunit :Variant of JUnit for Jenkins
•Codexwiki : Mediawiki syntax
•Console : Server console
•Doc : Semantic HTML
•Dot :Awesome Dots
•JSON
•JUnit
•Raw : CFML Struct
•Simple :A basic HTML reporter
•Text : Back to the 80’s,Text
•MinText : Great for CommandBox Runners
•XML
•Tap :A test anything protocol reporter
•Min : Minimalistic Report

20. Developer Automation

21. HOW IT WORKS?
• Execute bundles via the URL
• http://localhost/tests/bundle.cfc?method=runRemote
• SOAP Runner
• HTTP/REST Runner
• ANT Runner
• NodeJS Runner
• CommandBox Runner
• Custom Runner
Bundle CFC Runner

22. HTML Runner

23. CommandBox Runner
Update box.json with runner location

24. testbox run

25. {
"runner": "http://127.0.0.1:57538/tests/
runner.cfm",
"directory": "/tests/specs",
"recurse": true
}
NodeJS Test Runner
npm install -g testbox-runner
create .testbox-runnerrc
1
2
Run it : testbox-runner2

26. HOW IT WORKS?
• Execute bundles via the URL
• http://localhost/tests/bundle.cfc?method=runRemote
• SOAP Runner
• HTTP/REST Runner
• ANT Runner
• NodeJS Runner
• CommandBox Runner
• Custom Runner
ANT TESTING

27. HOW IT WORKS?
DEVELOPER
AUTOMATION
1.CommandBox Watchers
2.Grunt Watchers
3.Grunt Watchers + testbox-runner
4.Grunt Watchers + CommandBox
5.ColdBox Elixir

28. HOW IT WORKS?
Code -> Tests -> Commit ->
Pull -> Run -> Results ->
Refactor -> Repeat ->
CONTINUOUS INTEGRATION

29. HOW IT WORKS?
Code -> Tests -> Commit ->
Pull -> Run -> Results ->
Refactor -> Repeat ->
CI SERVERS

30. • Luis Majano
• Computer Engineer
Jenkins + CommandBox
wget http://mirrors.jenkins-ci.org/war-stable/latest/jenkins.war
box server start warpath=jenkins.war
1
2

31. Jenkins Features
•FREE - Open Source Java application
•Not so easy to configure and use
•CI = Jenkins Jobs
•Thousands of plugins and extensions
•Jobs can be scheduled, parallelized, executed over different
machines
•Jenkins Slaves
•Powerful but convoluted

32. Travis CI - Sign Up
https://travis-ci.org/
Sign Up
1
2

33. Travis Features
•FREE for Open Source Projects
•Runs distributedVM’s and Container Support
•Triggers Build Script via git repository commits (.travis.yml)
•Multiple language support
•Many integrations and extensions
•Many notification types
•No ability to schedule/manual builds
•Great for open source projects!

34. Accounts
Activate For Repositories
1
2
Create .travis.yml3
Commit to repository4
Travis CI - Getting Started
CI Script runs on
each commit, pull
request, branches

35. TRAVIS + ANTlanguage: java
branches:
only:
- development
sudo: required
dist: trusty
before_install:
- sudo apt-key adv --keyserver keys.gnupg.net --recv 6DA70622
- sudo echo "deb http://downloads.ortussolutions.com/debs/noarch /" |
sudo tee -a
/etc/apt/sources.list.d/commandbox.list
install:
- sudo apt-get update && sudo apt-get --assume-yes install commandbox
- box install
- box server start
script:
- ant -f tests/build.xml

36. TRAVIS +
COMMANDBOXlanguage: java
sudo: required
dist: trusty
before_install:
- sudo apt-key adv --keyserver keys.gnupg.net --recv 6DA70622
- sudo echo "deb http://downloads.ortussolutions.com/debs/noarch /" |
sudo tee -a
/etc/apt/sources.list.d/commandbox.list
install:
- sudo apt-get update && sudo apt-get --assume-yes install commandbox
- cd tests
- box install
- box server start
script:
- box testbox run

37. TRAVIS + MULTIPLE
ENGINES
language: java
sudo: required
env:
matrix:
- ENGINE=lucee@4.5
- ENGINE=lucee@5
- ENGINE=adobe@2016
- ENGINE=adobe@11
- ENGINE=adobe@10
before_install:
- sudo apt-key adv --keyserver keys.gnupg.net --recv 6DA70622
- sudo echo "deb http://downloads.ortussolutions.com/debs/noarch /" | sudo tee -a
/etc/apt/sources.list.d/commandbox.list
install:
- sudo apt-get update && sudo apt-get --assume-yes install commandbox
- box install
before_script:
- box server start cfengine=$ENGINE port=8500
script:
- box testbox run

38. Gitlab
https://gitlab.com/users/sign_in
Sign Up
1
2
or Self-Hosted3

39. Gitlab Features
• All in one tool: CI, repository, docker registry, issues/milestones, etc.
• Same asTravis in concept - CI Runner
• Docker based
• CI Runners can be decoupled to a docker swarm
• Idea of CI pipelines
• Pipelines composed of jobs executed in stages
• Jobs can have dependencies, artifacts, services, etc
• Schedule Pipelines (Cron)
• Jobs can track environments
• Great stats + charting

40. .GITLAB-CI.YML
image: ortussolutions/commandbox:alpine
stages:
- build
- test
- deploy
build_app:
stage: build
script:
# Install dependencies
- box install production=true
run_tests:
stage: test
only:
- master
# when: manual, always, on_success, on_failure
script:
- box server start
- box testbox run

41. Resources
• TestBox : ortussolutions.com/products/testbox
• CommandBox: ortussolutions.com/products/commandbox
• Slack: boxteam.herokuapp.com
• CFML Slack: #box-products
• travis-ci.org
• jenkins.io
• gitlab.com
• Twitter
• @lmajano
• @ortussolutions

42. Q & A