Using Acceptance Test Driven Approach to Security Test, easy for stakeholder involvement, and a whole team approach to address security needs in software development process. Examples in this session are based on RobotFramework, making it easy for people with limited programming skills.

1. Continuous Security Testing
Acceptance Test Driven Approach
Sunday, 15 December, 13

2. Who am I?
•Agile, TDD Coaching, Ugly Code
Cleaning Dude
•I love coding - Java, C#, Javascript, C/
C++, PHP, Perl, and some weird ones
•I speak English, Cantonese, and
Mandarin
2
Odd-e Pte. Ltd.
Steven Mak 麥天志
Agile Coach
Hong Kong
Email: steven@odd-e.com
Web: www.odd-e.com
Twitter: stevenmak
Sunday, 15 December, 13

3. Do you automate your tests?
3
Sunday, 15 December, 13

4. Is that what you feel?
4
Script Unreadable?
Keep Changing?
Time consuming
to write?
Sunday, 15 December, 13

5. Technical
Activity
Workflow
Specification pyramid
5
RuleClarity
Stability
Specification
Users can
understand
Automation
Technical
Sunday, 15 December, 13

6. Use Examples
6
With 3 judges giving
scores 4, 20, and 18,
the displayed score
should be 42.
When the first 2
judges have given
their scores, e.g. 10
and 5, the
intermediate score of
15 should be displayed
already.
No scores displayed as
a dash (–), not zero.
Maximum score from
a judge is 20 points!
Sunday, 15 December, 13

7. Examples, Tests, and Spec
7
Examples Tests
Requirements
can become
elaborate
verify
Sunday, 15 December, 13

8. More ideas from
• Threat Modelling
• Session-Based Test Management / Exploratory Testing
• Product Requirement
• Experts
8
Sunday, 15 December, 13

9. Avoid handoff
9
Sunday, 15 December, 13

10. Avoid imperative
• login
• enter username
• enter password
• enter homepage
• click category
• choose product
• put it on shopping cart
• click generate order
• .....
10
Sunday, 15 December, 13

11. Avoid imperative
• login
• enter username
• enter password
• enter homepage
• click category
• choose product
• put it on shopping cart
• click generate order
• .....
11
Given I selected a doll in shopping cart
When I generate order
Then the order should contain doll
and the price is 83.55
Sunday, 15 December, 13

12. Avoid imperative
• login
• enter username
• enter password
• enter homepage
• click category
• choose product
• put it on shopping cart
• click generate order
• .....
12
Given I selected a doll in shopping cart
When I generate order
Then the order should contain doll
and the price is 83.55
This “Given When Then” is a
common pattern called Gherkin
Sunday, 15 December, 13

13. Good ones
• Focus on business, not software design
• Not coupled with code
• Not coupled with UI
• Concise
• Use domain languages
13
Getting us towards Living Documentation
and can be executed against existing
system
Sunday, 15 December, 13

14. Robot Framework
www.robotframework.org
14
Sunday, 15 December, 13

15. Test Tools
Robot Architecture
15
Test Data (Tables)
Robot Framework
Test Libraries
System Under Test
Test Library API
application interfaces
Robot comes with a number of built-in test libraries and
you can (should!) add your own.
Test libraries can use any test tool necessary to interact
with the system under test.
Sunday, 15 December, 13

16. It's all in the tables
16
Sunday, 15 December, 13

17. Test Cases are composed of
keyword-driven actions
17
!"#$%&'()*+%),'-./()0
Sunday, 15 December, 13

18. Test Cases are composed of
keyword-driven actions
17
!"#$%&'()*+%),'-./()0
this is the name of a test case
Sunday, 15 December, 13

19. Test Cases are composed of
keyword-driven actions
17
!"#$%&'()*+%),'-./()0
this is the name of a test case
these keywords form the test case
Sunday, 15 December, 13

20. Test Cases are composed of
keyword-driven actions
17
!"#$%&'()*+%),'-./()0
this is the name of a test case
these keywords form the test case
keywords receive arguments
Sunday, 15 December, 13

21. 2 types of keywords
18
Sunday, 15 December, 13

22. 2 types of keywords
18
We can import keyword libraries for a test case
Sunday, 15 December, 13

23. 2 types of keywords
18
We can import keyword libraries for a test case
...and libraries may be configured, too.
Sunday, 15 December, 13

24. 2 types of keywords
18
We can import keyword libraries for a test case
...and libraries may be configured, too.
This keyword comes from the imported library.
Sunday, 15 December, 13

25. 2 types of keywords
18
We can import keyword libraries for a test case
...and libraries may be configured, too.
This keyword comes from the imported library.
This is a user keyword, implemented in table format.
(Think macros composed of other macros.)
Sunday, 15 December, 13

26. 19
Data-driven test cases
this is the name of a test case
these keywords form the test case
keywords receive arguments
Sunday, 15 December, 13

27. 20
using Template
*** Test Cases ***
Email Delivered Acceptance Rule
[Template] Confirm Email Delivered Workflow
sender@mail.com user@example.com 3asyp3asy 1
sender@nonexist.com user@example.com 3asyp3asy 0
*** Keywords ***
Confirm Email Delivered Workflow
[Arguments] ${sender} ${recipient} ${password} ${number_of_emails_expected}
Open Mail Box ${MAIL_SERVER} ${recipient} ${password}
Count Mail Received ${sender} ${number_of_emails_expected}
Keyword used as template
test data feed as arguments
Sunday, 15 December, 13

28. Given-when-then (BDD)
21
*** Test Cases ***
Addition
Given calculator has been cleared
When user types "1 + 1"
and user pushes equals
Then result is "2"
*** Keywords ***
Calculator has been cleared
Push button C
User types "${expression}"
Push buttons ${expression}
User pushes equals
Push button =
Result is "${result}"
Result should be ${result}
this is the name of a test case
these keywords form the test case
Sunday, 15 December, 13

29. Variables
22
!"#$"%&'(
)#*+,-*++"./,&$.'0
Sunday, 15 December, 13

30. Other choices
• Cucumber
• Fitnesse
23
Sunday, 15 December, 13

31. 24
An Example
Sunday, 15 December, 13

32. 25
*** Settings ***
Resource resource.txt
*** Test Cases ***
Checking Opened Ports [Template] Only these ports are opened
22 25 80 135 139 445
*** Keywords ***
Only these ports are opened [Arguments] @{expected_ports}
@{actual_ports_opened}= Scan with Fast Mode ${HOST}
List Should Contain Sub List ${actual_ports_opened} ${expected_ports}
*** Settings ***
Library nmapLibrary
Library Collections
*** Variables ***
${HOST} www.scrumprimer.org
import nmap
class nmapLibrary:
def scan_with_fast_mode(self, host):
nm = nmap.PortScanner()
nm.scan(str(host), arguments="-F")
return [str(port) for port in nm[str(nm.all_hosts()[0].encode())].all_tcp()]
resource.txt
port_scanning.txt
nmapLibrary.py (with python-nmap)
Sunday, 15 December, 13

33. 26
pybot -d output nmap.txt
==============================================================================
Port Scaning
==============================================================================
Checking Openned Ports | PASS |
------------------------------------------------------------------------------
Nmap | PASS |
1 critical test, 1 passed, 0 failed
1 test total, 1 passed, 0 failed
==============================================================================
Output: /Users/stevenmak/Work/robotframework/securityTests/2013.12.14VXCon/output/output.xml
Log: /Users/stevenmak/Work/robotframework/securityTests/2013.12.14VXCon/output/log.html
Report: /Users/stevenmak/Work/robotframework/securityTests/2013.12.14VXCon/output/report.html
run the test:
report:
(also available in xml format for Jenkins integration)
Sunday, 15 December, 13

34. 27
Sunday, 15 December, 13

35. More to wrap & integrate
• w3af
• garmr
• arachni
• dirb
• sslyze
• sqlmap
28
Sunday, 15 December, 13

36. Acceptance Test Driven
Development
29
Discuss
in workshop
Develop
in concurrence
Deliver
for acceptance
Sunday, 15 December, 13

37. 30
Discuss
in workshop
Develop
in concurrence
Deliver
for acceptance
Focus on customer
collaboration and user
engagement. Try to get as
many of these people attend
as you can.
Product Owner
Dev Team
Users
IT operations
Help Desk
Tech Writers
?
Sunday, 15 December, 13

38. 31
Discuss
in workshop
Develop
in concurrence
Deliver
for acceptance
With 3 judges giving
scores 4, 20, and 18,
the displayed score
should be 42.
When the first 2
judges have given
their scores, e.g. 10
and 5, the
intermediate score of
15 should be displayed
already.
No scores displayed as
a dash (–), not zero.
Maximum score from
a judge is 20 points!
Sunday, 15 December, 13

39. 31
Discuss
in workshop
Develop
in concurrence
Deliver
for acceptance
With 3 judges giving
scores 4, 20, and 18,
the displayed score
should be 42.
When the first 2
judges have given
their scores, e.g. 10
and 5, the
intermediate score of
15 should be displayed
already.
No scores displayed as
a dash (–), not zero.
Maximum score from
a judge is 20 points!
Robot tests are written in tables
so that computers can read them
Sunday, 15 December, 13

40. 32
Deliver
for acceptance
Discuss
in workshop
Develop
in concurrence
Sunday, 15 December, 13

41. Collaboration is key
33
team gets
feedback
earlier
scope of work
is clear and
understood by
all
team
understands
what they're
implementing
shared
language and
vocabulary is
built
team
collaborates
closely with
product owner
Sunday, 15 December, 13

42. CITCON Hong Kong
34
• When: Apr 11 & 12, 2014
• Cost: Free
• Registration: contact me
• Sponsorship Welcome!
http://citconf.com/hongkong2014/
Sunday, 15 December, 13

43. Thank you for spending time with me this evening.
More feedback can be sent to:
35
Odd-e Hong Kong Ltd.
Steven Mak 麥天志
Agile Coach
Hong Kong
Email: steven@odd-e.com
Web: www.odd-e.com
Twitter: stevenmak
Sunday, 15 December, 13