This document discusses testing programmable infrastructure. It describes testing different aspects of infrastructure including a cloud broker and Kubernetes cluster. It recommends applying software development testing practices like unit testing, integration testing, and acceptance testing to infrastructure code and configuration. Various tools are presented for different types of infrastructure testing including Serverspec, Bats, Goss, Terraform_validate, and using Test Kitchen with Inspec. The document emphasizes that testing is important for infrastructure but often overlooked, and that testers and operations teams should work together to address new challenges in testing programmable systems.

1. MATT LONG
TESTING PROGRAMMABLE
INFRASTRUCTURE

2. PROGRAMMABLE
INFRASTRUCTURE IS GREAT, BUT
WE'RE MISSING SOMETHING.
TESTING.

3. I'M A TESTER
HELLO, I'M MATT

4. I WORK HERE ↑

5. I AM NOT A
SYSADMIN

6. WHAT IS
PROGRAMMABLE
INFRASTRUCTURE?

7. TESTING PROGRAMMABLE INFRASTRUCTURE
THE APPLICATION OF METHODS AND TOOLING
FROM SOFTWARE DEVELOPMENT TO
MANAGEMENT OF IT INFRASTRUCTURE
PROGRAMMABLE INFRASTRUCTURE IS..
THE INTERNET

8. TESTING PROGRAMMABLE INFRASTRUCTURE
EXAMPLES OF PROGRAMMABLE INFRASTRUCTURE
▸ Automated provisioning & configuration
▸ Configuration as code
▸ Version / source controlled

9. TESTING PROGRAMMABLE INFRASTRUCTURE
TOOLING EXAMPLES

10. PROGRAMMABLE
INFRASTRUCTURE
IS AWESOME!
Credit: Vault Boy, Bethesda Softworks

11. IT'S
FAST!

12. IT'S
AUTOMATIC!

13. IT'S ALL
CODE!

14. BUT IT GETS
COMPLEX

15. TESTING IS USED TO
MITIGATE COMPLEXITY
& RISK

16. BUT TESTING IS RARE
Credit: Gunshow, KC Green

17. TESTING PROGRAMMABLE INFRASTRUCTURE
WHAT I'M GOING TO TALK ABOUT
▸PART 1: Testing a cloud broker
▸PART 2: Building a Kubernetes cluster
▸CONCLUSIONS

18. TESTING A
CLOUD BROKER
AN INFRASTRUCTURE HEAVY PRODUCT

19. THE PROBLEM

20. TESTING PROGRAMMABLE INFRASTRUCTURE
WE WANT TO MOVE TO THE CLOUD...
BUT WE'RE WARY OF LOCK IN
Large organisation

21. TESTING PROGRAMMABLE INFRASTRUCTURE
USERS
USE MULTIPLE CLOUD PROVIDERS

22. TESTING PROGRAMMABLE INFRASTRUCTURE
PROBLEMS
▸ Different interfaces, feature sets & lingo
▸ Can't switch easily
▸ Spending difficult to track
▸ Temptation to fall back on most popular

23. TESTING PROGRAMMABLE INFRASTRUCTURE
USERS
CLOUD BROKER

24. TESTING PROGRAMMABLE INFRASTRUCTURE
BENEFITS
▸ Quick, easy provisioning
▸ one team previously took 3 months
▸ Common interface to cloud features
▸ Templates for common dev environments
▸ Built in best practice: monitoring, security
▸ Track spending

25. THIS IS A REALLY
COMPLICATED
APPLICATION

26. TESTING PROGRAMMABLE INFRASTRUCTURE

27. TESTING PROGRAMMABLE INFRASTRUCTURE
WORKFLOW
▸ Log into Web UI
▸ Fill in information about environment
▸ Broker creates and bootstraps resources
▸ SSH into resources

28. TESTING PROGRAMMABLE INFRASTRUCTURE
WEB TESTING
▸ Log into Web UI
▸ Fill in information about environment

29. TESTING PROGRAMMABLE INFRASTRUCTURE
???
▸ Broker creates and bootstraps resources
▸ SSH into resources

30. HOW DO YOU TEST
INFRASTRUCTURE?

31. TESTING PROGRAMMABLE INFRASTRUCTURE
WHAT TO TEST?
Do our deployment
scripts work?
Does the VPN server work?
Can instances
access one another?
Are services running?
Can I SSH into a server?

32. THIS SEEMS
FAMILIAR..

33. TESTING PROGRAMMABLE INFRASTRUCTURE
Does the VPN box work?
Can I SSH into a server?
Do our deployment scripts work?
Are services running?
ANOTHER TESTING PYRAMID?
credit: Ubuntu dev quality guide
https://developer.ubuntu.com/en/phone/platform/quality/
Can instances access one another?

34. TOOLING

35. TESTING PROGRAMMABLE INFRASTRUCTURE
TOOLS AVAILABLE
▸ Bats
▸ ShUnit2
▸ Goss
▸ ServerSpec / Inspec / TestInfra
▸ Test Kitchen

36. UNIT TESTING

37. TESTING PROGRAMMABLE INFRASTRUCTURE
BATS
▸ "Bash Automated Testing
System"
▸ Unit testing for bash
▸ Like JUnit

38. TESTING PROGRAMMABLE INFRASTRUCTURE
SH UNIT 2
▸ Shell unit testing framework
▸ Runs on all Bourne shells
▸ sh, BASH, DASH, ksh, zsh
▸ No activity or support?

39. INTEGRATION TESTING
OR: SERVER VALIDATION

40. TESTING PROGRAMMABLE INFRASTRUCTURE
GOSS
▸ Go based
▸ Specs in YAML
▸ Minimal, fast, and simple
▸ Some neat features
▸ .. have to run on the server
▸ .. no Windows support

41. TESTING PROGRAMMABLE INFRASTRUCTURE
SERVERSPEC
▸ Server based assertions
▸ Ruby/RSpec based
▸ Probably the most famous
▸ Can SSH into instances

42. TESTING PROGRAMMABLE INFRASTRUCTURE
INSPEC
▸ Written & maintained by Chef
▸ Very similar to ServerSpec
▸ Different feature set
▸ More focused on compliance

43. TESTING PROGRAMMABLE INFRASTRUCTURE
TESTINFRA
▸ ServerSpec, but in Python

44. TEST
HARNESS

45. TESTING PROGRAMMABLE INFRASTRUCTURE
TEST KITCHEN
▸ Orchestrates setup, test, teardown
▸ Runs BATS, shUnit2, RSpec,
Serverspec
▸ Popular in the Chef community
▸ Not suitable for our cloud broker

46. OUR
SOLUTION

47. TESTING PROGRAMMABLE INFRASTRUCTURE
USERS
CLOUD BROKER

48. TESTING PROGRAMMABLE INFRASTRUCTURE
USERS
WEB TEST FRAMEWORK

49. TESTING PROGRAMMABLE INFRASTRUCTURE
USERS
INFRASTRUCTURE TEST FRAMEWORK

50. TESTING PROGRAMMABLE INFRASTRUCTURE
USERS
WEB TESTS
https://github.com/opencredo/test-automation-quickstart

51. TESTING PROGRAMMABLE INFRASTRUCTURE
INFRASTRUCTURE TESTS
Serverspec

52. TESTING PROGRAMMABLE INFRASTRUCTURE
INFRASTRUCTURE TESTING STACK
/ Serverspec
???

53. TESTING PROGRAMMABLE INFRASTRUCTURE
WHY RUBY?
▸ Fantastic testing community
▸ More suitable for SSHing into boxes
▸ "Win RM" gem
▸ Ops already familiar with it
▸ Reduces tech stack

54. TESTING PROGRAMMABLE INFRASTRUCTURE
SERVERSPEC SMOKE TESTS
▸ Run before everything else
▸ Really quick
▸ Catches obvious errors
▸ Not complex tasks

55. TESTING PROGRAMMABLE INFRASTRUCTURE
SERVERSPEC EXAMPLE
describe package('jenkins') do
it { should be_installed }
end
describe service('jenkins') do
it { should be_enabled }
it { should be_running }
end
describe port(8080) do
it { should be_listening }
end

56. TESTING PROGRAMMABLE INFRASTRUCTURE
Background:
Given environment has been created
And the following user details:
| user_alias | username | public_key |
| userA | envoy | test |
Scenario: IPA - Login via SSH Key authentication succeeds
Given user "userA" is authorised to access environment vms
When user "userA" starts ssh session in host "env"
Then I should be able to echo "hello world"
CUCUMBER FOR ACCEPTANCE TESTING

57. TESTING PROGRAMMABLE INFRASTRUCTURE
Background:
Given environment has been created
And the following user details:
| user_alias | username | public_key |
| userA | envoy | test |
Scenario: IPA - Login via SSH Key authentication succeeds
Given user "userA" is authorised to access environment vms
When user "userA" starts ssh session in host "env"
Then I should be able to echo "hello world"
CUCUMBER FOR ACCEPTANCE TESTING
Cloud broker APIs

58. TESTING PROGRAMMABLE INFRASTRUCTURE
Background:
Given environment has been created
And the following user details:
| user_alias | username | public_key |
| userA | envoy | test |
Scenario: IPA - Login via SSH Key authentication succeeds
Given user "userA" is authorised to access environment vms
When user "userA" starts ssh session in host "env"
Then I should be able to echo "hello world"
CUCUMBER FOR ACCEPTANCE TESTING
Standard Ruby

59. TESTING PROGRAMMABLE INFRASTRUCTURE
Background:
Given environment has been created
And the following user details:
| user_alias | username | public_key |
| userA | envoy | test |
Scenario: IPA - Login via SSH Key authentication succeeds
Given user "userA" is authorised to access environment vms
When user "userA" starts ssh session in host "env"
Then I should be able to echo "hello world"
CUCUMBER FOR ACCEPTANCE TESTING
RSpec assertions

60. TESTING PROGRAMMABLE INFRASTRUCTURE
UNDER THE CUCUMBER, PLAIN RUBY
Then(/^I should be able to echo "([^"]*)"$/) do |text|
cmd = "echo #{text}"
output = @session.exec!(cmd)
close_ssh(@session)
expect(output.to_s.strip).to eql(text)
end

61. THOUGHTS

62. TESTING PROGRAMMABLE INFRASTRUCTURE
THE GOOD
▸ Specialised tests for each layer
▸ Really quick, expressive
ServerSpec tests
▸ Power of a full programming
language for user tests

63. TESTING PROGRAMMABLE INFRASTRUCTURE
THE BAD
▸ Over reliance on acceptance
tests
▸ Awkward switching between
two suites
▸ Out of my comfort zone

64. TESTING PROGRAMMABLE INFRASTRUCTURE
THE UGLY
▸ Starting infrastructure is SLOW.
▸ It's expensive...

65. IT WAS WORTH IT
DESPITE ALL THAT

66. BUILDING A
KUBERNETES CLUSTER
APPLYING TDD TO INFRASTRUCTURE

67. INTERNAL DEVOPS
TRAINING COURSE

68. I LEARNED A LOT!
Credit: The Simpsons, Fox

69. TESTING PROGRAMMABLE INFRASTRUCTURE
BUILD THIS:
WITH THESE:

70. TESTING PROGRAMMABLE INFRASTRUCTURE
NOT A STRAIGHTFORWARD TASK

71. TESTING PROGRAMMABLE INFRASTRUCTURE
BUT HOW TO TEST IT?
▸ This is a dev activity
▸ Want fast feedback
▸ Complexity is mitigated by
testing!

72. TESTING PROGRAMMABLE INFRASTRUCTURE
▸ Provisions cloud infrastructure
▸ Declarative files
▸ Some support for variables
TERRAFORM

73. TESTING PROGRAMMABLE INFRASTRUCTURE
TERRAFORM COMMANDS
▸ terraform plan
▸ Tells you what will change
▸ terraform apply
▸ Applies changes
▸ terraform validate
▸ Lints terraform syntax

74. TESTING PROGRAMMABLE INFRASTRUCTURE
TERRAFORM FILE EXAMPLE
resource "aws_instance" "etcd-node" {
count = 3
ami = "ami-7abd0209" # centos
availability_zone = "eu-west-1a" # ireland
instance_type = "t2.micro"
subnet_id = ....
private_ip = ....
key_name = "${aws_key_pair.my-key.key_name}"
}

75. TESTING PROGRAMMABLE INFRASTRUCTURE
LINT WITH 'TERRAFORM VALIDATE' COMMAND
Omitting a variable:

76. TESTING PROGRAMMABLE INFRASTRUCTURE
BUT IT DOESN'T CATCH ALL PROBLEMS
Duplicate subnet CIDRS:

77. TESTING PROGRAMMABLE INFRASTRUCTURE
LINTING ISN'T ENOUGH
▸ Devs don’t just rely on compilers
▸ We need something more
powerful
Credit: Nick Cave, "Soundsuit"

78. UNIT TESTING

79. TESTING PROGRAMMABLE INFRASTRUCTURE
TERRAFORM_VALIDATE
▸ Python based unit testing
▸ NOT to be confused with 'validate' command
▸ Builds map of resources & properties
▸ Totally offline
▸ New and incomplete
https://github.com/elmundio87/terraform_validate

80. TESTING PROGRAMMABLE INFRASTRUCTURE
TERRAFORM_VALIDATE FORK
OC has forked the terraform validate repo
https://github.com/opencredo/terraform_validate

81. INTEGRATION
TESTING

82. TESTING PROGRAMMABLE INFRASTRUCTURE
GOSS
▸ Easy to get up and running
▸ Doesn’t support remote
# example usage: ./goss-test.sh 34.248.91.167
TARGET='centos@'$1
SSH_KEY_PATH=~/.ssh/aws
ssh -t -t -i $SSH_KEY_PATH $TARGET 'curl -fsSL https://goss.rocks/install | sudo sh'
scp ./goss.json $TARGET:~/goss.yaml
ssh -t -t -i $SSH_KEY_PATH $TARGET 'goss validate'
https://gist.github.com/burythehammer/081d6ee11cc33c2f4c4729ae67622f5b

83. TESTING PROGRAMMABLE INFRASTRUCTURE
▸ Terraform compatibility
▸ Already a talk about this
▸ “Untangling Infrastructure Code” by
Nell Shamrell-Harrington
TEST KITCHEN + INSPEC

84. TESTING PROGRAMMABLE INFRASTRUCTURE
TestCreate Config Destroy
]TEST KITCHEN MANAGES YOUR TEST LIFECYCLE

85. TESTING PROGRAMMABLE INFRASTRUCTURE
TestCreate Config Destroy
TEST KITCHEN DOESN'T SUPPORT MULTIPLE PROVISIONERS

86. TESTING PROGRAMMABLE INFRASTRUCTURE
TEST KITCHEN DOESN'T SUPPORT MULTIPLE PROVISIONERS
https://github.com/test-kitchen/test-kitchen/issues/329

87. TESTING PROGRAMMABLE INFRASTRUCTURE
TERRAFORM 'NULL RESOURCE'
resource "null_resource" "ansible" {
triggers {
instance_ids = "${join(",", aws_instance.etcd-node.*.id)}"
}
provisioner "local-exec" {
command = "sleep 20 && cd ../ansible/ && ansible-playbook etcd.yaml"
}
}

88. TESTING PROGRAMMABLE INFRASTRUCTURE
TestCreate
Config
Destroy

89. THOUGHTS

90. TESTING PROGRAMMABLE INFRASTRUCTURE
THE GOOD
▸ Tooling exists!
▸ You can totally get a test
suite working
Credit: Overwatch, Blizzard Entertainment

91. TESTING PROGRAMMABLE INFRASTRUCTURE
THE BAD
▸ Unit testing extremely immature
▸ Tools immature in general
Credit: Overwatch, Blizzard Entertainment

92. TESTING PROGRAMMABLE INFRASTRUCTURE
THE HACKY
▸ Be prepared to hack
▸ It might not even be possible
Credit: Overwatch, Blizzard Entertainment

93. THIS IS BRAND
NEW GROUND
REMEMBER:

94. TESTING TOOLS
DEPEND ON YOUR
STACK

95. CONCLUSIONS

96. TESTING IS
IMPORTANT
BUT OFTEN IGNORED

97. TESTERS AND OPS
SHOULD WORK TOGETHER
WE NEED TO GET OUT OF
OUR COMFORT ZONES

98. TOOLS EXIST
BUT BE PREPARED
TO HACK

99. FINALLY...

100. TESTING PROGRAMMABLE INFRASTRUCTURE
THE APPLICATION OF METHODS AND TOOLING
FROM SOFTWARE DEVELOPMENT TO
MANAGEMENT OF IT INFRASTRUCTURE
PROGRAMMABLE INFRASTRUCTURE IS..

101. TESTING IS A SOFTWARE
DEVELOPMENT METHOD
WE SHOULD APPLY IT TO
INFRASTRUCTURE

102. THANKS
QUESTIONS?
@burythehammer
matt.long@opencredo.com