Slides from my talk at DevOps Singapore meetup in March 2019

1. Terraform modules and some of best-
practices
Anton Babenko
@antonbabenko
March 2019

2. Anton Babenko
Terraform AWS fanatic since 2015
Organiser of HashiCorp UG, AWS UG, DevOps Norway, DevOpsDays Oslo
I 💚 open-source:
terraform-community-modules + terraform-aws-modules
antonbabenko/pre-commit-terraform — clean code and documentation
antonbabenko/terraform-docs-as-pdf
antonbabenko/modules.tf-lambda — generate Terraform code from visual diagrams
www.terraform-best-practices.com
medium.com/@anton.babenko
@antonbabenko — Twitter, GitHub, Linkedin

3. Collection of open-source Terraform AWS modules supported by the community.
More than 2 mil. downloads since September 2017.
(VPC, Autoscaling, RDS, Security Groups, ELB, ALB, Redshift, SNS, SQS, IAM, EKS, ECS…)
github.com/terraform-aws-modules
registry.terraform.io/modules/terraform-aws-modules

4. Write, plan and manage infrastructure as code
www.terraform.io

11. Google Cloud
Deployment Manager
Azure Resource
Manager

13. +morethan100providers

14. Why Terraform and not AWS CloudFormation,
Azure ARM, Google Cloud Deployment Manager?
• Terraform manages 100+ providers, has easier syntax (HCL), has native
support for modules and remote states, has teamwork related features, is an
open-source project.
• Provides a high-level abstraction of infrastructure (IaC)
• Allows for composition and combination
• Supports parallel management of resources (graph, fast)
• Separates planning from execution (dry-run)

15. Terraform — universal tool for everything with an API
GSuite
Dropbox files and access
New Relic metrics
Datadog users and metrics
Jira issues
All Terraform providers

16. Let’s start!

17. Terraform modules

18. Modules in Terraform are self-contained
packages of Terraform configurations that are
managed as a group.

19. Resource modules
Create resources in a very flexible configuration
Open-source

20. Resource modules

21. Resource modules

22. Resource modules

23. Resource modules

24. Q: Why use resource modules instead of resources?
A: Resources can’t be versioned, but modules can.
Resource modules

26. Infrastructure modules
Consist of resource modules
Enforce tags and company standards
Use preprocessors, jsonnet, cookiecutter

27. Infrastructure modules

28. Infrastructure modules

29. Infrastructure modules

31. Types of Terraform modules
Resource modules (github.com/terraform-aws-modules , for eg)
Infrastructure modules

32. - [ ] How to write modules?
- [ ] How to call modules?
- [ ] How to work with the code?

33. Tip №0
Check Terraform Registry before writing resource modules

34. Hide implementation details

37. Size

38. Size
https://github.com/mbtproject/mbt

39. Things to avoid in modules

40. Exception: logical providers (template, random, local, http, external)
Providers in modules — evil

42. Provisioner — evil
Avoid provisioner in all resources

43. Provisioner — evil
Avoid provisioner in all resources

44. Provisioner — evil
Avoid provisioner even in EC2 resources

45. Provisioner — evil
Avoid provisioner even in EC2 resources

48. null_resource provisioner — good

49. Traits of good Terraform modules
Documentation and examples
Feature rich
Sane defaults
Clean code
Tests
Read more: http://bit.ly/common-traits-in-terraform-modules

50. - [x] How to write modules?
- [x] Do not write, if you can
- [x] Avoid providers in modules, provisioners
- [ ] How to call modules?
- [ ] How to work with the code?

51. Call Terraform modules
Amount of resources and code keeps growing
How to organize and call?
How to orchestrate calls?

52. All-in-one
Good:
Declare variables and outputs in
fewer places
Bad:
Large blast radius
Everything is blocked at once
Impossible to specify dependencies
between modules (depends_on)

53. 1-in-1
Good:
Smaller blast radius
Possible to join invocation
Easier and faster to work with
Bad:
Declare variables and outputs
in more places

54. Which way do you group your code?
All-in-one or 1-in-1?

55. Correct
MFA (Most Frequent Answer):
Somewhere in between

56. What kind of orchestration tool do you use?
-target
Makefile
…

57. Orchestration in Terraform

58. No really, do not try this at home!

59. Orchestration = Terragrunt
https://github.com/gruntwork-io/terragrunt/

60. Orchestration = Terragrunt

61. Orchestration = Terragrunt

62. Orchestration = Terragrunt

63. Orchestration = Terragrunt

64. tfvars can’t contain dynamic values :(
Orchestration = Terragrunt

65. Orchestration = Terragrunt
tfvars can’t contain dynamic values, so I
have fixed it :)

66. before_hook + shell script
https://github.com/antonbabenko/modules.tf-lambda/blob/master/templates/
terragrunt-common-layer/template/common/scripts/
update_dynamic_values_in_tfvars.sh
or use modules.tf

67. - [x] How to write modules?
- [x] How to call modules?
- [x] 1-in-1 works beter over time
- [x] Orchestration = Terragrunt
- [x] Dynamic values in tfvars
- [ ] How to work with the code?

68. Add new features
Usually it is easy…

69. Create new or use existing

70. Create new or use existing

71. Create new or use existing

72. Create new or use existing

73. Work with lists

74. Work with lists

75. Work with lists

76. https://jsonnet.org/
Work with stateful lists

77. Work with stateful lists

78. Work with stateful lists

79. Work with stateful lists

80. Work with stateful lists

81. Integration

82. Integration

83. Auto-integration

84. Edge cases
Different AWS regions (version of S3 signature, EC2 ClassicLink,
IPv6)
Date of creation of AWS account
Limits on resources in AWS
Services and features availability

85. Avoid in Terraform
Not secret arguments should not be specified as command line
arguments => put them in tfvars
Reduce usage of "-target" and "-parallelism"
"Terraform workspaces" evil in=> separate by directories
Dependency hell in modules

86. - [x] How to write modules?
- [x] How to call modules?
- [x] How to work with the code?
- [x] Lists in Terraform 0.11 can be painful
- [x] Perceive Terraform easier

87. - [x] How to write modules?
- [x] How to call modules?
- [x] How to work with the code?
- [x] Terraform 0.12 beta!

88. Terraform 0.12
HCL2 — simplified syntax
Loops ("for")
Dynamic blocks ("for_each")
Correct conditional operators (… ? … : …)
Extended types of variables
Templates in values
Links between resources are supported (depends_on everywhere)
Read more — https://www.hashicorp.com/blog/announcing-terraform-0-1-2-beta

89. Summary
Write less and simpler (Terraform 0.12 won’t fix your code for you!)
Use existing modules and utilities

90. How to handle secrets in Terraform?
• Can you accept secrets to be saved in state file in plaintext? Probably not.
• AWS IAM password & access secret keys — use PGP as keybase.io
• AWS RDS — set dummy password and change after DB is created
• AWS RDS — use iam_database_authentication_enabled = true
• EC2 instance user-data + AWS KMS
• EC2 instance user-data + AWS System Manager’s Parameter Store
• AWS Secrets Manager
• https://github.com/opencredo/terrahelp
• Other options:
• Secure remote state location (S3 bucket policy, KMS key)

91. What are the tools/solutions out there?
• Terraform Registry — collection of public Terraform modules for common
infrastructure configurations for any provider — https://registry.terraform.io/
• Terraform linter to detect errors that can not be detected by `terraform plan`
— https://github.com/wata727/tflint
• Terraform version manager — https://github.com/kamatama41/tfenv
• A web dashboard to inspect Terraform States — https://github.com/
camptocamp/terraboard
• Jsonnet — The data templating language — http://jsonnet.org
• terraform-compliance - BDD style Terraform validation/compliancy check —
https://github.com/eerkunt/terraform-compliance

92. Atlantis — Start working on Terraform as a team
A unified workflow for collaborating on Terraform through GitHub, GitLab and
Bitbucket
https://www.runatlantis.io
https://github.com/terraform-aws-modules/terraform-aws-atlantis

93. Bonus

98. ✓ cloudcraft.co — design, plan and visualize
✓ terraform-aws-modules — building blocks of AWS infrastructure
✓ Terraform — infrastructure as code

99. Infrastructure as code generator — from visual diagrams to Terraform
https://github.com/antonbabenko/modules.tf-lambda
Demo video: https://www.youtube.com/watch?v=F1Ax1zfZbiY

100. 1. Go to cloudcraft.co
2. Sign up, sign in (free account)
3. Draw your AWS infrastructure
4. Click "Export"
5. Click "Terraform code export"
Try it yourself!

101. modules.tf — generated code
✓ Potentially ready-to-use Terraform configurations
✓ Suits best for bootstrapping
✓ Enforces Terraform best-practices
✓ Batteries included (terraform-aws-modules, terragrunt, pre-commit)
✓ 100% free and open-source (https://github.com/antonbabenko/
modules.tf-lambda)
✓ Released under MIT license

102. Thanks!
Questions?
github.com/antonbabenko
twitter.com/antonbabenko