In this presentation we will show how to integrate New Relic monitoring with Terraform infrastructure as code templates, setting up alerts, dashboards, and other monitoring artifacts as part of your application deployment pipeline. We will demonstrate an open source example and show how it behaves under a load as it fails.

1. Monitoring and Alerting
as Code with Terraform
and New Relic
http://bit.ly/terraform-newrelic
January 30, 2019
Richard Bullington-McGuire
Principal Architect, Modus Create
richard@moduscreate.com
@obscurerichard

3. My New Relic & Terraform Experience
2014: Used New Relic to improve performance 40x in public safety training
system - supported $MM exit - Java, Windows
2014-2017: Used New Relic to improve performance 10x in hospitality system -
saving the client’s reputation - .NET, SQL Server, Windows, JMeter load tests
2015: Took over ProductShop project - Terraform
2017: Used Terraform & New Relic to migrate 14 critical systems to AWS for a
large education company.
2017: Taught real estate information software firm how to do cloud migration
right with Terraform and New Relic.
2018: Built out devops-infra-demo Terraform / Packer / Ansible / Jenkins

4. Configuring monitoring by hand is fragile and tough
Use Infrastructure as Code techniques to make it repeatable
Load test to ensure that monitoring & alerting actually work
Work around chicken or egg problems
Monitoring and Alerting as Code

5. The old way:
Set up New Relic by hand on each server.
Tweak configs until they match.
Standing up a new environment or application?
Copy and paste and pray
Configuring monitoring by hand: just say no

6. ● Set up your application stack and monitoring & alerting with
the same toolkit
● Leverage tools such as Ansible to reduce boilerplate
● Use an image bakery pattern - consider immutable
infrastructure or a hybrid pattern.
● Consider AWS CodeDeploy for legacy migration - easier than
dockerizing everything!
● Use Jenkins or another CI process to drive the bakery
Use Infrastructure as Code for repeatability

7. ● ab is good, JMeter is better
● Use New Relic to monitor load tests and check validity
● Build a representative test resembling production peak load
● Run load tests through CI at scale or use BlazeMeter
● jmeter-ec2 or its forks can run massive load tests
○ But the code is ugly and not maintained much and you
will have to hack it
Load test to ensure that it all actually works

8. ● Terraform can only define resources for items that exist
○ So how do you define monitoring for a stack that is not
yet created?
● count to the rescue
● See this Gruntwork blog article on Terraform Tips & Tricks
Work around chicken or egg problems

9. Case Study:
Terraform & New Relic & JMeter at work
Driven by Jenkins CI

10. Case Study:
Terraform & New Relic & JMeter at work
Driven by Jenkins CI
● Education company cloud migration (4mo -> prod)
● Apps w/> 30,000 RPM at peak measured with New Relic
● Production with 80+ sizeable AMIs baseline
● Auto Scaling to 200+ AMIs under heavy load
● Multiple environments: dev, qa, staging, prod
● Terabyte-scale MySQL Aurora cluster, 50+ TB in S3
● Jenkins, Terraform, Ansible, Packer, CodeDeploy,
JMeter load tests, New Relic monitoring

11. Implementation
http://bit.ly/terraform-newrelic

12. Local Development

13. CI Environment
Jenkins
Elastic Load
Balancer
EC2 Auto Scaling
Group - Web App
Terraform Provision
Packer
Provision

14. Demo
http://bit.ly/terraform-newrelic

15. Credit where Credit is Due
● Sergio Bruder (@sdbruder) built the JMeter test harness and
CodeDeploy app used in the demo
● Don Anderson (@dranderson) integrated New Relic APM,
Infrastructure, and Dashboards for this demo
I could not have done this without their help!

16. Thank You!
http://bit.ly/terraform-newrelic
richard@moduscreate.com
@obscurerichard