Monitoring Cloud Foundry using Sensu and Graphite

1. GE Digital
Monitoring Cloud Foundry
Using Sensu and Graphite
2

2. GE Digital 4
GE Digital and Predix

3. GE Digital
Cloud Platform for
Industrial Internet of Things
5
Predix

4. GE Digital 6
Aviation

5. GE Digital 7
Transportation

6. GE Digital 8
Oil and Gas

7. GE Digital 9
Power and Water

8. GE Digital 11
Healthcare

9. GE Digital
Predix and CloudFoundry
12

10. GE Digital
It all started simply…
15
From POC to Production
Built a Development CF Environment in a few months
Leadership challenge: Deliver 4 production apps to a paying customer in 3 months
Game On! Time to operationalize CF!
Showed Developers could get apps to MVP phase quickly!

11. GE Digital
Step One: Monitor all the things!
17
Have you *looked* at all_the_CF_things?

12. 19
Cloud Controller
NATS
Runners
HM9000
Go Routers
ETCD
UAA
UAA DB
Cloud Controller DB
Doppler
Loggregator
BOSH
External Load balancers
Consul
MUCH WOW
SUCH AMAZE

13. GE Digital 21
SUCH AMAZE!
This is going to be challenging!

14. GE Digital
CloudWatch?!?
24

15. GE Digital
Time to build…
25

16. GE Digital
Monitoring and Metrics Goals
26
AutomaticUtility Service Extensible
Driven by
Configuration
Management

17. GE Digital
Build MVP Solution
27
Monitoring
Framework
Data VisualizationMetrics Collection

18. GE Digital
Sensu
28

19. GE Digital
Sensu Architecture
29
Sensu Clients
Check Execution
External data input
RabbitMQ Cluster
Message Transport
Redis
Health Check
StateSensu Servers
Sensu Servers
Publish Check Requests
Processes Events
Sensu APIs
Sensu API Servers
REST interface to
Monitoring System

20. GE Digital
Service Check Flow
31
Sensu Servers
1
Sensu Server publishes
check request to
subscriber queues
2
Sensu Clients listen to
queue ‘subscriptions’
and execute
commands
Updates check
state in Redis
3
Sensu Clients
publish check
execution
responses to
message queue
Sensu Servers
processes check
responses
4
5
Trigger handlers
if configured

21. GE Digital
Service Check Anatomy
• Command or Script to run which outputs data to
STDOUT or STDERR
• Produce exit codes to indicate state
• 0 - OK, 1 - Warning, 2 - Critical, >=3 - Custom
• Optional response payload (JSON)
• Subscribers
• Group of nodes that should execute check
• Check Interval
• Handlers
• Actions to take on event (if any)

22. GE Digital
Service Check Request
{
"checks": {
"check_disk_usage": {
"command": "check-disk-usage.rb -w :::disk.warn|80::: -c :::disk.crit|90:::",
"subscribers": [
"prod-DEA"
],
"handlers": [
"pagerduty"
],
"interval": 60
}
}
}

23. GE Digital
Take Action! - Sensu Event Handlers
• Handlers are actions executed by a Sensu server when
events are received
• Send PagerDuty Alert - Send metric to Graphite - Send to IRC
• 4 Handler Types
• Pipe - External commands that consume event data via STDIN
• TCP / UDP - Forward event data to external TCP/UDP sockets!
• Transport - Publish event data to named message queue
• Handler Sets - Bundle of handlers e.g. email, slack, pager duty

24. GE Digital
Metrics
Graphite - Grafana
36

25. GE Digital
Graphite Architecture
38
Carbon Relay
Metrics Ingest & Routing
Consistent Hashing - Replicas
Carbon Cache - WhisperDB
Metrics Storage
Graphite API
Metrics Retrieval
Grafana
Data Visualization
Dashboards
Sensu Graphite Handler
Process Metrics Events
from message queueSensu Servers
Sensu Client
Graphite Metrics
Health Checks
(REST)

26. GE Digital
Monitoring Cloud Foundry
39
Monitoring Cloud Foundry

27. GE Digital
Automatic Coverage
40
• Created BOSH release of Sensu-Client
• Sensu-Client Job included in all BOSH deployments
• Every node belongs to ‘All’ Sensu Subscription by
default
• Now capturing base Linux stats for ALL nodes
• CPU, Memory, Network, Disk stored in Graphite

28. GE Digital
Metrics Names
42
Allow easy aggregation of stats in Graphite!
uswest02-pr-cf.runner_z1.0.interface.eth0.txBytes
BOSH Deployment BOSH Job Index Metric

29. GE Digital 43

30. GE Digital
Cloud Foundry Visibility
44
• CF ‘Collector’ - Open Source project
• Listens on NATS bus for CF subsystem announcements
• Polls components /healthz and /varz endpoints
• Publishes results directly to Graphite via Carbon daemon
• Being phased out in favor of Doppler / Nozzle
implementation

31. GE Digital 46

32. GE Digital 47

33. GE Digital
Sweet Dashboard bro
48
I can haz automated pagerduty alertz pleas?!?

34. GE Digital
Sensu Graphite Data Check
49
$ curl “http://graphite-api/render?target=sumSeries(USW02-PR-
COLLECTOR.CloudController.*.*.healthy)&from=-5min&until=now&format=json&maxDataPoints=100"
Graphite API
Metrics Retrieval
Sensu Client
HTTP Health Checks
Carbon Cache - WhisperDB
Metrics Storage
{
"target": “sumSeries(USW02-PR-COLLECTOR.CloudController.*.*.healthy)",
“datapoints":
[[12.0, 1463551380], [12.0, 1463551440], [12.0, 1463551500], [12.0, 1463551560], [12.0, 1463551620]]
}

35. GE Digital
Monitoring Cloud Foundry
50
Questions?
@barrows_jeff
jeff.barrows@ge.com

36. General Electric Company reserves the right to make changes in specifications and features, or discontinue the product or service described at
any time, without notice or obligation. These materials do not constitute a representation, warranty or documentation regarding the product or
service featured. Illustrations are provided for informational purposes, and your configuration may differ. This information does not constitute legal,
financial, coding, or regulatory advice in connection with your use of the product or service. Please consult your professional advisors for any such
advice. No part of this document may be distributed, reproduced or posted without the express written permission of General Electric Company.
GE, Predix and the GE Monogram are trademarks of General Electric Company. ©2015 General Electric Company – All rights reserved.