Joel Cooklin and I cover the usage and design of Snap, an open telemetry framework. I also refer to Jason Dixon a few times because I'm a fan of his presentation that influenced my view of monitoring. If you love monitoring and don't mind profanity, check this out first: https://speakerdeck.com/obfuscurity/the-state-of-open-source-monitoring

1. Snap
A story of telemetry, data, python
And references to Jason Dixon at
GrafanaCon 2016

2. Hey

3. The Snap Telemetry Framework

4. 4
what my friends think telemetry is what my parents think telemetry is
what society thinks telemetry is
what my boss thinks telemetry is what I think telemetry is what telemetry actually is

5. (Do I get bonus points for quoting Jason at GrafanaCon?)
“Telemetry refers to the collection of
measurements, typically of remote
instruments, for the purposes of
monitoring.”
~ Jason Dixon in The Graphite Book

6. Telemetry Framework ™

7. Client libraries for:
● Prometheus
● Sensu
● Statsd
● & more
● beats
● tcollector
● scollector
● Nagios
● Snap
● collectd
● telegraf
● diamond

8. Requirements
Composable Declarative Extensible

9. Composable
Monitoring
Snap

10. Composable
Monitoring
Snap

11. Declarative
Explicit definition of
- Interval
- Collected metrics
- Published endpoints
Allows for
- Collect from multiple sensors
- Publish to multiple endpoints
- Multiple workflows at different intervals
- Metrics available upon plugin update
Let’s look at a Task Manifest
“I’ll make you a workflow you can’t refuse”

12. Extensible
Collect Process Publish

13. Extensible
Available in the Plugin Catalog
Collectors written:
● Nginx
● Couchbase
● MongoDB
● Netstat
● Procstat
Publishers written:
● Blueflood
● Couchbase

14. Extensible
Available in the Plugin Catalog
Collectors written:
● Nginx
● Couchbase
● MongoDB
● Netstat
● Procstat
Publishers written:
● Blueflood
● Couchbase

15. Extensible
Available in the Plugin Catalog
Collectors written:
● Nginx
● Couchbase
● MongoDB
● Netstat
● Procstat
Publishers written:
● Blueflood
● Couchbase

16. Extensible
Available in the Plugin Catalog
Collectors written:
● Nginx
● Couchbase
● MongoDB
● Netstat
● Procstat
Publishers written:
● Blueflood
● Couchbase
Grafana Labs

17. The Anatomy of a Snap Plugin

18. Plugin Authoring - Plugin Types
Collectors Processors Publishers

19. Plugin Authoring - Lifecycle
Loading a plugin

20. Plugin Authoring - Lifecycle
Loading a plugin
1. Snap starts the plugin

21. Plugin Authoring - Lifecycle
Loading a plugin
1. Snap starts the plugin
2. Snap negotiates with the
plugin over stdout (handshake)

22. Plugin Authoring - Lifecycle
Loading a plugin
1. Snap starts the plugin
2. Snap negotiates with the plugin over stdout (handshake)
3. Snap calls the plugin
a. To get its ConfigPolicy

23. Plugin Authoring - Lifecycle
Loading a plugin
1. Snap starts the plugin
2. Snap negotiates with the plugin over stdout (handshake)
3. Snap calls the plugin
a. To get its config policy
b. To get what metrics it
collects

24. Plugin Authoring - Lifecycle
Loading a plugin
1. Snap starts the plugin
2. Snap negotiates with the plugin over stdout (handshake)
3. Snap calls the plugin
a. To get its config policy
b. To get what metrics it exposes
4. Snap stops the plugin

25. Plugin Authoring - Lifecycle
Starting a task

26. Plugin Authoring - Lifecycle
Starting a task
1. Snap ensures plugins required by the task are running

27. Plugin Authoring - Lifecycle
Starting a task
1. Snap ensures plugins required by the task are running
a. Plugins that are used have their
subscriptions increased

28. Starting a task
1. Snap ensures needed plugins required by the task are running
a. Plugins that are used have their subscriptions increased
Plugin subscriptions drive the logic
related to plugin routing and concurrency
Plugin Authoring - Lifecycle

29. Plugin Authoring - Lifecycle
Stopping a task

30. Plugin Authoring - Lifecycle
Stopping a task
Causes plugins to stop
When the plugin’s subscription count falls below zero

31. Plugin Authoring - Meta
Plugin Meta

32. Plugin Authoring - Meta
Plugin Meta
Is communicated over STDOUT between
the plugin and Snap when a plugin starts

33. Plugin Authoring - Meta
Plugin Meta
Is communicated over STDOUT between the plugin and Snap when a
plugin starts
{"Meta":{"Type":2,"Name":"graphite","Version":5,"RPCType":2,"
RPCVersion":1,"ConcurrencyCount":5,"Exclusive":false,"Cach
eTTL":0,"RoutingStrategy":0},"ListenAddress":"127.0.0.1:37166","
PprofAddress":"0"}

34. Plugin Authoring - Meta
Plugin Meta
Is communicated over STDOUT between the plugin and Snap when a plugin starts
● Name - Plugin name as it will appear in the plugin
catalog

35. Plugin Authoring - Meta
Plugin Meta
Is communicated over STDOUT between the plugin and Snap when a plugin starts
● Name - name as it will appear in the plugin catalog
● Type - collector, process, publisher

36. Plugin Authoring - Meta
Plugin Meta
Is communicated over STDOUT between the plugin and Snap when a plugin starts
● Name - name as it will appear in the plugin catalog
● Type - collector, process, publisher
● Version - version of the plugin

37. Plugin Authoring - Meta
Plugin Meta
Is communicated over STDOUT between the plugin and Snap when a plugin starts
● Name - name as it will appear in the plugin catalog
● Type - collector, process, publisher
● Version - version of the plugin
● RPCType - RPC mechanism to be used

38. Plugin Authoring - Meta
Plugin Meta
Is communicated over STDOUT between the plugin and Snap when a plugin starts
● Name - name as it will appear in the plugin catalog
● Type - collector, process, publisher
● Version - version of the plugin
● RPCType - RPC mechanism to be used
○ The default is ‘gRPC’
○ We will maintain legacy support for
GORPC based plugins

39. Plugin Authoring - Meta
Plugin Meta
Is communicated over STDOUT between the plugin and Snap when a plugin starts
● Name - name as it will appear in the plugin catalog
● Type - collector, process, publisher
● Version - version of the plugin
● RPCType - RPC mechanism to be used
● RPCVersion - Defines the version of the service the plugin
implements

40. Plugin Authoring - Meta
Plugin Meta
Is communicated over STDOUT between the plugin and Snap when a plugin starts
● Name - name as it will appear in the plugin catalog
● Type - collector, process, publisher
● Version - version of the plugin
● RPCType - RPC mechanism to be used
● RPCVersion - Defines the version of the service the plugin
implements
service Collector {
rpc CollectMetrics(MetricsArg) returns (MetricsReply) {}
rpc GetMetricTypes(GetMetricTypesArg) returns (MetricsReply) {}
rpc Ping(Empty) returns (ErrReply) {}
rpc Kill(KillArg) returns (ErrReply) {}
rpc GetConfigPolicy(Empty) returns (GetConfigPolicyReply) {}
}

41. Plugin Authoring - Meta
Plugin Meta
Is communicated over STDOUT between the plugin and Snap when a plugin starts
● Name - name as it will appear in the plugin catalog
● Type - collector, process, publisher
● Version - version of the plugin
● RPCType - RPC mechanism to be used
● RPCVersion - Defines the version of the service the plugin
implements
service Processor {
rpc Process(PubProcArg) returns (MetricsReply) {}
rpc Ping(Empty) returns (ErrReply) {}
rpc Kill(KillArg) returns (ErrReply) {}
rpc GetConfigPolicy(Empty) returns (GetConfigPolicyReply) {}
}

42. Plugin Authoring - Meta
Plugin Meta
Is communicated over STDOUT between the plugin and Snap when a plugin starts
● Name - name as it will appear in the plugin catalog
● Type - collector, process, publisher
● Version - version of the plugin
● RPCType - RPC mechanism to be used
● RPCVersion - Defines the version of the service the plugin
implements
service Publisher {
rpc Publish(PubProcArg) returns (ErrReply) {}
rpc Ping(Empty) returns (ErrReply) {}
rpc Kill(KillArg) returns (ErrReply) {}
rpc GetConfigPolicy(Empty) returns (GetConfigPolicyReply) {}
}

43. Plugin Authoring - Meta
Plugin Meta
Is communicated over STDOUT between the plugin and Snap when a plugin starts
● Name - name as it will appear in the plugin catalog
● Type - collector, process, publisher
● Version - version of the plugin
● RPCType - RPC mechanism to be used
● RPCVersion - Defines the version of the service the plugin implements
● RoutingStrategy - Least Recently Used
Sticky (task)
Config based

44. Plugin Authoring - Meta
Plugin Meta
Is communicated over STDOUT between the plugin and Snap when a plugin starts
● Name - name as it will appear in the plugin catalog
● Type - collector, process, publisher
● Version - version of the plugin
● RPCType - RPC mechanism to be used
● RPCVersion - Defines the version of the service the plugin implements
● RoutingStrategy - Least recently used, Sticky (task) or Config based
● ConcurrencyCount - Defines the maximum number of
subscriptions the plugin can have before
starting another instance of the plugin

45. Plugin Authoring - Meta
Plugin Meta
Is communicated over STDOUT between the plugin and Snap when a plugin starts
● Name - name as it will appear in the plugin catalog
● Type - collector, process, publisher
● Version - version of the plugin
● RPCType - RPC mechanism to be used
● RPCVersion - Defines the version of the service the plugin implements
● RoutingStrategy - Least recently used, Sticky (task) or Config based
● ConcurrencyCount - Defines the maximum number of subscriptions the plugin
can have before starting another instance of the plugin
● Exclusive - Results in a single instance of the plugin
running for any number of subscriptions
(ConcurrencyCount is ignored)

46. Plugin Authoring - Meta
Plugin Meta
Is communicated over STDOUT between the plugin and Snap when a plugin starts
● Name - name as it will appear in the plugin catalog
● Type - collector, process, publisher
● Version - version of the plugin
● RPCType - RPC mechanism to be used
● RPCVersion - Defines the version of the service the plugin implements
● RoutingStrategy - Least recently used, Sticky (task) or Config based
● ConcurrencyCount - Defines the maximum number of subscriptions the plugin
can have before starting another instance of the plugin
● Exclusive - Results in a single instance of the plugin running for any
number of subscriptions
● CacheTTL - Overrides the default CacheTTL of 500ms

47. Plugin Authoring - Meta
Plugin Meta
Is communicated over STDOUT between the plugin and Snap when a plugin starts
● Name - name as it will appear in the plugin catalog
● Type - collector, process, publisher
● Version - version of the plugin
● RPCType - RPC mechanism to be used
● RPCVersion - Defines the version of the service the plugin implements
● RoutingStrategy - Least recently used, Sticky (task) or Config based
● ConcurrencyCount - Defines the maximum number of subscriptions the plugin
can have before starting another instance of the plugin
● Exclusive - Results in a single instance of the plugin running for any
number of subscriptions
● CacheTTL - Overrides the default CacheTTL of 500ms
If the framework attempts to call a plugin for a
metric before it’s TTL has expired it will be
retrieved from the cache instead

48. Plugin Authoring - Plugin libs
Want to write a plugin?

49. Plugin Authoring - Plugin libs
Want to write a plugin?
Start with one of the Snap plugin libs

50. Plugin Authoring - Plugin libs
Want to write a plugin?
Start with one of the Snap plugin libs

51. Plugin Authoring - Plugin libs
Want to write a plugin?
Start with one of the Snap plugin libs

52. Plugin Authoring - Plugin libs
Want to write a plugin?
Start with one of the Snap plugin libs

53. Plugin Authoring
Leveraging existing investments
● Using the plugin-libs wrap existing tools
○ Example: snap-plugin-collector-diamond

54. The Snap Telemetry Framework
To Review

55. The Snap Telemetry Framework
Snap is an Open Source telemetry framework
that allows you to collect, process and publish
measurements.
Collectors Processors Publishers

56. The Snap Telemetry Framework
Snap 1.0 is a general availability of a rock solid
daemon with a collection of 67 plugins from
multiple companies.

57. The Snap Telemetry Framework
Snap 1.0 is the beginning. And we want you
involved.
snap-telemetry.io

58. Thank you!

59. Backup Notes

60. Everything is Challenging At
Scale
60

61. Add new task
61

62. Add new task
62

63. define as a
tribe
Scaling with Tribe
63

64. Scaling with Tribe
Add new task
64

65. Snap | Distributed Workflow
Physical/Virtual
Host
Scheduler
Processing
Publishing
Collection
65

66. Snap | Distributed Workflow
66
Physical/VM Host
Physical/VM Host
Physical/VM Host
Physical/VM Host
Physical/VM Host Physical/VM Host
Collection
Collection
Collection
Scheduler
Processing Publishing

67. Customizable definition of task and related
workflow:
Collect
Publish
Publish
Collect Publish ProcessCollect Publish
Collect
Process Publish
Process Publish
Snap | Overview – Example Workflows
67