Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to Use Telegraf and Its Plugin Ecosystem

Telegraf is the open source server agent which is used to collect metrics from your stacks, sensors and systems. It is InfluxDB’s native data collector that supports over 250+ inputs and outputs. Learn how to send data from a variety of systems, apps, databases and services in the appropriate format to InfluxDB. Discover tips and tricks on how to write your own plugins.

Join this webinar as Jessica Ingrassellino and Samantha Wang dive into:

Types of Telegraf plugins (i.e. input, output, aggregator and processor)
Specific plugins including Execd input plugins and the Starlark processor plugin
How to create your own Telegraf plugin

  • Be the first to comment

  • Be the first to like this

How to Use Telegraf and Its Plugin Ecosystem

  1. 1. Jessica Ingrassellino Engineering Manager, InfluxData Samantha Wang Product Manager, InfluxData December 2020 How to Use Telegraf and Its Plugin Ecosystem
  2. 2. © 2020 InfluxData. All rights reserved.2 © 2020 InfluxData. All rights reserved.2 – What is Telegraf – Plugin Ecosystem – Configuration Basics – Extending the Ecosystem Agenda
  3. 3. © 2020 InfluxData. All rights reserved.3 InfluxDB & Telegraf
  4. 4. © 2020 InfluxData. All rights reserved.4 © 2020 InfluxData. All rights reserved.4 InfluxData Platform Architecture
  5. 5. © 2020 InfluxData. All rights reserved.5 © 2020 InfluxData. All rights reserved.5 InfluxData products and offerings * Availability TBD
  6. 6. © 2020 InfluxData. All rights reserved.6 © 2020 InfluxData. All rights reserved.6 InfluxDB 2.0 – Changing the Game
  7. 7. © 2020 InfluxData. All rights reserved.7 © 2020 InfluxData. All rights reserved.7 Current Data Acquisition - All Methods Agent-based Push (aka Telegraf) Agentless Pull (aka Scrapers) Client Libraries Native/Ecosystem ● 200+ Telegraf plugins ● Regular cadence of releases ● Why use it? ○ No code ○ Large community ○ Lightweight but powerful ○ Customizable ● Prometheus scraper (OSS only) ● Flux prometheus.from ● Flux csv.from(url) ● Why use them? ○ Get data in quickly ○ Doesn’t require agent downloads on monitoring device ● 7 Libraries: Python, C#, Java, GO, Javascript/Node.js, Ruby, PHP ● Handles batching, chunking, setting right headers, etc. ● Why use them? ○ Easy way to get started ○ Need libraries when building custom applications ● Source system speaks line protocol ● Examples: JMeter, NiFi, Vector, FluentD ● Influx CLI CSV Import ● Why use them? ○ Know what you want to monitor, quick and easy integration
  8. 8. © 2020 InfluxData. All rights reserved.8 © 2020 InfluxData. All rights reserved.8 Telegraf: Agent for Collecting Metrics & Events Plugin-driven server agent for collecting and reporting metrics • Written in Go • Single Binary, No external dependencies • Minimal memory footprint • Optimized for writing to InfluxDB • Optimized for streaming data Input plugins to source a variety of metrics • directly from the system it’s running on • pull metrics from third party APIs • listen for metrics via a statsd and Kafka consumer services
  9. 9. © 2020 InfluxData. All rights reserved.9 © 2020 InfluxData. All rights reserved.9 Telegraf: Agent for Collecting Metrics & Events Output plugins to send metrics to a variety of other datastores, services, and message queues • InfluxDB, Graphite, OpenTSDB, Datadog, and many others • Writes to multiple outputs at the same time Metrics routing and conversion Aggregation stage for InfluxDB
  10. 10. © 2020 InfluxData. All rights reserved.10 © 2020 InfluxData. All rights reserved.10 Telegraf Agent – Core of Telegraf functionality providing the following – Robust scheduler – Adjustments for clock-drift – Adjustments for job scheduling issues that may occur – In-memory metric buffers – Metric tracking with flow back-pressure in plugins like Kafka – Full-streaming support – Metric routing: name & field pass & drop – Flexible parsing, formatting, serializing
  11. 11. © 2020 InfluxData. All rights reserved.11 Plugins! Plugins! Plugins!
  12. 12. © 2020 InfluxData. All rights reserved.12 © 2020 InfluxData. All rights reserved.12 Telegraf Plugin Ecosystem Input Plugins ● collect metrics from system, services, or third party APIs Output Plugins ● write data to various destinations Processors ● transform, decorate, and/or filter metrics Aggregators ● create aggregate metrics (e.g. mean, min, max, quantiles, etc.)
  13. 13. © 2020 InfluxData. All rights reserved.13 © 2020 InfluxData. All rights reserved.13 Telegraf Plugin Types 170+ 40+ 15+ 5+ Input Output Processors Aggregators
  14. 14. © 2020 InfluxData. All rights reserved.14 © 2020 InfluxData. All rights reserved.14 Input Plugins activemq aerospike amqp_consumer apache apcupsd aurora azure_storage_que ue bcache beanstalkd bind bond burrow cassandra ceph cgroup chrony cisco_telemetry_mdt clickhouse cloud_pubsub cloud_pubsub_push cloudwatch conntrack consul couchbase couchdb cpu dcos disk diskio disque dmcache dns_query docker docker_log dovecot ecs elasticsearch ethtool eventhub_consumer exec execd fail2ban fibaro file filecount filestat fireboard fluentd github gnmi graylog haproxy hddtemp http http_listener_v2 http_response httpjson icinga2 infiniband influxdb
  15. 15. © 2020 InfluxData. All rights reserved.15 © 2020 InfluxData. All rights reserved.15 Input Plugins (2) influxdb_listener influxdb_v2_listener intel_rdt internal interrupts ipmi_sensor ipset iptables ipvs jenkins jolokia jolokia2 jti_openconfig_telemetr y kafka_consumer kafka_consumer_legacy kapacitor kernel kernel_vmstat kibana kinesis_consumer kube_inventory kubernetes lanz leofs linux_sysctl_fs logparser logstash lustre2 mailchimp marklogic mcrouter mem memcached mesos minecraft modbus mongodb monit mqtt_consumer multifile mysql nats nats_consumer neptune_apex net net_response nginx nginx_plus nginx_plus_api nginx_sts nginx_upstream_check nginx_vts nsd nsq nsq_consumer nstat ntpq nvidia_smi opcua openldap
  16. 16. © 2020 InfluxData. All rights reserved.16 © 2020 InfluxData. All rights reserved.16 Input Plugins (3) openntpd opensmtpd openweathermap passenger pf pgbouncer phpfpm ping postfix postgresql postgresql_extensib le powerdns powerdns_recursor processes procstat prometheus proxmox puppetagent rabbitmq raindrops ras redfish redis rethinkdb riak salesforce sensors sflow smart snmp snmp_legacy snmp_trap socket_listener solr sqlserver stackdriver statsd suricata swap synproxy syslog sysstat system systemd_units tail tcp_listener teamspeak temp tengine tomcat trig twemproxy udp_listener unbound uwsgi varnish vsphere webhooks win_eventlog win_perf_counte rs win_services wireguard wireless x509_cert zfs zipkin zookeeper
  17. 17. © 2020 InfluxData. All rights reserved.17 © 2020 InfluxData. All rights reserved.17 Processor Plugins clone converter date dedup defaults enum execd filepath ifname override parser pivot port_name printer regex rename reverse_dns s2geo starlark strings tag_limit template topk unpivot
  18. 18. © 2020 InfluxData. All rights reserved.18 © 2020 InfluxData. All rights reserved.18 Aggregator Plugins basicstats final histogram merge minmax valuecounter
  19. 19. © 2020 InfluxData. All rights reserved.19 © 2020 InfluxData. All rights reserved.19 Output Plugins amon amqp application_insights azure_monitor cloud_pubsub cloudwatch cratedb datadog discard dynatrace elasticsearch exec execd file graphite graylog health http influxdb influxdb_v2 instrumental kafka kinesis librato logzio mqtt nats newrelic nsq opentsdb prometheus_client riemann riemann_legacy socket_writer stackdriver sumologic syslog timestream warp10 wavefront yandex_cloud_monito ring
  20. 20. © 2020 InfluxData. All rights reserved.20 © 2020 InfluxData. All rights reserved.20 Telegraf CPU Mem Disk Docker Kubernetes /metrics Kafka MySQL Process - transform - decorate - filter Aggregate - mean - min,max - count - variance - stddev InfluxDB File Kafka CloudWatch CloudWatch Input Output One Telegraf, Multiple Plugins
  21. 21. © 2020 InfluxData. All rights reserved.21 © 2020 InfluxData. All rights reserved.21 Can’t find the plugin you need? Telegraf is 100% open source with a strong community of contributors It’s easy to write your own Telegraf plugin! 1) Follow the contribution guide for Go 2) Write your plugin in any language and run it externally with ExecD
  22. 22. © 2020 InfluxData. All rights reserved.22 © 2020 InfluxData. All rights reserved.22 Popular Plugins Out-of-the-box Custom Kubernetes (kubelet) HTTP/socket listener Kube_inventory (apiserver) HTTP (formatted endpoints) Kafka (consumer) Prometheus (/metrics) SNMP Exec/Execd AMQP (mq metadata) StatsD Redis Nginx HAproxy Jolokia2
  23. 23. © 2020 InfluxData. All rights reserved.23 © 2020 InfluxData. All rights reserved.23 Parsing JSON CSV Graphite CollectD Dropwizard Form URL- encoded Grok
  24. 24. © 2020 InfluxData. All rights reserved.24 Configuration Basics
  25. 25. © 2020 InfluxData. All rights reserved.25 © 2020 InfluxData. All rights reserved.25 Telegraf Configuration Snippets Include many plugins per Telegraf - multiple inputs, outputs, processors, aggregators
  26. 26. © 2020 InfluxData. All rights reserved.26 © 2020 InfluxData. All rights reserved.26 The Telegraf config file needs to be specified for Telegraf agent to operate properly. It contains setup for the agent, global tags, and enabled outputs (through commenting out or removing unnecessary lines) What’s in a configuration file?
  27. 27. © 2020 InfluxData. All rights reserved.27 © 2020 InfluxData. All rights reserved.27 Set and Test Your Configuration $ telegraf --input-filter cpu:system --output-filter influxdb --processor-filter starlark --aggregator-filter valuecounter config > telegraf.conf WRITE DESIRED CONFIG FILE RUN CONFIG FILE TEST $ telegraf --config telegraf.conf --test VERIFY YOUR CONFIG WORKS
  28. 28. © 2020 InfluxData. All rights reserved.28 Extending the Ecosystem
  29. 29. © 2020 InfluxData. All rights reserved.29 © 2020 InfluxData. All rights reserved.29 ● Work with the open-source community or submit upgrades or enhancements for existing plugins ● Use ExecD to write a plugin in Go or a language of your choice You can extend Telegraf by:
  30. 30. © 2020 InfluxData. All rights reserved.30 © 2020 InfluxData. All rights reserved.30 ● Plugin runs in its own process ● Requires line protocol ● Avoid the need for review by Telegraf team ● Supports the same API as an internal plugin ● Can use for non-GO plugins ● Can use for licensed software plugins ● Can use for any type of plugin (input, output, processor, aggregator) External plugins via ExecD
  31. 31. © 2020 InfluxData. All rights reserved.31 © 2020 InfluxData. All rights reserved.31 Execd Plugin Arch.
  32. 32. © 2020 InfluxData. All rights reserved.32 © 2020 InfluxData. All rights reserved.32 [[processors.execd]] command = ["python", "./processors/forecasting.py"] Sample ExecD Config: Input, Output, Processor
  33. 33. © 2020 InfluxData. All rights reserved.33 © 2020 InfluxData. All rights reserved.33 ExecD: To Shim, or not to Shim shim Use this when building an external plugin using Go no shim Use this when building a plugin in other languages (Python, Ruby, etc)
  34. 34. © 2020 InfluxData. All rights reserved.34 © 2020 InfluxData. All rights reserved.34 Customer Quotes “Telegraf is like a swiss army knife for connecting various MQTT sources and OPC UA sources.” — Fr. Ant. Niedermayr “Our next-generation pipeline takes advantage of Kafka and the Telegraf streaming service to create a more robust data topology. Essentially this allows us to explicitly implement the four R’s: routability, retention, resilience, and redundancy.” — Wayfair
  35. 35. © 2020 InfluxData. All rights reserved.35 © 2020 InfluxData. All rights reserved.35 References Getting started with Telegraf Telegraf plugins Telegraf GitHub Page External Plugins Guide
  36. 36. Thank You
  37. 37. © 2020 InfluxData. All rights reserved.37 © 2020 InfluxData. All rights reserved.37 Input Plugins activemq aerospike amqp_consumer apache apcupsd aurora azure_storage_queu e bcache beanstalkd bind bond burrow cassandra ceph cgroup chrony cisco_telemetry_mdt clickhouse cloud_pubsub cloud_pubsub_push cloudwatch conntrack consul couchbase couchdb cpu dcos disk diskio disque dmcache dns_query docker docker_log dovecot ecs elasticsearch ethtool eventhub_consumer exec execd fail2ban fibaro file filecount filestat fireboard fluentd github gnmi graylog haproxy hddtemp http http_listener_v2 http_response httpjson icinga2 infiniband influxdb
  38. 38. © 2020 InfluxData. All rights reserved.38 © 2020 InfluxData. All rights reserved.38 Input Plugins (2) influxdb_listener influxdb_v2_listener intel_rdt internal interrupts ipmi_sensor ipset iptables ipvs jenkins jolokia jolokia2 jti_openconfig_telemetry kafka_consumer kafka_consumer_legacy kapacitor kernel kernel_vmstat kibana kinesis_consumer kube_inventory kubernetes lanz leofs linux_sysctl_fs logparser logstash lustre2 mailchimp marklogic mcrouter mem memcached mesos minecraft modbus mongodb monit mqtt_consumer multifile mysql nats nats_consumer neptune_apex net net_response nginx nginx_plus nginx_plus_api nginx_sts nginx_upstream_check nginx_vts nsd nsq nsq_consumer nstat ntpq nvidia_smi opcua openldap
  39. 39. © 2020 InfluxData. All rights reserved.39 © 2020 InfluxData. All rights reserved.39 Input Plugins (3) openntpd opensmtpd openweathermap passenger pf pgbouncer phpfpm ping postfix postgresql postgresql_extensibl e powerdns powerdns_recursor processes procstat prometheus proxmox puppetagent rabbitmq raindrops ras redfish redis rethinkdb riak salesforce sensors sflow smart snmp snmp_legacy snmp_trap socket_listener solr sqlserver stackdriver statsd suricata swap synproxy syslog sysstat system systemd_units tail tcp_listener teamspeak temp tengine tomcat trig twemproxy udp_listener unbound uwsgi varnish vsphere webhooks win_eventlog win_perf_counte rs win_services wireguard wireless x509_cert zfs zipkin zookeeper
  40. 40. © 2020 InfluxData. All rights reserved.40 © 2020 InfluxData. All rights reserved.40 Processor Plugins clone converter date dedup defaults enum execd filepath ifname override parser pivot port_name printer regex rename reverse_dns s2geo starlark strings tag_limit template topk unpivot
  41. 41. © 2020 InfluxData. All rights reserved.41 © 2020 InfluxData. All rights reserved.41 Aggregator Plugins basicstats final histogram merge minmax valuecounter
  42. 42. © 2020 InfluxData. All rights reserved.42 © 2020 InfluxData. All rights reserved.42 Output Plugins amon amqp application_insights azure_monitor cloud_pubsub cloudwatch cratedb datadog discard dynatrace elasticsearch exec execd file graphite graylog health http influxdb influxdb_v2 instrumental kafka kinesis librato logzio mqtt nats newrelic nsq opentsdb prometheus_client riemann riemann_legacy socket_writer stackdriver sumologic syslog timestream warp10 wavefront yandex_cloud_monitor ing

×