How to Use Telegraf and Its Plugin Ecosystem

Jessica Ingrassellino
Engineering Manager, InfluxData
Samantha Wang
Product Manager, InfluxData
December 2020
How to Use Telegraf
and Its Plugin
Ecosystem

© 2020 InfluxData. All rights reserved.2 © 2020 InfluxData. All rights reserved.2
– What is Telegraf
– Plugin Ecosystem
– Configuration Basics
– Extending the Ecosystem
Agenda

© 2020 InfluxData. All rights reserved.3
InfluxDB & Telegraf

InfluxData Platform Architecture

InfluxData products and offerings
* Availability TBD

InfluxDB 2.0 – Changing the Game

Current Data Acquisition - All Methods
Agent-based Push (aka
Telegraf)
Agentless Pull (aka Scrapers)
Client Libraries
Native/Ecosystem
● 200+ Telegraf plugins
● Regular cadence of releases
● Why use it?
○ No code
○ Large community
○ Lightweight but powerful
○ Customizable
● Prometheus scraper (OSS only)
● Flux prometheus.from
● Flux csv.from(url)
● Why use them?
○ Get data in quickly
○ Doesn’t require agent downloads on
monitoring device
● 7 Libraries: Python, C#, Java, GO,
Javascript/Node.js, Ruby, PHP
● Handles batching, chunking, setting right
headers, etc.
● Why use them?
○ Easy way to get started
○ Need libraries when building custom
applications
● Source system speaks line protocol
● Examples: JMeter, NiFi, Vector,
FluentD
● Influx CLI CSV Import
● Why use them?
○ Know what you want to monitor,
quick and easy integration

Telegraf:
Agent for Collecting Metrics & Events
Plugin-driven server agent for
collecting and reporting metrics
• Written in Go
• Single Binary, No external
dependencies
• Minimal memory footprint
• Optimized for writing to InfluxDB
• Optimized for streaming data
Input plugins to source a variety of
metrics
• directly from the system it’s running
on
• pull metrics from third party APIs
• listen for metrics via a statsd and
Kafka consumer services

Telegraf:
Agent for Collecting Metrics & Events
Output plugins to send metrics to a
variety of other datastores, services,
and message queues
• InfluxDB, Graphite, OpenTSDB,
Datadog, and many others
• Writes to multiple outputs at the
same time
Metrics routing and conversion
Aggregation stage for InfluxDB

Telegraf Agent
– Core of Telegraf functionality providing the following
– Robust scheduler
– Adjustments for clock-drift
– Adjustments for job scheduling issues that may occur
– In-memory metric buffers
– Metric tracking with flow back-pressure in plugins like Kafka
– Full-streaming support
– Metric routing: name & field pass & drop
– Flexible parsing, formatting, serializing

Plugins! Plugins! Plugins!

Telegraf Plugin Ecosystem
Input Plugins
● collect metrics from system,
services, or third party APIs
Output Plugins
● write data to various destinations
Processors
● transform, decorate, and/or filter
metrics
Aggregators
● create aggregate metrics (e.g.
mean, min, max, quantiles, etc.)

Telegraf Plugin Types
170+ 40+ 15+ 5+
Input Output Processors Aggregators

Input Plugins
activemq
aerospike
amqp_consumer
apache
apcupsd
aurora
azure_storage_que
ue
bcache
beanstalkd
bind
bond
burrow
cassandra
ceph
cgroup
chrony
cisco_telemetry_mdt
clickhouse
cloud_pubsub
cloud_pubsub_push
cloudwatch
conntrack
consul
couchbase
couchdb
cpu
dcos
disk
diskio
disque
dmcache
dns_query
docker
docker_log
dovecot
ecs
elasticsearch
ethtool
eventhub_consumer
exec
execd
fail2ban
fibaro
file
filecount
filestat
fireboard
fluentd
github
gnmi
graylog
haproxy
hddtemp
http
http_listener_v2
http_response
httpjson
icinga2
infiniband
influxdb

Input Plugins (2)
influxdb_listener
influxdb_v2_listener
intel_rdt
internal
interrupts
ipmi_sensor
ipset
iptables
ipvs
jenkins
jolokia
jolokia2
jti_openconfig_telemetr
y
kafka_consumer
kafka_consumer_legacy
kapacitor
kernel
kernel_vmstat
kibana
kinesis_consumer
kube_inventory
kubernetes
lanz
leofs
linux_sysctl_fs
logparser
logstash
lustre2
mailchimp
marklogic
mcrouter
mem
memcached
mesos
minecraft
modbus
mongodb
monit
mqtt_consumer
multifile
mysql
nats
nats_consumer
neptune_apex
net
net_response
nginx
nginx_plus
nginx_plus_api
nginx_sts
nginx_upstream_check
nginx_vts
nsd
nsq
nsq_consumer
nstat
ntpq
nvidia_smi
opcua
openldap

Input Plugins (3)
openntpd
opensmtpd
openweathermap
passenger
pf
pgbouncer
phpfpm
ping
postfix
postgresql
postgresql_extensib
le
powerdns
powerdns_recursor
processes
procstat
prometheus
proxmox
puppetagent
rabbitmq
raindrops
ras
redfish
redis
rethinkdb
riak
salesforce
sensors
sflow
smart
snmp
snmp_legacy
snmp_trap
socket_listener
solr
sqlserver
stackdriver
statsd
suricata
swap
synproxy
syslog
sysstat
system
systemd_units
tail
tcp_listener
teamspeak
temp
tengine
tomcat
trig
twemproxy
udp_listener
unbound
uwsgi
varnish
vsphere
webhooks
win_eventlog
win_perf_counte
rs
win_services
wireguard
wireless
x509_cert
zfs
zipkin
zookeeper

Processor Plugins
clone
converter
date
dedup
defaults
enum
execd
filepath
ifname
override
parser
pivot
port_name
printer
regex
rename
reverse_dns
s2geo
starlark
strings
tag_limit
template
topk
unpivot

Aggregator Plugins
basicstats
final
histogram
merge
minmax
valuecounter

Output Plugins
amon
amqp
application_insights
azure_monitor
cloud_pubsub
cloudwatch
cratedb
datadog
discard
dynatrace
elasticsearch
exec
execd
file
graphite
graylog
health
http
influxdb
influxdb_v2
instrumental
kafka
kinesis
librato
logzio
mqtt
nats
newrelic
nsq
opentsdb
prometheus_client
riemann
riemann_legacy
socket_writer
stackdriver
sumologic
syslog
timestream
warp10
wavefront
yandex_cloud_monito
ring

Telegraf
CPU
Mem
Disk
Docker
Kubernetes
/metrics
Kafka
MySQL
Process
- transform
- decorate
- filter
Aggregate
- mean
- min,max
- count
- variance
- stddev
InfluxDB
File
Kafka
CloudWatch
CloudWatch
Input
Output
One Telegraf, Multiple Plugins

Can’t find the plugin you need?
Telegraf is 100% open source
with a strong community of
contributors
It’s easy to write your own
Telegraf plugin!
1) Follow the contribution
guide for Go
2) Write your plugin in any
language and run it externally
with ExecD

Popular Plugins
Out-of-the-box Custom
Kubernetes (kubelet) HTTP/socket listener
Kube_inventory (apiserver) HTTP (formatted endpoints)
Kafka (consumer) Prometheus (/metrics)
SNMP Exec/Execd
AMQP (mq metadata) StatsD
Redis
Nginx
HAproxy
Jolokia2

Parsing
JSON
CSV
Graphite
CollectD
Dropwizard
Form URL-
encoded
Grok

Configuration Basics

Telegraf Configuration Snippets
Include many plugins per
Telegraf
- multiple inputs, outputs,
processors, aggregators

The Telegraf config file needs to be specified for Telegraf agent to operate
properly. It contains setup for the agent, global tags, and enabled outputs
(through commenting out or removing unnecessary lines)
What’s in a configuration file?

Set and Test Your Configuration
$ telegraf --input-filter cpu:system --output-filter influxdb --processor-filter
starlark --aggregator-filter valuecounter config > telegraf.conf
WRITE DESIRED CONFIG FILE
RUN CONFIG FILE TEST
$ telegraf --config telegraf.conf --test
VERIFY YOUR CONFIG WORKS

Extending the Ecosystem

● Work with the open-source community or
submit upgrades or enhancements for existing
plugins
● Use ExecD to write a plugin in Go or a language
of your choice
You can extend Telegraf by:

● Plugin runs in its own process
● Requires line protocol
● Avoid the need for review by Telegraf team
● Supports the same API as an internal plugin
● Can use for non-GO plugins
● Can use for licensed software plugins
● Can use for any type of plugin (input, output, processor,
aggregator)
External plugins via ExecD

Execd
Plugin
Arch.

[[processors.execd]]
command = ["python", "./processors/forecasting.py"]
Sample ExecD Config: Input, Output, Processor

ExecD: To Shim, or not to Shim
shim
Use this when
building an
external plugin
using Go
no
shim
Use this when building a
plugin in other languages
(Python, Ruby, etc)

Customer Quotes
“Telegraf is like a swiss army
knife for connecting various
MQTT sources and OPC UA
sources.”
— Fr. Ant. Niedermayr
“Our next-generation
pipeline takes advantage of
Kafka and the Telegraf
streaming service to create a
more robust data topology.
Essentially this allows us to
explicitly implement the four
R’s: routability, retention,
resilience, and redundancy.”
— Wayfair

References
Getting started with Telegraf
Telegraf plugins
Telegraf GitHub Page
External Plugins Guide

Input Plugins
activemq
aerospike
amqp_consumer
apache
apcupsd
aurora
azure_storage_queu
e
bcache
beanstalkd
bind
bond
burrow
cassandra
ceph
cgroup
chrony
cisco_telemetry_mdt
clickhouse
cloud_pubsub
cloud_pubsub_push
cloudwatch
conntrack
consul
couchbase
couchdb
cpu
dcos
disk
diskio
disque
dmcache
dns_query
docker
docker_log
dovecot
ecs
elasticsearch
ethtool
eventhub_consumer
exec
execd
fail2ban
fibaro
file
filecount
filestat
fireboard
fluentd
github
gnmi
graylog
haproxy
hddtemp
http
http_listener_v2
http_response
httpjson
icinga2
infiniband
influxdb

Input Plugins (2)
influxdb_listener
influxdb_v2_listener
intel_rdt
internal
interrupts
ipmi_sensor
ipset
iptables
ipvs
jenkins
jolokia
jolokia2
jti_openconfig_telemetry
kafka_consumer
kafka_consumer_legacy
kapacitor
kernel
kernel_vmstat
kibana
kinesis_consumer
kube_inventory
kubernetes
lanz
leofs
linux_sysctl_fs
logparser
logstash
lustre2
mailchimp
marklogic
mcrouter
mem
memcached
mesos
minecraft
modbus
mongodb
monit
mqtt_consumer
multifile
mysql
nats
nats_consumer
neptune_apex
net
net_response
nginx
nginx_plus
nginx_plus_api
nginx_sts
nginx_upstream_check
nginx_vts
nsd
nsq
nsq_consumer
nstat
ntpq
nvidia_smi
opcua
openldap

Input Plugins (3)
openntpd
opensmtpd
openweathermap
passenger
pf
pgbouncer
phpfpm
ping
postfix
postgresql
postgresql_extensibl
e
powerdns
powerdns_recursor
processes
procstat
prometheus
proxmox
puppetagent
rabbitmq
raindrops
ras
redfish
redis
rethinkdb
riak
salesforce
sensors
sflow
smart
snmp
snmp_legacy
snmp_trap
socket_listener
solr
sqlserver
stackdriver
statsd
suricata
swap
synproxy
syslog
sysstat
system
systemd_units
tail
tcp_listener
teamspeak
temp
tengine
tomcat
trig
twemproxy
udp_listener
unbound
uwsgi
varnish
vsphere
webhooks
win_eventlog
win_perf_counte
rs
win_services
wireguard
wireless
x509_cert
zfs
zipkin
zookeeper

Processor Plugins
clone
converter
date
dedup
defaults
enum
execd
filepath
ifname
override
parser
pivot
port_name
printer
regex
rename
reverse_dns
s2geo
starlark
strings
tag_limit
template
topk
unpivot

Aggregator Plugins
basicstats
final
histogram
merge
minmax
valuecounter

Output Plugins
amon
amqp
application_insights
azure_monitor
cloud_pubsub
cloudwatch
cratedb
datadog
discard
dynatrace
elasticsearch
exec
execd
file
graphite
graylog
health
http
influxdb
influxdb_v2
instrumental
kafka
kinesis
librato
logzio
mqtt
nats
newrelic
nsq
opentsdb
prometheus_client
riemann
riemann_legacy
socket_writer
stackdriver
sumologic
syslog
timestream
warp10
wavefront
yandex_cloud_monitor
ing

How to Use Telegraf and Its Plugin Ecosystem

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to How to Use Telegraf and Its Plugin Ecosystem

Similar to How to Use Telegraf and Its Plugin Ecosystem (20)

More from InfluxData

More from InfluxData (20)

Recently uploaded

Recently uploaded (20)

How to Use Telegraf and Its Plugin Ecosystem