How we use Zabbix at BlaBlaCar. What we did to be able to deal with >25k items, >300 values per second & >6.5k triggers - Trappers everywhere - Low level discovery - python-protobix - jmx-zabbix

1. at

2. Who am I ?
Ops Lead @BlaBlaCar
@jbfavre
http://{blog,www}.jbfavre.org

3. Agenda
●BlaBlaCar ?
●Zabbix
●
(very quick) introduction
●
at BlaBlaCar
●python-protobix
●Jmx-zabbix

4. BlaBlaCarBlaBlaCar

5. BlaBlaCarBlaBlaCar

6. Zabbix
●Server / Proxy / Agent
●3 ways to get items
●
Agent
●
Agent (active)
●
Trappers
●SNMP
●JMX

7. Zabbix
●Many other (very) cool features like
●
Low Level Discovery
●
VmWare monitoring
●
Web Scenario

8. Zabbix
25k items
Biggest host>1k items

9. Zabbix at BlaBlaCar
●~ 200 hosts
●~ 6.5k triggers
●75 templates

10. StandardizationStandardization

11. Standardization
As few zabbix-agent items as possible
Zabbix-agent (active) triggers a script

12. Standardization
All probes written in Python
Easy to maintain
Widely deployed on systems
Or Java
When accurate
Use LLD wherever possible
Avoid template duplication
Only 2 zabbix-agent (active) items
Triggers script to update items
Triggers script for LLD

13. python-protobixpython-protobix

14. python-protobix
At first
A personal project to learn Python
What it is now
Unified way of dealing with trappers
https://github.com/jbfavre/python-protobix
(Soon on pypi.python.org)

15. python-protobix
Support LLD
#!/usr/bin/env python
import protobix
''' create DataContainer, providing data_type, zabbix server and port '''
zbx_container = protobix.DataContainer('lld', 'localhost', 10051)
hostname='myhost'
item='hardware.power_supply'
value=[
{ '{#SLOT}': 0, '{#PLUGGED}' : 1 },
{ '{#SLOT}': 1, '{#PLUGGED}' : 0 },
]
zbx_container.add_item( hostname, item, value)
try:
zbx_response = zbx_container.send(zbx_container)
except protobix.SenderException:
print 'Oups...'

16. python-protobix
As well as item's update
#!/usr/bin/env python
import protobix
''' create DataContainer, providing data_type, zabbix server and port '''
zbx_container = protobix.DataContainer('items', 'localhost', 10051)
hostname='myhost'
item='hardware.power_supply[0,status]'
value=1
zbx_container.add_item( hostname, item, value)
try:
zbx_response = zbx_container.send(zbx_container)
except protobix.SenderException:
print 'Oups...'

17. python-protobix : RabbitMQ
LLD for vhosts & queues
blacklist queues (support regex)
LLD also send limits
message number
receive/consume ratio
Can override limits per queue

18. python-protobix : MariaDB
Low Level Discovery
for « plugins »
Galera
for storage engine
Only support InnoDB & MyISAM for now
Spider coming...
for multi-replication

19. python-protobix
Plugin's list :
hadoop
couchbase
elasticsearch
memcached
mariadb
pacemaker
php-fpm
rabbitmq
redis
supervisord
varnish

20. Jmx-zabbixJmx-zabbix

21. Jmx-zabbix
●Because python is not enough :)
●Trapper's power brought to Java JMX
https://github.com/n0rad/jmx-zabbix

22. Jmx-zabbix
●Embedded inside a Java process
●
Internal Java daemons
●Aside any Java process (separate service)
●
Cassandra
●
Elasticsearch
●
…

23. Jmx-zabbix
Configure jmx-zabbix :
serverName: <hostname in Zabbix>
pushIntervalSecond: 60
inMemoryMaxQueueSize: 10
zabbix:
host: <Zabbix server hostname or IP>
port: 10051
jmx:
url: service:jmx:rmi:///jndi/rmi://localhost:7199/jmxrmi
username: zabbix
password: zabbix
timeoutSecond: 30
[...]

24. Jmx-zabbix
Map JMX metrics with Zabbix ones
(example with Cassandra)
metrics:
cassandra.status.failure: org.apache.cassandra.net:type=FailureDetector
cassandra.status.timeouts: org.apache.cassandra.net:type=MessagingService
cassandra.db.storage: org.apache.cassandra.db:type=StorageProxy
And specify which values you want
valuesCaptured:
org.apache.cassandra.gms.FailureDetector: ["DownEndpointCount"]
org.apache.cassandra.net.MessagingService: ["RecentTotalTimouts"]
org.apache.cassandra.service.StorageProxy: ["RecentRangeLatencyMicros",
"RecentReadLatencyMicros", "RecentWriteLatencyMicros"]

25. Questions ?

26. BlaBlaCar
We hire !
https://careers.smartrecruiters.com/BlaBlaCar/