Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Elasticsearch logstash kibana meetup

1,108 views

Published on

Published in: Data & Analytics
  • Login to see the comments

  • Be the first to like this

Elasticsearch logstash kibana meetup

  1. 1. Who Am I
  2. 2. • • • •
  3. 3. • • • • • • • • •
  4. 4. • • • • • • • • • • • • •
  5. 5. • • • • • • • • • • RAM, CPU type/cores, DISK, Networks matter a lot, But the cluster design, data structures of documents and queries has a huge impact on the Elasticsearch Clusters and your search experiences. I have experienced it a lot. You can check out some of the scenarios here : https://www.found.no/foundation/crash- elasticsearch/
  6. 6. • • • •
  7. 7. • • • • •
  8. 8. • • • • • • •
  9. 9. • • • • • • • • • • • • •
  10. 10. • • • • • • • • • • • •
  11. 11. Centralize Analyze Monitor Share
  12. 12. • • •
  13. 13. • • • • • •
  14. 14. #!/bin/bash sudo apt-get purge openjdk-* wget --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.tar.gz tar -xvf jdk-7u79-linux-x64.tar.gz sudo mkdir -p /usr/lib/jvm sudo mv ./jdk1.7.0_79 /usr/lib/jvm/ sudo update-alternatives --install "/usr/bin/java" "java" "/usr/lib/jvm/jdk1.7.0_79/bin/java" 1 sudo update-alternatives --install "/usr/bin/javac" "javac" "/usr/lib/jvm/jdk1.7.0_79/bin/javac" 1 sudo update-alternatives --install "/usr/bin/javaws" "javaws" "/usr/lib/jvm/jdk1.7.0_79/bin/javaws" 1 sudo chmod a+x /usr/bin/java sudo chmod a+x /usr/bin/javac sudo chmod a+x /usr/bin/javaws sudo chown -R root:root /usr/lib/jvm/jdk1.7.0_79 sudo update-alternatives --config java ##########Skip above if you have java already available. wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.4.deb sudo dpkg -i elasticsearch-1.4.4.deb sudo update-rc.d elasticsearch defaults 95 10
  15. 15. script.disable_dynamic: false cluster.name: give_your_cluster_name node.name:"es-master-3" node.master: true node.data: false node.max_local_storage_nodes: 1 index.number_of_shards: 4 index.number_of_replicas: 1 bootstrap.mlockall: true transport.tcp.port: 9300 http.enabled:false discovery.zen.minimum_master_nodes:2 discovery.zen.ping.timeout:10s discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts:["es-master-1:9300","es-master-2:9300", "es-master-3:9300"] action.disable_delete_all_indices: false action.destructive_requires_name: true marvel.agent.exporter.es.hosts: ['es-monitor-1:6200'] /etc/elasticsearch/elasticsearch.yml Configuration for Master Node. Give half of total available RAM to ES: vim /etc/init.d/elasticsearch ES_HEAP_SIZE=2g ( on a 4 GB RAM, the more heap size the better performance, But need to keep sweet spot of 32 GB in mind)
  16. 16.
  17. 17. Shipper Logstash Shipper Logstash Shipper Logstash Broker Redis/RabbitMQ Indexer Logstash Elasticsearch Using Logstash Shippers and Redis or RabiitMQ as a broker
  18. 18. img source: https://deviantony.wordpress.com/2014/05/19/centralized-logging-with-an-elk-stack-elasticsearch-logback-kibana/ Using Logstash Forwarder : Lumberjack Protocol
  19. 19. img source: http://blog.greg.lu/ Or a combination of both:
  20. 20. Don’t want to use any agent/shipper for system events logging?? Use Rsyslog: 1. Send logs directly to Elasticsearch: omelasticsearch http://www.rsyslog.com/doc/v8- stable/configuration/modules/omelasticsearch.html 2. Use base logs sending : Create and edit the file /etc/rsyslog.d/logstash.conf with *.* @logserver.example.com:5544 and parse the logs on centralized logstash server.
  21. 21. Rivers are finally removed: https://github.com/elastic/elasticsearch/pull/11568#event-332821650 Want to get a deep dive into Elasticsearch?? Join us @ 3rd Delhi Elasticsearch Meetup http://www.meetup.com/Delhi-Elasticsearch-Meetup/events/223470631/

×