Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Delhi elasticsearch meetup

1,786 views

Published on

Slides from the first delhi elasticsearch meetup.

Published in: Data & Analytics
  • Login to see the comments

Delhi elasticsearch meetup

  1. 1. Delhi Elasticsearch Meetup Bharvi Dixit @d_bharvi Nov 29, 2014 Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch"
  2. 2. Agenda  Whatisasearchengine?  LuceneOverviewandIndexingPipeline.  DataDrivenApproaches&Problems.  ElasticsearchComestoRescue.  UnderstandingElasticsearchArchitecture.  Logstash&KibanaOverview.  TheELKstacktogether.  Sometips. Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  3. 3. About Me  Softwareengineer@Orkash.  LovesJava,Data,Elasticsearch,MongoDB,Eclipse.  Interestedinallthingsscale,search,security&DevOps.  Creator:CIBETProManager  WorkingonElasticsearchformorethanayear.  SocialMediaandNewsMediaIntelligence.(Complexschemas&Querydesigns) Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  4. 4. What is a search engine? • Aninformationretrievalsystemdesignedtofindinformationstoredincomputersystem. Asearchenginehasdifferentmodules: • But what about the relevant or irrelevant results?? Data collected from various sources Data stored in indexes Data is queried Indexing Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  5. 5. What is a search engine? Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014 Auto completion Did-You-Mean Spell correction Multi-lingual Stemming Synonyms Highlighting More-Like-This
  6. 6. Lucene Overview Lucene: • Open source, Fast, high performance, search/IR library. • Written in Java. • Initially developed by Doug Cutting (Also author of Hadoop) • Indexing and Searching. • Inverted Index of documents. • Provides advanced Search options like synonyms, stopwords, similarity, proximity. Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  7. 7. Lucene Internals-Inverted Index Credit: https://developer.apple.com/library/mac/documentation/userexperience/conceptual/SearchKitConcepts/searchKit_basics/searchKit_basics.html Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  8. 8. Lucene Internals-Continued • Defines documents Model • Index contains documents. • Each document consist of fields. • Each Field has attributes. – What is the data type (FieldType) – How to handle the content (Analyzers, Filters) – Is it a stored field (stored="true") or Index field (indexed="true") Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  9. 9. Indexing Pipeline •Analyzer : create tokens using a Tokenizerand/or applying Filters(Token Filters) •Each field can define an Analyzer at index time/query time or the both at same time. Document Tokenizer Document Writer Token Filter Inverted Index Analysis Phase Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  10. 10. Everything starts with a problem..!! • Data Driven Decisions • Logfiles for scaling up/down • Warehouse withdrawal triggers orders • History for fraud detection • Assembly line, throughput improvement ... data explosion Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  11. 11. Everything starts with a problem..!! Betterdecisions==moredata? Data Big Data BIG DATA Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  12. 12. Big Data Problem goes on.. • I need BIG DATA. • I need to analyze this data. • I need to enrich this big data & make it more bigger. • I need fast searching. • I need real-time analytics. • Ohh wait.. I need relational queries on this big data to get more insights.. • I need .. I need .. I need.. Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  13. 13. And I guess this is why someone nailed it.. Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  14. 14. Elasticsearch comes to rescue.. WhatisElasticsearch: • “youknow,forsearch” • Schema-free,REST&JSONBaseddistributedFullTextsearchengine&documentstore. • WritteninJAVA&BuildontopofLucene. • Highlyreliable,scalable,faulttolerant. • SupportdistributedIndexing,Replication,andloadbalancedquerying. • PowerfulGeo-SpatialQueries. • LatestRelease:1.4.1 Wait..!!SchemaFree??Therealgotcha.. Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  15. 15. Elasticsearch comes to rescue.. WhatdoesitaddtoLucene: • RESTservice:JsonAPI’soverHTTP • HighAvailability&Performance:Clustering&Replication • APowerfulqueryDSL. • Interoperationwithnon-Java/JVMlanguages. • MoreandmoreResilience. • Multitenancy • Andthebestone:Itallowstomaintainrelationshipamongdocuments. Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  16. 16. The Elasticsearch Open Source Model Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  17. 17. The Popularity of Elasticsearch 10M downloads in 2 years and counting.. Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  18. 18. The Popularity of Elasticsearch Have a look at the case studies here: http://www.elasticsearch.org/case-studies/ Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  19. 19. Understanding Elasticsearch Structure A live demo is better then nothing Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  20. 20. Logstash • ToolforReceiving,processingandoutputtinglogs. (Input======Filter======Output) • Allkindsoflogs:Systemlogs,errorlogs,webserverlogs, applicationlogs&justaboutanythingyoucanthrowatit. • OpenSource:ApacheLicense2.0. Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  21. 21. Kibana • Execute queries on your data & visualize results. • Add/remove widgets. • Share/Save/Load dashboards. • No need to know coding. • Open Source: Apache License 2.0. Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  22. 22. The ELK Stack Together Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  23. 23. meetup.com RSVP stream • All RSVPs are written out to a HTTP stream • Each line is a JSON document • Available at http://stream.meetup.com/2/rsvps Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  24. 24. meetup.com RSVP stream Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  25. 25. In the end.. • Look out for best practices. (Proper cluster formation, Bulk Indexing) • Continuous monitoring: Marvel, Bigdesk, HQ • Open-JDK strictly prohibited. • Elasticsearch is the always hungry: Give me more RAM..!! • Benchmarking of data to create indexes/shards. (Once created; can’t be broken) • And don’t forget to create mappings. • Manage your security.. But Now It’s coming soon.. Elasticsearch Shield.. “you know, for security” Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014
  26. 26. Thank You for Listening bharvidixit@yahoo.com https://twitter.com/d_bharvi slideshare.net/bharvidixit/ Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world "Elasticsearch" Nov 29, 2014

×