Slides from the first delhi elasticsearch meetup.

1. Delhi Elasticsearch Meetup
Bharvi Dixit
@d_bharvi
Nov 29, 2014 Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"

2. Agenda

Whatisasearchengine?

LuceneOverviewandIndexingPipeline.

DataDrivenApproaches&Problems.

ElasticsearchComestoRescue.

UnderstandingElasticsearchArchitecture.

Logstash&KibanaOverview.

TheELKstacktogether.

Sometips.
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

3. About Me

Softwareengineer@Orkash.

LovesJava,Data,Elasticsearch,MongoDB,Eclipse.

Interestedinallthingsscale,search,security&DevOps.

Creator:CIBETProManager

WorkingonElasticsearchformorethanayear.

SocialMediaandNewsMediaIntelligence.(Complexschemas&Querydesigns)
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

4. What is a search engine?
•
Aninformationretrievalsystemdesignedtofindinformationstoredincomputersystem.
Asearchenginehasdifferentmodules:
•
But what about the relevant or irrelevant results??
Data collected from various sources
Data stored in indexes
Data is queried
Indexing
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

5. What is a search engine?
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014
Auto completion
Did-You-Mean
Spell correction
Multi-lingual
Stemming
Synonyms
Highlighting
More-Like-This

6. Lucene Overview
Lucene:
•
Open source, Fast, high performance, search/IR library.
•
Written in Java.
•
Initially developed by Doug Cutting (Also author of Hadoop)
•
Indexing and Searching.
•
Inverted Index of documents.
•
Provides advanced Search options like synonyms, stopwords, similarity, proximity.
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

7. Lucene Internals-Inverted Index
Credit: https://developer.apple.com/library/mac/documentation/userexperience/conceptual/SearchKitConcepts/searchKit_basics/searchKit_basics.html
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

8. Lucene Internals-Continued
•
Defines documents Model
•
Index contains documents.
•
Each document consist of fields.
•
Each Field has attributes.
–
What is the data type (FieldType)
–
How to handle the content (Analyzers, Filters)
–
Is it a stored field (stored="true") or Index field (indexed="true")
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

9. Indexing Pipeline
•Analyzer : create tokens using a Tokenizerand/or applying Filters(Token Filters)
•Each field can define an Analyzer at index time/query time or the both at same time.
Document
Tokenizer
Document Writer
Token Filter
Inverted Index
Analysis Phase
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

10. Everything starts with a problem..!!
•
Data Driven Decisions
•
Logfiles for scaling up/down
•
Warehouse withdrawal triggers orders
•
History for fraud detection
•
Assembly line, throughput improvement
... data explosion
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

11. Everything starts with a problem..!!
Betterdecisions==moredata?
Data
Big Data
BIG DATA
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

12. Big Data Problem goes on..
•
I need BIG DATA.
•
I need to analyze this data.
•
I need to enrich this big data & make it more bigger.
•
I need fast searching.
•
I need real-time analytics.
•
Ohh wait.. I need relational queries on this big data to get more insights..
•
I need .. I need .. I need..
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

13. And I guess this is why someone nailed it..
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

14. Elasticsearch comes to rescue..
WhatisElasticsearch:
•
“youknow,forsearch”
•
Schema-free,REST&JSONBaseddistributedFullTextsearchengine&documentstore.
•
WritteninJAVA&BuildontopofLucene.
•
Highlyreliable,scalable,faulttolerant.
•
SupportdistributedIndexing,Replication,andloadbalancedquerying.
•
PowerfulGeo-SpatialQueries.
•
LatestRelease:1.4.1
Wait..!!SchemaFree??Therealgotcha..
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

15. Elasticsearch comes to rescue..
WhatdoesitaddtoLucene:
•
RESTservice:JsonAPI’soverHTTP
•
HighAvailability&Performance:Clustering&Replication
•
APowerfulqueryDSL.
•
Interoperationwithnon-Java/JVMlanguages.
•
MoreandmoreResilience.
•
Multitenancy
•
Andthebestone:Itallowstomaintainrelationshipamongdocuments.
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

16. The Elasticsearch Open Source Model
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

17. The Popularity of Elasticsearch
10M downloads in 2 years and counting..
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

18. The Popularity of Elasticsearch
Have a look at the case studies here:
http://www.elasticsearch.org/case-studies/
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

19. Understanding Elasticsearch Structure
A live demo is better then nothing
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

20. Logstash
•
ToolforReceiving,processingandoutputtinglogs. (Input======Filter======Output)
•
Allkindsoflogs:Systemlogs,errorlogs,webserverlogs, applicationlogs&justaboutanythingyoucanthrowatit.
•
OpenSource:ApacheLicense2.0.
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

21. Kibana
•
Execute queries on your data & visualize results.
•
Add/remove widgets.
•
Share/Save/Load dashboards.
•
No need to know coding.
•
Open Source: Apache License 2.0.
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

22. The ELK Stack Together
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

23. meetup.com RSVP stream
•
All RSVPs are written out to a HTTP stream
•
Each line is a JSON document
•
Available at http://stream.meetup.com/2/rsvps
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

24. meetup.com RSVP stream
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

25. In the end..
•
Look out for best practices. (Proper cluster formation, Bulk Indexing)
•
Continuous monitoring: Marvel, Bigdesk, HQ
•
Open-JDK strictly prohibited.
•
Elasticsearch is the always hungry: Give me more RAM..!!
•
Benchmarking of data to create indexes/shards. (Once created; can’t be broken)
•
And don’t forget to create mappings.
•
Manage your security.. But Now It’s coming soon..
Elasticsearch Shield.. “you know, for security”
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014

26. Thank You for Listening
bharvidixit@yahoo.com
https://twitter.com/d_bharvi
slideshare.net/bharvidixit/
Delhi Elasticsearch Meetup: Talk About Most Advance Search Engine of The world
"Elasticsearch"
Nov 29, 2014