The ultimate guide for Elasticsearch plugins
โขDownload as PPTX, PDFโข
13 likesโข9,338 views
This document discusses Elasticsearch plugins, including: - The different types of plugins like analysis, scripting, and site plugins - Integration points in Elasticsearch that can be extended by plugins - Examples of plugins like custom analyzers and percolators - Considerations for writing plugins like maintenance, versioning, and testing - How to package and install plugins within Elasticsearch
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to The ultimate guide for Elasticsearch plugins
Similar to The ultimate guide for Elasticsearch plugins (20)
Recently uploaded
Recently uploaded (20)
The ultimate guide for Elasticsearch plugins
- 1. Itamar Syn-Hershko http://code972.com @synhershko The ultimate guide for Elasticsearch plugins
- 2. Agenda โข Integration points & plugin types โข Showcases โข Gotchas โข When-to, How-to โข Q & A
- 3. REST API Analysis chain Search Querying Query parser Lucene Index Perform indexing Indexing Make Lucene document ElasticsearchServer
- 4. Lucene extension points Analysis chain Search Query parser Lucene Index Perform indexing
- 5. Lucene extension points Analysis chain Search Query parser Lucene Index Perform indexing
- 6. Lucene extension points Analysis chain Search Query parser Lucene Index Perform indexing
- 7. Harry Potter and the Goblet of Fire Tokenizer Harry Potter and the Goblet of Fire Lower case filter harry potter and the goblet of fire Stop-words filter harry potter goblet fire Step 1: Tokenization Step 2: Filtering
- 8. Welcome to Malmรถ! Tokenizer Welcome to Malmรถ ASCII folding filter Lowercase filter Step 1: Tokenization Step 2: Filtering Welcome to Malmo welcome to malmo
- 9. Harry Potter and the Goblet of Fire Tokenizer Harry Potter and the Goblet of Fire Lower case filter harry potter and the goblet of fire Stop-words filter harry potter goblet fire Potter Tokenizer Potter Lower case filter potter Stop-words filter potter QueryIndexing
- 10. itamar@code972.com Tokenizer itamar code 972 com Lower case filter itamar code 972 com Step 1: Tokenization Step 2: Filtering
- 11. Try searching on German compound wordsโฆ
- 12. Analyzers The quick brown fox jumped over the lazy dog, bob@hotmail.com 123432. StandardAnalyzer: [quick] [brown] [fox] [jumped] [over] [lazy] [dog] [bob@hotmail.com] [123432] StopAnalyzer: [quick] [brown] [fox] [jumped] [over] [lazy] [dog] [bob] [hotmail] [com] SimpleAnalyzer: [the] [quick] [brown] [fox] [jumped] [over] [the] [lazy] [dog] [bob] [hotmail] [com] WhitespaceAnalyzer: [The] [quick] [brown] [fox] [jumped] [over] [the] [lazy] [dog,] [bob@hotmail.com] [123432.] KeywordAnalyzer: [The quick brown fox jumped over the lazy dog, bob@hotmail.com 123432.]
- 13. Custom analyzers from code New in Elasticsearch v1.1.0
- 14. Showcase: Custom Analyzer - Hebrew analysis plugin for Elasticsearch โข https://github.com/synhershko/elasticsearch- analysis-hebrew โข Available on QBox.io
- 15. Lucene extension points Analysis chain Search Query parser Lucene Index Perform indexing
- 16. Scripting โข Sorting, filters, facets, script fields, custom scoring, aggregations, document updates โข MVEL, but others are supported โข Generally speaking: SLOOOOOOOW โข Mostly useful as quick mocks / PoC โข Native scripts using Java by implementing AbstractExecutableScript & AbstractSearchScript
- 17. Custom scoring & similarity โข Function score query โ Previously known as Custom Score Query โข Similarity
- 18. Lucene extension points Analysis chain Search Query parser Lucene Index Perform indexing
- 19. Codecs
- 21. Controlling shard allocation โข Filtering built in โ By tags, groups, racks, IPs โ Black list / white list โข Total shards per node โข Disk based โข EXPERT: Roll your own by implementing AllocationDecider
- 23. Transports โข Exposes the Elasticsearch RESTful API over protocols other than HTTP โ Apache Thrift โ Memcached โ Servlet โ Redis โ ZeroMq
- 25. Showcase: The bubble plugin
- 26. Site plugins โข Monitoring โ BigDesk, ElasticHQ, Paramedic, โฆ โข Hammer (GUI for REST interface) โข Inquisitor (debugging queries) โข SegmentSpy โข WhatsOn
- 27. Discovery โข Default is Zen discovery โ Unicast: I know who my nodes are โ Multicast: Auto discovery for nodes โข Multicast discovery support for cloud environments โ AWS โ Azure โ Google Compute โข ProTip: Unicast in production unless you know what youโre doing โข ZooKeeper plugin
- 28. Snapshot / restore repositories โข File system โข AWS S3 โข HDFS โข Azure โข Roll your own (e.g. Glacier)
- 29. River plugins โข Obsolete โข Use the โshovellerโ approach โข logstash, stream2es
- 30. Summary: Plugin types โข Lucene components โ Analysis โ Similarity โ Scoring โข REST endpoints โข Scripting โข ES infrastructure (Discovery, Transport, Snapshot/restore) โข Site plugins โข River plugins
- 31. Installing plugins โข Manual under /plugins โข Official / GitHub / Maven installation: โข From zip: โข Plugin management:
- 32. When to write a plugin?
- 33. Writing your own plugin: Gotchas โข Maintenance โ the deeper you go in the API the harder it is to keep it up to date โข Versioning and installation on (large) clusters โ Though can be solved using puppet, docker et al โข Auxiliary data (like dictionaries etc) โข Testing & Debugging
- 34. Code: Writing your own plugin โข JAR file with bootstrap code: โข Embed this as es-plugin.properties: plugin=org.elasticsearch.plugin.example.ExamplePlugin
Editor's Notes
- About me โ freelancing, consultant, lucene.net committer Rationale: Elasticsearch can do search, aggregations, percolation and at scale Sometimes we need more than that This talk: birds eye view. Covering a lot of ground here. From experience Skim over EXPERT features
- Rationale: Elasticsearch can do search, aggregations, percolation and at scale
- Elasticsearch in a nutshell: REST, JSON wrapping Lucene Cluster forming and cluster metadata Server distributes Lucene shards (replication, sharding, multi-tenancy)
- Not so interesting since ES discourages the use of them. There are lighter implementations Only applicable for query_string queries Have to be done via code, example will follow
- Analysis chain very important for indexing Some queries will still go through the analysis chain (Match family etc)
- What is the analysis chain? Splitting words Query term should match the indexed term. Term query is the most basic unit. Stop words obsolete => common words query Term query is the most basic building block of a query. Term match is what we need to have
- What is the analysis chain? Splitting words Query term should match the indexed term. Term query is the most basic unit. Stop words obsolete => common words query Term query is the most basic building block of a query. Term match is what we need to have
- Analysis chain should generally match in both ends There scenarios where they differ on purpose This is why you can set search_analyzer & index_analyzer
- Importance of proper tokenization Discussion: on what characters should we tokenize? The curious case of email addresses This is why you probably want to roll your own analyzer if you are doing a lot of FTS
- To finalize my case
- Some basic analyzers shipping with Lucene
- What happens when you try to From code โ custom analyzers, token filters & token filters that you can use
- Hebrew is a tough language to tackle HebMorph - Open-source solution (AGPL3) Requires auxiliary files
- Powered by MVEL โ Java-like syntax Other languages include Groovy, JS, Python You _could_ implement your own scripting engine Dynamic scripting disabled by default Scripts need to be loaded from disk
- Function score query: lookup Brittaโs talk Similarity: replacement for TF/IDF. Out of the box: BM25, DFR, IB and more EXPERT EXPERT EXPERT
- EXPERT ONLY Lucene 4.0 feature Can provide performance boosts for searches and aggregations
- Zoom out In the integration point Stats, management, โฆ
- Some of the built-in features Roll your own if needed Con: requires tons of testing, multiple deciders are at play
- Thanks to Found A way to expose new plugin functionality to consumers not using Java Or leverage the HTTP server capabilities of ES for your requirements Parsing request, performing action, creating and sending response
- Better query filtering for performance (less queries) Highlighting More logs + custom logs Various other optimizations
- A la significant terms facet We could have done this client-side only. This would have been linear in time We made this sharded Java client code Debugging
- A static website that can be served using ES HTTP server
- Multicast: the more the merrier problem Zookepper plugin โ not up to date, not official Aphyrโs finding re partial partitions
- The idea behind them Why rivers are obsolete: node comes down, backlog Always prefer push over pull Official guidance is not to use them going forward
- Plugin names specify the folder name under /plugins Node info API can provide
- Donโt be that guy: ignore the urge to write custom stuff The defaults are good + A lot can be done w/ scripting Basically, when you really need custom distributed behavior Or REST endpoint exposed cluster wise Or EXPERT FEATURES
- Aux data โ open ES ticket for enabling analyzers to read docs
- JAR (has to be JVM code) Boilerplate setup and code Modules; AnalysisBinderProcessor; TransportActions; RestActions; Everything in Elasticsearch is implemented as an Action Client / server reuse of request/response classes, when in Java
- Summary