Elasticsearch quick Intro (English)

Elasticsearch
federico.panini@fazland.com - CTO
Federico Panini
CTO @ fazland.com
email : federico.panini@fazland.com
LikedIn : https://uk.linkedin.com/in/federicopanini
slides : http://www.slideshare.net/FedericoPanini

What is Elasticsearch
full-text search engine
“A search engine is an automated system which, upon
request, uses a set of data and return an index of its content
classifying them based on math/stats algorithm used to set
the relevance, based in a search key.”

What’s Elasticsearch ?

“It’s a distributed, scalable, and highly available
Real-time search and analytics software.”

Real-time data
Realtime data analysis
Distributed system
High Availability
Full-text searches
Document oriented DB
Schemaless DB
RESTFul Api
Persistence per-operation
Open Source
Based on Apache Lucene
Optimistic version control
features

Apache Lucene #1
It’s the heart of Elasticsearch
Lucene is the search engine of Elasticsearch

Apache Lucene #1
It’s in Java
It’s an Apache Software Foundation, so Open Source!

What has more than Lucene
full-text searches
horizontal scaling
high availability
Easy to use
near real time

Architecture
requirements - CPU
Elasticsearch doesn’t need a lot of CPU.
The advice is to use the last CPU model available.
In general is a good practice to use machines with 2 to 8
cores.

Architecture
requirements - Disco
The I/O disk need is really important for all clusters.
Please use SSD disks.

Architecture
requirements - HD - bonus slide …
One very important thing to know is you have to pay attention where
data is stored and mostly how. The word you have to remember is
scheduler. The scheduler on *nix system is responsible to decide
when data should be “written” to disc and on which priority. Usually
common unix OS setup cfq as scheduler, which for instance is a
scheduler for rotating disks and optimised for them. The advice is to
use SSD disks and to setup the SO to use “noop” or “deadline”
which are scheduler optimised for SSD’s.
If you use the right scheduler you can reach improvements of
500x !!!

Operating Systems
Elasticsearch is written in
Java, so it’s a multiplatform
solution. Use the last JDK
available.
Architecture

requirements - RAM
Elasticsearch is eager of RAM!!!
https://www.elastic.co/guide/en/elasticsearch/guide/current/
heap-sizing.html
Architecture

memory !?!?
Use solutions with 64GB is ﬁne not more
give to the Java heap size not more than 32GB of RAM
use more than one machine for elasticsearch in order
setup correctly the cluster.
Architecture

Installation
curl -L -O http://download.elasticsearch.org/PATH/TO/
VERSION.zip
unzip elasticsearch-$VERSION.zip
cd elasticsearch-$VERSION
There are availbes packages for many distribution as
Debian or RPM, and Puppet or Chef modules
Architecture

Java based
elasticsearch
Elasticsearch has been developed in JAVA
Robust
Scalable
Multiplatform

Talking to Elasticsearch
clients Java #1
There are 2 clients available in JAVA:
Node client : the client join the cluster
as non-data node, this mean that the
client knows perfectly where data are
and on which node of the cluster.

clients Java #2
Transport client : is a lightweight client
and is the tool used to comunicate
with the cluster remotely.

clients Java #2
Both Java clients talk to the cluster on
port 9300, which is the same port use by
the cluster itself.

client API RESTful
All programming languages other than Java can
talk to the Elasticsearch cluster through
its API Rest available on port 9200.
There are many ofﬁcial clients
available in different programming
languages.:
Groovy, JavaScript, .NET, PHP,
Perl, Python, e Ruby

Elastic
Document oriented
NoSql
Elasticsearch is a document
oriented database. This mean
Elasticsearch is a schema-less
database.
After inserting documents inside
Elasticsearch, the documents will
be immediately indexed.

Elastic
Document oriented
JSON
Elasticseach uses JSON as
interchange language between
the server and the API clients.

Elastic
glossary
cluster
nodes
indexes
shards
replica
segments
in-memory buffers
translog

Elastic
cluster
The cluster is a set which belong one or more nodes,
which shares the same property cluster.name. The
cluster is used to balance the load of the server itself.
A node could be deleted or inserted to the cluster, the
cluster itself will re-organise itself.

Elastic
cluster
Inside a cluster a node is elected as Master. The
Master node is responsible to manage operations as
creation or removal indexes, join or deletion of a node.
Every node could be elected as Master.

Elastic
nodes
A node is a minimum element of Elasticsearch that
ensures the proper working of the cluster.

Elastic
Index
Database RDBMS Elasticsearch
DATABASE INDEX

Elastic
Type
Table TYPE

Elastic
Document
ROW DOCUMENT

Elastic
Fields
COLUMNS FIELDS

Elastic
shards
If we want to start indexing data on Elasticsearch we
need to create an index. Index is the term used only to
identify a logical deﬁnition, which represent a pointer to
one or more elements called SHARDS.

Elastic
shards
The shard is the low level element of Elasticsearch, and
contains a subset of all the data inside and index.
The shard is in fact a single instance of Apache Lucene.

Elastic
Replica shards
Replica shards are mirrors of shards used to protect our
data from hardware failures. As the shards they are used
exactly as the shards.

Elastic
shards immutability
The number of shards for an index is deﬁned at Index
creation time and is IMMUTABLE.

Elastic
shards immutability
curl -X http://localhost:9200/blogs
-d ‘{
"settings" : {
"number_of_shards" : 3,
"number_of_replicas" : 1
}
}’

Elastic
shards immutability
curl http://localhost:9200/_cluster/health“{
"cluster_name": "elasticsearch",
"status": "yellow",
"timed_out": false,
"number_of_nodes": 1,
"number_of_data_nodes": 1,
"active_primary_shards": 3,
"active_shards": 3,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 3
}”

Elastic
shards immutability
Replica shards on a single node instance are useless, the
meaning for cluster is nothing in this case. To make
replica shard useful we need at least 2 nodes to have
data redundancy.

Elastic
BONUS : manage data conflicts #1

Elastic
BONUS : manage conflicts #2 : Pessimistic Concurrency Control
Used in standard RDBMS
This approach is based on the concept that conﬂict could
happened frequently and so to avoid them the RDBMS
lock the resource.
The process lock the access to the row before reading it,
this way we the RDBMS is sure that only one process will
access to this thread and can subsequently modify it and
nobody else.
At the end of its process (update/delete) the thread will
release the LOCK.

Elastic
BONUS : manage conflicts #3 : Optimistic Concurrency Control
Elasticsearch uses OCC
This approach will consider conﬂicts as infrequent. The
database won’t lock the resource when access to it.
The responsibility is given to the application : when data is
amended between a read and write then the update fails.
In this case you need to re-get the fresh new data and
trying to update it.

Elastic
BONUS : manage conflicts#4 : Optimistic Concurrency Control
Elasticsearch is a distributed solution, concurrent and
asynchronous. When a document is created / updated /
deleted is absolutely necessary to replicate this
information across the whole cluster.
Every command sent to the nodes is sent in parallel and
could happen that some data will reach its destination
(node) already expired.

Elastic
We need a way to understand that the entry
we’re trying to update as been already
updated by another process.

Elastic
VERSIONING

Elastic
In Elasticsearch every document has a field named:
_version
This system field is incremented every time an operation
(update / delete) occurs over a document. In this way an
update to _version:3 won’t be never applied to a
document whose _version field value is at 4.

Elastic
This approach move all the responsibility from the
database to the application! so WE are responsible to not
create conﬂicts over a document or and index. If we want
to be sure to not have loss of data we nee to implement
writes with the use of versioning!

Elastic
http://www.jillesvangurp.com/2014/12/03/optimistic-
locking-for-updates-in-elasticsearch/
https://aphyr.com/posts/317-call-me-maybe-
elasticsearch
https://www.elastic.co/guide/en/elasticsearch/resiliency/
current/index.html

Elastic
Simple searches #1
Create Index
API Rest
GET
DELETE
POST
SEARCH

Elastic
Simple searches - CREATE AN INDEX
curl -XPUT http://fazlab.fazland.com:9200/fazlab
-d
"{ "settings" :
{
"number_of_shards" : 3,
"number_of_replicas" : 1
}
}"

Elastic
Simple searches - INDEX A DOCUMENT
curl -XPUT
http://fazlab.fazland.com:9200/fazlab/categories/1?pretty
-d '
{
nome: "Federico"
}'

Elastic
Simple searches - GET A DOCUMENT
curl http://fazlab.fazland.com:9200/fazlab/categories/1?pretty

Elastic
Simple searches - DELETE A DOCUMENT
curl -XDELETE
http://fazlab.fazland.com:9200/fazlab/categories/2?pretty

Elastic
Simple searches #1
DEMO SEARCHES!

Elastic
mapping and analysis
EXACT MATCH vs FULL TEXT

Elastic
Exact match Full Text
where name = ‘Federico’
and user_id = 2
and date > “2014-09-15”
“Frank has been to
South beach”
Frank / FRANK / frank

Elastic
Exact match
Full Text
binary : the document contains these values ?
How much is relevant the document compared to the
term used inside the query ?

Elastic
Elasticsearch to help a full-text search analyse the text
and uses this result to build an inverted index.
Inverted Index Analyzer

Elastic
Inverted Index
1. The quick brown fox jumped
over the lazy dog
2. Quick brown foxes leap over
lazy dogs in summer

Elastic
Inverted Index
If we want to search the word
“quick” and “brown” we will pick
only the documents where these 2
words are.
1. The quick brown fox jumped
over the lazy dog
2. Quick brown foxes leap over
lazy dogs in summer

Elastic
ANALYZERS
An analyzer has 3 functions:
Character ﬁlters
Tokenizer
Token Filters

Elastic
ANALYZERS - Character Filters
The ﬁrst part of an analyser is to parse every string with
character ﬁler which will clean / reorganize the strings
before tokenization.
During this phase special HTML chars will be removed
or & will be converted in AND.

Elastic
ANALYZERS - Tokenizer
The second phase of an analyser is tokenisation which
will divide a sentence in small terms.

Elastic
ANALYZERS - Token Filters
Successivamente alla fase di Tokenizzazione delle
stringhe in singoli termini (terms), i ﬁltri (selezionati) sono
applicati in sequenza.
After tokenisation ﬁlters will be applied in sequence.
For example :
- put lower case the whole text
- remove stop words
- add synonyms

Elastic
Standard Analyzer
“Set the shape to semi-transparent by calling
set_trans(5)”
The standard analyzer is the default analyzer of
Elasticsearch. Divide text in single words and remove
most of punctuation.
“set, the, shape, to, semi, transparent, by, calling,
set_trans, 5”

Elastic
Simple Analyzer
set_trans(5)”
The simple analyser removes all characters which are
not letters and put the whole text lowercase
“set, the, shape, to, semi, transparent, by, calling,
set, trans”

Elastic
Whitespace Analyzer
set_trans(5)”
The whitespace analyser will create token by white
space and put text in lowercase
“Set, the, shape, to, semi, transparent, by, calling,
set_trans(5)”

Elastic
Language Analyzer
set_trans(5)”
This analyser uses a language speciﬁc feature to
remove stop words or to do stemming.
“set, shape, semi, transpar, call, set_tran, 5”

Elastic
Language Analyzer
arabic, armenian, basque, brazilian, bulgarian, catalan,
chinese, cjk, czech, danish, dutch, english, ﬁnnish,
french, galician, german, greek, hindi, hungarian,
indonesian, irish, italian, latvian, norwegian, persian,
portuguese, romanian, russian, sorani, spanish,
swedish, turkish, thai.

Elastic
Pre-built Analyzers
Standard Analyzer
Simple Analyzer
Whitespace Analyzer
Stop Analyzer
Keyword Analyzer
Pattern Analyzer
Language Analyzers
Snowball Analyzer
Custom Analyzer

Elastic
Tokenizer
Standard Tokenizer
Edge NGram Tokenizer
Keyword Tokenizer
Letter Tokenizer
Lowercase Tokenizer
NGram Tokenizer
Whitespace Tokenizer
Pattern Tokenizer
UAX Email URL Tokenizer
Path Hierarchy Tokenizer

Elastic
Token Filters
Standard Token Filter
ASCII Folding Token Filter
Length Token Filter
Lowercase Token Filter
NGram Token Filter
Edge NGram Token Filter
Porter Stem Token Filter
Shingle Token Filter
Stop Token Filter
…
more than 32 Filters

Elastic
Token Filters
THE END.

References
• Elasticsearch : The Deﬁnitive Guide
• https://en.wikipedia.org/wiki/Full_text_search
• https://www.elastic.co/guide/en/elasticsearch/guide/current/
hardware.html
• https://www.elastic.co/guide/en/elasticsearch/guide/current/
heap-sizing.html
• https://mtalavera.wordpress.com/2015/02/16/monitoring-with-
collectd-and-kibana/
• Fuzzy search : https://www.found.no/foundation/fuzzy-search/
• Phonetic-plugin : https://github.com/elastic/elasticsearch-
analysis-phonetic

Elasticsearch quick Intro (English)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (12)

Similar to Elasticsearch quick Intro (English)

Similar to Elasticsearch quick Intro (English) (20)

More from Federico Panini

More from Federico Panini (6)

Recently uploaded

Recently uploaded (20)

Elasticsearch quick Intro (English)