1. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Getting Down and Dirty
with Elasticsearch
@clintongormley
Berlin Buzzwords 2013
2. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch
3. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch
real time,
search and
analytics engine
4. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch
real time,
search and
analytics engine
distributed
5. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch
real time,
search and
analytics engine
distributed
scales
massively
6. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch
real time,
search and
analytics engine
distributed
scales
massively
high
availability
7. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch
real time,
search and
analytics engine
distributed
scales
massively
high
availability
RESTful
API
8. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch
real time,
search and
analytics engine
distributed
scales
massively
high
availability
RESTful
API
JSON
over HTTP
9. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch
real time,
search and
analytics engine
distributed
scales
massively
high
availability
RESTful
API
JSON
over HTTP
schema
free
10. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch
real time,
search and
analytics engine
distributed
scales
massively
high
availability
RESTful
API
JSON
over HTTP
schema
free
multi
tenancy
11. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch
real time,
search and
analytics engine
open-source
distributed
scales
massively
high
availability
RESTful
API
JSON
over HTTP
schema
free
multi
tenancy
12. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Elasticsearch
real time,
search and
analytics engine
open-source
Lucene
based
distributed
scales
massively
high
availability
RESTful
API
JSON
over HTTP
schema
free
multi
tenancy
13. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Cool.
14. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Cool. Bonsai cool...
15. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
This is WHY we use it...
16. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
> ./bin/elasticsearch
> _
17. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
But HOW do we use it?
18. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
> curl -XGET localhost:9200/?pretty
19. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
> curl -XGET localhost:9200/?pretty
verb
20. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
> curl -XGET localhost:9200/?pretty
node
21. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
> curl -XGET localhost:9200/?pretty
HTTP port
22. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
> curl -XGET localhost:9200/?pretty
path
23. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
> curl -XGET localhost:9200/?pretty
query string
24. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
> curl -XGET localhost:9200/?pretty
25. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /
26. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /
{
"name" : "Exploding Man",
"tagline" : "You Know, for Search",
"ok" : true,
"status" : 200,
"version" : {
"number" : "0.90.1",
"snapshot_build" : false
}
}
27. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Where do we start?
28. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
With data
29. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
{
"tweet": "I think #elasticsearch is AWESOME",
"nick": "@clintongormley",
"name": "Clinton Gormley",
"date": "2013-06-03",
"rt" : 5,
"loc": {
! "lat": 13.4,
! "lon": 52.5
}
30. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
How to put it into ES?
31. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
PUT /index/type/id
32. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
PUT /index/type/id
where?
33. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
PUT /myapp/type/id
34. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
PUT /myapp/type/id
what?
35. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
PUT /myapp/tweet/id
36. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
PUT /myapp/tweet/id
which?
37. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
PUT /myapp/tweet/1
38. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
PUT /myapp/tweet/1 -d '
{
"tweet": "I think #elasticsearch is AWESOME",
"nick": "@clintongormley",
"name": "Clinton Gormley",
"date": "2013-06-03",
"rt": 5,
"loc": {
! "lat": 13.4,
! "lon": 52.5
}
}
'
39. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
{
"_index": "myapp",
"_type": "tweet",
"_id": "1",
"_version": 1,
"ok": true
}
# 201 CREATED
40. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Get
41. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /myapp/tweet/1
42. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
{
"_index": "myapp",
"_type": "tweet",
"_id": "1",
"_version": 1,
"exists": true,
"_source": { ...OUR TWEET... }
}
# 200 OK
43. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Exists?
44. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
HEAD /myapp/tweet/1
45. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
HEAD /myapp/tweet/1 # 200 OK
46. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
HEAD /myapp/tweet/1 # 200 OK
HEAD /myapp/tweet/2 # 404 Not Found
47. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Update
48. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
PUT /myapp/tweet/1 -d '
{
"tweet": "I know #elasticsearch is AWESOME",
"nick": "@clintongormley",
"name": "Clinton Gormley",
"date": "2013-06-03",
"rt": 5,
"loc": {
! "lat": 13.4,
! "lon": 52.5
}
}
'
49. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
! atomic DELETE & PUT
50. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
{
"_index": "myapp",
"_type": "tweet",
"_id": "1",
"_version": 2,
"ok": true
}
# 200 OK
51. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Delete
52. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
DELETE /myapp/tweet/1
53. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
{
"_index": "myapp",
"_type": "tweet",
"_id": "1",
"_version": 3,
"ok": true,
"found": true
}
# 200 OK
54. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Optimistic
concurrency control
55. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Optimistic
concurrency control
without locking
56. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
PUT /myapp/tweet/1?version=3 -d '
{
...
}
'
# 200 OK
57. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
PUT /myapp/tweet/1?version=2 -d '
{
...
}
'
# 409 Conflict
58. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Update in place
59. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
POST /myapp/tweet/1/_update -d '
{
"script": "ctx._source.count+=1",
"retry_on_conflict": 3
}
'
60. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
POST /myapp/tweet/1/_update -d '
{
"script": "ctx._source.count+=1",
"retry_on_conflict": 3
}
'
GET ! change ! PUT
61. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Cheaper in bulk
62. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Any
datastore
Elasticsearch
Client
Mirror external DB
63. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Any
datastore
Elasticsearch
Client
Standalone
64. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
"Empty" Search
65. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search
66. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search
{
"took" : 2,
}
67. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search
{
"took" : 2,
"timed_out" : false,
}
68. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 10,
"successful" : 10,
"failed" : 0
},
}
70. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search
"hits" : [
{
"_index" : "de",
"_type" : "tweet",
"_id" : "4",
"_source" : { ... },
"_score" : 1.0,
},
...
]
71. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Multi-index
Multi-type
72. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /index/_search
73. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /index/_search
GET /index1,index2/_search
74. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /index/_search
GET /index1,index2/_search
GET /ind*/_search
75. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /index/_search
GET /index1,index2/_search
GET /ind*/_search
GET /index/type/_search
76. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /index/_search
GET /index1,index2/_search
GET /ind*/_search
GET /index/type/_search
GET /index/type1,type2/_search
77. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /index/_search
GET /index1,index2/_search
GET /ind*/_search
GET /index/type/_search
GET /index/type1,type2/_search
GET /index/type*/_search
78. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /index/_search
GET /index1,index2/_search
GET /ind*/_search
GET /index/type/_search
GET /index/type1,type2/_search
GET /index/type*/_search
GET /_all/type*/_search
79. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Pagination
80. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Pagination
size = num of results
81. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Pagination
size = num of results
from = results to skip
82. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search?size=5&from=0
GET /_search?size=5&from=5
GET /_search?size=5&from=10
83. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Search Lite
84. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Search Lite
GET /_search?q=name:john
85. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
+tweet:foo +name:john +date:>2013-05-01
86. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
+tweet:foo +name:john +date:>2013-05-01
→ percent encoding →
87. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
+tweet:foo +name:john +date:>2013-05-01
?q=%2Btweet%3Afoo+%2Bname%3Ajohn+
%2Bdate%3A%3E2013-05-01
→ percent encoding →
88. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search?q=mary
89. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
→ user named "Mary"
→ tweets by "Mary"
→ tweet mentioning "@mary"
GET /_search?q=mary
90. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search?q=_all:mary
→ user named "Mary"
→ tweets by "Mary"
→ tweet mentioning "@mary"
91. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
_all field
string values from
all other fields
92. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search?q=2013
! 12 results
93. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search?q=2013
! 12 results
GET /_search?q=2013-06-03
! 12 results!!
94. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search?q=2013
! 12 results
GET /_search?q=2013-06-03
! 12 results!!
GET /_search?q=date:2013-06-03
! 1 result
95. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search?q=2013
! 12 results
GET /_search?q=2013-06-03
! 12 results!!
GET /_search?q=date:2013-06-03
! 1 result
GET /_search?q=date:2013
! 0 results!!
96. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
datatype differences?
97. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
check "mapping"
(field definitions)
98. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /myapp/tweet/_mapping
100. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
date = type:date
_all = type:string
101. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Exact value vs Full text
102. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Exact value vs
10
4.5
2013-01-01
true
Foo
foo
Full text
103. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Exact value vs
10
4.5
2013-01-01
true
Foo
foo
Full text
The quick
brown fox
jumped
over the
lazy dog
104. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Inverted index
“The quick brown fox jumped over the lazy dog”
“Quick brown foxes leap over lazy dogs in summer”
105. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Inverted index
→ separate words / terms
“The quick brown fox jumped over the lazy dog”
“Quick brown foxes leap over lazy dogs in summer”
106. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Inverted index
→ separate words / terms
The,quick,brown,fox,jumped,over,the,lazy,dog
Quick,brown,foxes,leap,over,lazy,dogs,in,summer
107. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Inverted index
The,quick,brown,fox,jumped,over,the,lazy,dog
Quick,brown,foxes,leap,over,lazy,dogs,in,summer
→ separate words / terms
→ sort unique terms
108. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Inverted index
→ separate words / terms
→ sort unique terms
The,brown,dog,fox,jumped,lazy,over,quick,the
Quick,brown,dogs,foxes,in,lazy,leap,over,summer
109. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Inverted index
→ separate words / terms
→ sort unique terms
→ list docs containing terms
The,brown,dog,fox,jumped,lazy,over,quick,the
Quick,brown,dogs,foxes,in,lazy,leap,over,summer
110. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Term Doc 1 Doc 2
Quick
The
brown
dog
dogs
fox
foxes
in
jumped
lazy
leap
over
quick
summer
the
111. Term Doc 1 Doc 2
Quick
The
brown
dog
dogs
fox
foxes
in
jumped
lazy
leap
over
quick
summer
the
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
q=quick brown
112. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Term Doc 1 Doc 2
Quick
The
brown
dog
dogs
fox
foxes
in
jumped
lazy
leap
over
quick
summer
the
113. Term Doc 1 Doc 2
Quick
The
brown
dog
dogs
fox
foxes
in
jumped
lazy
leap
over
quick
summer
the
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
q=+Quick +foxes
114. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Term Doc 1 Doc 2
Quick
The
brown
dog
dogs
fox
foxes
in
jumped
lazy
leap
over
quick
summer
the
115. Term Doc 1 Doc 2
Quick
The
brown
dog
dogs
fox
foxes
in
jumped
lazy
leap
over
quick
summer
the
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
No matches!
116. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Improving recall
117. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Term Doc 1 Doc 2
Quick
The
brown
dog
dogs
fox
foxes
in
jumped
lazy
leap
over
quick
summer
the
118. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Term Doc 1 Doc 2
brown
dog
dogs
fox
foxes
in
jumped
lazy
leap
over
quick
summer
the
119. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Term Doc 1 Doc 2
brown
dog
dogs
fox
foxes
in
jumped
lazy
leap
over
quick
summer
the
120. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Term Doc 1 Doc 2
brown
dog
fox
in
jumped
lazy
leap
over
quick
summer
the
121. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Term Doc 1 Doc 2
brown
dog
fox
in
jumped
lazy
leap
over
quick
summer
the
122. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Term Doc 1 Doc 2
brown
dog
fox
in
jump
leap
over
quick
summer
the
123. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
normalize terms
124. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Term Doc 1 Doc 2
brown
dog
fox
in
jump
leap
over
quick
summer
the
125. Term Doc 1 Doc 2
brown
dog
fox
in
jump
leap
over
quick
summer
the
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
q=+Quick +foxes
126. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Term Doc 1 Doc 2
brown
dog
fox
in
jump
leap
over
quick
summer
the
127. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
normalize terms
in query too!
128. Term Doc 1 Doc 2
brown
dog
fox
in
jump
leap
over
quick
summer
the
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
q=+Quick +foxes
129. Term Doc 1 Doc 2
brown
dog
fox
in
jump
leap
over
quick
summer
the
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
q=+quick +foxes
130. Term Doc 1 Doc 2
brown
dog
fox
in
jump
leap
over
quick
summer
the
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
q=+quick +fox
131. Term Doc 1 Doc 2
brown
dog
fox
in
jump
leap
over
quick
summer
the
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
132. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
"Analysis"
133. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
"Analysis"
tokenization + normalization
134. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
"Analysers"
tokenizer + token filters
135. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
standard analyzer
"The Quick Brown Fox jumped
over the Lazy Dog!"
136. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
standard analyzer
→ standard tokenizer
"The Quick Brown Fox jumped
over the Lazy Dog!"
137. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
standard analyzer
→ standard tokenizer
The,Quick,Brown,Fox,jumped,
over,the,Lazy,Dog
138. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
standard analyzer
→ standard tokenizer
→ lowercase filter
The,Quick,Brown,Fox,jumped,
over,the,Lazy,Dog
139. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
standard analyzer
→ standard tokenizer
→ lowercase filter
the,quick,brown,fox,jumped,
over,the,lazy,dog
140. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
standard analyzer
→ standard tokenizer
→ lowercase filter
→ stopwords filter
the,quick,brown,fox,jumped,
over,the,lazy,dog
141. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
standard analyzer
→ standard tokenizer
→ lowercase filter
→ stopwords filter
,quick,brown,fox,jumped,
over, ,lazy,dog
142. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
english analyzer
→ standard tokenizer
→ lowercase filter
the,quick,brown,fox,jumped,
over,the,lazy,dog
143. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
english analyzer
→ standard tokenizer
→ lowercase filter
→ english stemmer
the,quick,brown,fox,jumped,
over,the,lazy,dog
144. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
english analyzer
→ standard tokenizer
→ lowercase filter
→ english stemmer
the,quick,brown,fox,jumped,
over,the,lazy,dog
145. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
english analyzer
→ standard tokenizer
→ lowercase filter
→ english stemmer
the,quick,brown,fox,jump,
over,the,lazy,dog
146. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
english analyzer
→ standard tokenizer
→ lowercase filter
→ english stemmer
→ english stopwords
the,quick,brown,fox,jump,
over,the,lazy,dog
147. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
english analyzer
→ standard tokenizer
→ lowercase filter
→ english stemmer
→ english stopwords
,quick,brown,fox,jump,
over, ,lazy,dog
148. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
date = type:date
_all = type:string
149. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
date = exact value
_all = full text
150. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
date = 2013-06-03
_all = 2013,06,03
151. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search?q=2013
! 12 results
152. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search?q=2013
! 12 results
GET /_search?q=2013-06-03
! 12 results
153. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search?q=2013
! 12 results
GET /_search?q=2013 OR 06 OR 03
! 12 results
154. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search?q=2013
! 12 results
GET /_search?q=2013-06-03
! 12 results
GET /_search?q=date:2013-06-03
! 1 result
155. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search?q=2013
! 12 results
GET /_search?q=2013-06-03
! 12 results
GET /_search?q=date:2013-06-03
! 1 result
GET /_search?q=date:2013
! 0 results
156. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Field mapping
157. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Strings: string
Datetimes: date
Whole numbers: byte, short, integer, long
Floats: float, double
Booleans: boolean
Objects: object
Core field types
158. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Strings: string
Datetimes: date
Whole numbers: byte, short, integer, long
Floats: float, double
Booleans: boolean
Objects: object
Also: multi_field, ip, geo_point, geo_shape,
Core field types
159. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
"foo bar" string
"2013-01-01" date
10 byte, short, integer, long
10.0 float, double
true boolean
{ foo: "bar" } object
Dynamic detection
160. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
"foo bar" string
"2013-01-01" date
10 byte, short, integer, long
10.0 float, double
true boolean
{ foo: "bar" } object
["foo","bar"] No special mapping. Any
field can have multi-vals
Dynamic detection
161. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Most important: type
173. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Updating mappings
174. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Can: add new fields
175. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Can: add new fields
PUT /myapp/tweet/_mapping -d '
{
"tweet": {
"properties": {
...
}
}
}
'
176. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Cannot: change fields
177. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Cannot: change fields
DELETE /myapp
178. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Cannot: change fields
PUT /myapp -d '
{
"mappings": {
"tweet": {
"properties": {
...
}
}
}
}
'
179. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Full body search
180. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search -d '
{
"query": {
"match_all": {}
},
"from": 0,
"size": 10
}
'
181. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search -d '
{
"query": {
"match_all": {}
},
"from": 0,
"size": 10
}
'
182. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Query DSL
rich flexible query language
183. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
{
"match": { "tweet": "search" }
}
184. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search -d '{
{
"query": {
"match": { "tweet": "search" }
}
}
185. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Filters vs Queries
186. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Filters vs
exact matching
Queries
full text search
187. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Filters vs
exact matching
binary yes/no
Queries
full text search
relevance scoring
188. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Filters vs
exact matching
binary yes/no
fast
Queries
full text search
relevance scoring
heavier
189. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Filters vs
exact matching
binary yes/no
fast
cacheable
Queries
full text search
relevance scoring
heavier
not cacheable
193. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search -d '
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"term": { "nick": "@mary" }
}
}
}
}
'
Just a filter
194. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search -d '
{
"query": {
"filtered": {
"filter": {
"term": { "nick": "@mary" }
}
}
}
}
'
Just a filter
195. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search -d '
{
"query": {
"filtered": {
"filter": {
"term": { "nick": "@mary" }
}
}
},
"sort": { "date": "desc" }
}
'
User's tweets by date
196. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_search -d '
{
"query": {
"filtered": {
"filter": {
"range": {
"date": {
"gte": "2013-05-01",
"lt": "2013-06-01"
}
}
}
}}}
'
Tweets for last month
197. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_all/tweet/_search -d '
{
"facets": {
"top_tweeters": {
"terms": {
"field": "nick"
}
}
}
}
'
Top tweeters
198. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_all/tweet/_search -d '
{
"facets": {
"top_tweeters": {
"terms": {
"field": "nick"
}
}
},
"query": {
"match": { "tweet": "elasticsearch" }
}
}
'
Top tweeters for query
199. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
GET /_all/tweet/_search -d '
{
"facets": {
"tweets_by_month": {
"date_histogram": {
"field": "date",
"interval": "month"
}
}
}
}
'
Tweets by month
200. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Autocomplete
{ "match": { "name": "joh" }}
"
John Smith
Johnny Depp
Lyndon Johnson
201. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Autocomplete
{ "match": { "name": "joh" }}
"
John Smith
Johnny Depp
Lyndon Johnson
But "joh" doesn't exist in the index
202. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
N-grams == window-on-a-word:
Autocomplete
203. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
N-grams == window-on-a-word:
Length 1: j,o,h,n,s,m,i,t,h
Autocomplete
204. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
N-grams == window-on-a-word:
Length 1: j,o,h,n,s,m,i,t,h
Length 2: jo,oh,hn,sm,mi,it,th
Autocomplete
205. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
N-grams == window-on-a-word:
Length 1: j,o,h,n,s,m,i,t,h
Length 2: jo,oh,hn,sm,mi,it,th
Length 3: joh,ohn,smi,mit,ith
Length 4: john,smit,mith
Autocomplete
206. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
N-grams == window-on-a-word:
Length 1: j,o,h,n,s,m,i,t,h
Length 2: jo,oh,hn,sm,mi,it,th
Length 3: joh,ohn,smi,mit,ith
Length 4: john,smit,mith
Autocomplete
Good for partial word matching
207. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Edge N-grams == anchored N-grams:
Autocomplete
208. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Edge N-grams == anchored N-grams:
j
jo
joh
john
s
sm
smi
smit
smith
Autocomplete
209. Edge N-grams == anchored N-grams:
j
jo
joh
john
s
sm
smi
smit
smith
Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
Autocomplete
Perfect for
autocomplete
220. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
DELETE /myapp
Recreate the index
221. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
PUT /myapp -d '
{
"settings": {
"analysis": {
"analyzer": {...},
"filter": {...}
}
},
}
Recreate the index
222. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
PUT /myapp -d '
{
"settings": {
"analysis": {
"analyzer": {...},
"filter": {...}
}
},
"mappings": {
"tweet": {
"properties": {...}
}
}
}
Recreate the index
223. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
{
"match": {
"name.autocomplete": "john smi"
}
}
Autocomplete query
224. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
{
"match": {
"name.autocomplete": "john smi"
}
}
Autocomplete query
Better: favor whole word matches
226. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
{
"bool": {
"must": [{...},{...}],
"must_not": [{...},{...}],
"should": [{...},{...}]
}
}
Autocomplete query
MUST match
227. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
{
"bool": {
"must": [{...},{...}],
"must_not": [{...},{...}],
"should": [{...},{...}]
}
}
Autocomplete query
MUST NOT match
228. Copyright Elasticsearch 2013. Copying, publishing and/or distributing without written permission is strictly prohibited
{
"bool": {
"must": [{...},{...}],
"must_not": [{...},{...}],
"should": [{...},{...}]
}
}
Autocomplete query
"More relevant" if these match