第16回elasticsearch勉強会の発表資料です https://elasticsearch.doorkeeper.jp/events/46539

1. 16 elasticsearch
SPEEDA
Elasticsearch
2016/6/27

2. • / @tau3000
• SPEEDA
SPEEDA
•
• : Ruby, Java

5. :
: 2008 4 1
: 170 ( )

6. B2B
SPEEDA 550
300 / 100
M&A 10
B2C
NewsPicks
NewsPicks
2

9. 300 / 550
M&A
2009 6
600 ( )

10. DEMO

13. SPEEDA Elasticsearch
1. Elasticsearch
2.
3. NewsPicks SPEEDA
4.

14. SPEEDA Elasticsearch
1. Elasticsearch
2.
3. NewsPicks SPEEDA
4.

15. SPEEDA

16. SPEEDA
•
•
•
•

18. SPEEDA
•
•
•
•
•
•
•
•

19. •
•
1000
1000
motor

20. in MySQL
• ID × ID × ID →
• 300 × 2 × 60 ( )
• 6
• 6 ( )
7 40

21. in MySQL
•
• LIKE
•

22. • 10 × 100
MySQL+
• 10
& 5

23. • ( CTO )
• KVS
• Elasticsearch
• Elasticsearch
•
•

24. Elasticsearch

25. in Elasticsearch
• 1 =1 (=300 )
• 1
•
•
• + 6 ( 40 )

26. • 1 40MB (JSON )
• 11 (= )
• × +
•
•

27. •

28. • 10 24
• 1 CPU16
128GB SSD RAID
• 30
• 2

29. DEMO

30. •
•
1000
1000
"motor"

31. SPEEDA Elasticsearch
1. Elasticsearch
2.
3. NewsPicks SPEEDA
4.

32. 1
(precision) (recall)
(kuromoji_tokenizer )
ngram
(cjk_bigram filter )

33. mapping
• bigram,
• unigram
analyzer:
name_analyzer:
tokenizer: standard
filter:
- standard
- cjk_bigram # " " " " " " -> " " " "

34. • phrase_prefix
• max_expansions 1024
1

35. "petro china"
"query": {
"bool": {
"must": [ {
"multi_match": {
"query": "petro",
"type": "phrase_prefix",
"fields": [ "japanese_name", "english_name" ],
"max_expansions": 1024
}
},
{
"multi_match": {
"query": "china",
"type": "phrase_prefix",
"fields": [ "japanese_name", "english_name" ],
"max_expansions": 1024
}
} ]
}
}

36. phrase_prefix
(AP)
(ES)
(ES)
phrase_prefix
AND
petro china
petro
china
petro
china
NTT
China Petrotech Holdings
China Petrochemical
Development

37. max_expansions
• term
"multi_match": {
"query": " ",
"type": "phrase_prefix",
"fields": [ "japanese_name", "english_name" ],
"max_expansions": 1024
}
1. bigram 1024
…
2. bigram
…

38. max_expansions
•
• max_expansions>=131
• bigram X 131
• e 1024
• e term eXXX 1024
1

39. SPEEDA Elasticsearch
1. Elasticsearch
2.
3. NewsPicks SPEEDA
4.

40. NewsPicks
• SPEEDA β 2016/6/20 web
•

41. DEMO

43. 1. ID ES RDB
2. ID ID
ES

44. ID
1.
2.
3.
4.
5.
6.
Industry Index (ES)
Country Table (MySQL)
Account Title Table (MySQL)

45. 1.
2.
3.
4.
5.
6.
ID
Industry Index (ES)
Country Table (MySQL)
Account Title Table (MySQL)

46. ID
1.
2.
3.
4.
5.
6.
Industry Index (ES)
Country Table (MySQL)
Account Title Table (MySQL)

47. ID
1.
2.
3.
4.
5.
6.
Industry Index (ES)
Country Table (MySQL)
Account Title Table (MySQL)

48. ID
1.
2.
3.
4.
5.
6.
Industry Index (ES)
Country Table (MySQL)
Account Title Table (MySQL)

49. ID
1.
2.
3.
4.
5.
6.
Industry Index (ES)
Country Table (MySQL)
Account Title Table (MySQL)

50. ID
JPN( ) AND IDST014( ) AND AT00102( )
Media Index (ES)
"query": {
"bool": {
"must": [
"term": { "countries": "JPN" },
"term": { "industries": "IDST014" },
"term": { "accountTitles": "AT00102" }
]
}
}

51. 1. RDB
2. ES

52. SPEEDA Elasticsearch
1. Elasticsearch
2.
3. NewsPicks SPEEDA
4.

53. AP Server
Node 01
Node 02
Node 20
Batch Server

54. SPEEDA
• AP
•
• 1
40MB

55. Elasticsearch
(ES 50 300 )
• Full GC
• Full GC
• Full GC

56. AP Server
Node 01
Node 02
Node 20
Batch Server

57. AP Server
Master Node 01
Master Node 02
Client Node 01
Client Node 02
Data Node 01
Data Node 02
Data Node 20
Batch Server

58. • Full GC
• Full GC
Full GC
•

59. •
• Full GC
•
0.5 7.2% 2.7%
1.0 2.6% 0.9%
1.5 1.0% 0.5%
2.0 0.7% 0.4%

60. • Elasticsearch
• ES RDB
•
•
• Elasticsearch (1.4→2.3)
inner hits