HOW TO SCALE FROM ZERO TO BILLIONS!

Maziyar PANAHI
Big Data engineer / Cloud Architect
ARGOS - NoSQL / Big Data

Université Paris-Sud, LAL

25 November 2015
Engineer at CNRS

ISCPIF
INSTITUT DES SYSTÈMES COMPLEXES DE PARIS ÎLE-DE-FRANCE
UNITÉ CNRS UPS3611 - HTTP://ISCPIF.FR Creative commons, open science, open data, ressources mutualisées

ISCPIF
http://iscpif.fr/services/

ISCPIF
• Core Services
• ROOM RESERVATION

• EVENT ANNOUNCEMENT

• PROJECT HOSTING AND RESIDENCIES

• HIGH PERFORMANCE COMPUTING

• TRAINING SESSIONS

• COMMUNITY EXPLORER

• Open Platforms
• OpenMole

• Gargantext

• Big Data
• Linkrbrain
http://iscpif.fr/services/

ISCPIF
External Collaborators:
ISCPIF Partners:

• Elasticsearch
• MongoDB
• Redis
• RabbitMQ
• Big Data Platform
INSTITUT DES SYSTÈMES COMPLEXES DE PARIS ÎLE-DE-FRANCE (ISCPIF)
HOW TO SCALE FROM ZERO TO BILLIONS!

–Elasticsearch
“You Know, for Search!”

Elasticsearch

Elasticsearch | Real-Time Data

Elasticsearch | Real-Time Advanced Analytics

Elasticsearch | Massively Distributed

Elasticsearch | High Availability

Elasticsearch | Multi-tenancy
Host Index

Elasticsearch | Full-Text Search

Elasticsearch | Document-Oriented & Schema-Free

Elasticsearch | Developer-Friendly, RESTful API
• Single document APIs
• Index API

• Get API

• Delete API

• Update API
• Multi-document APIs
• Multi Get API

• Bulk API

• Bulk UDP API

• Delete By Query API

Elasticsearch | Developer-Friendly, RESTful API
Index
Type
ID
Document

Elasticsearch | Search & Analyze Data in Real Time
Apache 2 Open Source License Build on top of Apache Lucene™

Elasticsearch 2.0 | Installation - Package
1. curl -L -O https://download.elastic.co/elasticsearch/release/org/elasticsearch/distribution/
tar/elasticsearch/2.0.0/elasticsearch-2.0.0.tar.gz

2. tar -xvf elasticsearch-2.0.0.tar.gz

3. cd elasticsearch-2.0.0/bin

4. ./elasticsearch

Elasticsearch 2.0 | Installation - Repositories
echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/
apt/sources.list.d/elasticsearch-2.x.list
Download and install the Public Signing Key:
Repository deﬁnition APT -> /etc/apt/sources.list.d/elasticsearch-2.x.list
sudo apt-get update && sudo apt-get install elasticsearch
wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Install Elasticsearch 2.0:

Elasticsearch 2.0 | Installation - That’s it!
Simply run:
curl 'http://localhost:9200/?pretty'

Elasticsearch 2.0 | Conﬁguration - System
curl localhost:9200/_nodes/process?pretty
• #File descriptors
• Setting it to 32k or even 64k is
recommended

• #Memory settings
• Disable swap

• sudo swapoﬀ -a

• /etc/fstab

Elasticsearch 2.0 | Conﬁguration - System
-> /etc/default/elasticsearch• ES_HEAP_SIZE
• Leave enough for the OS

• Leave enough for the

• Neve ever go over 30.1 GB!!

• I’ll go with half < 30 GB

Elasticsearch 2.0 | Conﬁguration - Elasticsearch
curl localhost:9200/_nodes/process?pretty• #/etc/elasticsearch/elasticsearch.yml
• network:

• host : <MACHINE IP ADDRESS>

• path:
• logs: /var/log/elasticsearch

• data: /var/data/elasticsearch

• cluster:
• name: <NAME OF YOUR CLUSTER>

• node:
• name: <NAME OF YOUR NODE>

Node.name
Cluster.name
mlockall
# Elasticsearch performs poorly when JVM starts swapping: you should ensure that it _never_ swaps.

#Shards
#Replicas
Remember this?
• 1 cluster
• 3 nodes
• 6 shards
• 1 replica

Elasticsearch 2.0 | Shards and Replicas
Why Shards and Replicas?
• ES has built in clustering

• Scaling out index: (shards)

• Parallel work on an index: (shards)

• Increasing availability: (replicas)
• Can change number of replicas anytime!

• Cannot change number of shards after index creation! (must reindex)

Elasticsearch 2.0 | Shards and Replicas
What is Shard?
• You can't actually split an index!

• ES uses Multiple Lucene indexes (AKA SHARDS)

• Simply, a shard is a Lucene index!

• Over head, query hits all shards for scoring

• So they don’t come free!

Elasticsearch 2.0 | How many Shards and Replicas?
• Replicas:
• More replicas = More availability = Longer indexing!

• Shards
• How much data?

• How many queries?

• How complex are those queries?

• How much resources each node has?

• Number of nodes in your cluster

• Don’t know? over allocate few shards. (but not too many, they are not free!)

• Changing number of replicas: easy
curl -XPUT 'localhost:9200/my_index/_settings' -d '
{
"index" : {
"number_of_replicas" : 4
}
}'
• Changing number of shards: must be re-indexed

• For some, not a big deal.

• For some, it is a big deal!

• StackOverﬂow http://stackexchange.com/performance
• Scaling out:

• More shards than the #nodes

• Multiple shards in one node

• 3x nodes - 3x shards - 2x replicas

• Failing 2x nodes = cluster’s still healthy

• Doubling the storage need

• each replica = 1/3 of index size

• Storage is cheap, small price to pay for availability

Elasticsearch 2.0 | Let’s Use It!
• Wikipedia uses Elasticsearch to provide full-text search with highlighted search
snippets, and search-as-you-type and did-you-mean suggestions.

• The Guardian uses Elasticsearch to combine visitor logs with social -network data to
provide real-time feedback to its editors about the public’s response to new articles.

• Stack Overﬂow combines full-text search with geolocation queries and uses more-
like-this to ﬁnd related questions and answers.

• GitHub uses Elasticsearch to query 130 billion lines of code!
full-text search, structured search, analytics, and all three in combination:

Elasticsearch 2.0 | RESTful API with JSON over HTTP
• VERB: GET, POST, PUT, HEAD, or DELETE.
• PROTOCOL: http or https
• HOST: hostname of any node
• PORT: Elasticsearch HTTP service, which defaults to 9200
• PATH: API Endpoint (_count, _cluster/stats, _nodes/stats/jvm, etc.)
• QUERY_STRING: any optional query-string parameters for example ?pretty
• BODY: A JSON-encoded request body (if the request needs one.)
curl -X<VERB> '<PROTOCOL>://<HOST>:<PORT>/<PATH>?<QUERY_STRING>' -d '<BODY>'

Elasticsearch 2.0 | RESTful API with JSON over HTTP
curl -XGET 'http://localhost:9200/_count?pretty' -d '
{
"query": {
"match_all": {}
}
}
'
{
"count" : 0,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
}
}

Elasticsearch 2.0 | Clariﬁcation
Relational DB Databases Tables Rows Columns
Elasticsearch Indices Types Documents Fields
• Index (noun)
• Traditional relational database. It is the place to store related documents.

• Index (verb)
• To index a document is to store a document in an index (noun) so that it can be retrieved
and queried. (Like INSERT in SQL)

• Inverted index
• B-tree index in Relational databases add an index = Elasticsearch and Lucene use a
structure called an inverted index. Both to improve the speed of data retrieval.

Elasticsearch 2.0 | Indexing a document
PUT /cnrs/employee/1
{
"ﬁrst_name" : "John",
"last_name" : "Smith",
"age" : 25,
"about" : "I love to go rock climbing",
"interests": [ "sports", "music" ]
}
• cnrs
• The index name

• employee
• The type name

• /1
• The ID of this particular employee
PUT /cnrs/employee/2
{
"ﬁrst_name" : "Jane",
"age" : 32,
"about" : "I like to collect rock albums",
"interests": [ "music" ]
}

Elasticsearch 2.0 | Retrieving a document
GET /cnrs/employee/1
{
"_index" : "cnrs",
"_type" : "employee",
"_id" : "1",
"_version" : 1,
"found" : true,
"_source" : {
"ﬁrst_name" : "John",
"age" : 25,
"about" : "I love to go rock climbing",
}
}

Elasticsearch 2.0 | Deleting a document
DELETE /cnrs/employee/1

Elasticsearch 2.0 | Search
GET /cnrs/employee/_search
{
"took": 6,
"timed_out": false,
"_shards": { ... },
"hits": {
"total": 2,
"max_score": 1,
"hits": [
{
"_index": "cnrs",
"_type": "employee",
"_id": "1",
"_score": 1,
"_source": {
"ﬁrst_name": "John",
"last_name": "Smith",
"age": 25,
"about": "I love to go rock climbing",
}
},
{
"_index": "cnrs",
"_type": "employee",
"_id": "2",
"_score": 1,
"_source": {
"ﬁrst_name": "Jane",
"age": 32,
"about": "I like to collect rock albums",
}
}
]
}
}
GET /cnrs/employee/_search?q=last_name:Smith
{
...
"hits": {
"total": 2,
"max_score": 0.30685282,
"hits": [
{
...
"_source": {
"age": 25,
}
},
{
...
"_source": {
"age": 32,
}
}
]
}
}

Elasticsearch 2.0 | Search with Query DSL
{
"query" : {
"match" : {
"last_name" : "Smith"
}
}
}
Elasticsearch provides a rich, ﬂexible, query language called the query DSL, which allows us to build much more complicated,
robust queries.

{
"query" : {
"ﬁltered" : {
"ﬁlter" : {
"range" : {
"age" : { "gt" : 30 }
}
},
"query" : {
"match" : {
"last_name" : "smith"
}
}
}
}
}
{
...
"hits": {
"total": 1,
"max_score": 0.30685282,
"hits": [
{
...
"_source": {
"age": 32,
}
}
]
}
}

Elasticsearch 2.0 | Full-text Search
{
"query" : {
"match" : {
"about" : "rock climbing"
}
}
}
{
...
"hits": {
"total": 2,
"max_score": 0.16273327,
"hits": [
{
...
"_score": 0.16273327,
"_source": {
"age": 25,
}
},
{
...
"_score": 0.016878016,
"_source": {
"age": 32,
}
}
]
}
}

Elasticsearch 2.0 | Phrase Search
{
"query" : {
"match_phrase" : {
}
}
}
{
...
"hits": {
"total": 1,
"max_score": 0.23013961,
"hits": [
{
...
"_score": 0.23013961,
"_source": {
"age": 25,
}
}
]
}
}

Elasticsearch 2.0 | Highlighting Searches
{
"query" : {
"match_phrase" : {
}
},
"highlight": {
"ﬁelds" : {
"about" : {}
}
}
}
{
...
"hits": {
"total": 1,
"max_score": 0.23013961,
"hits": [
{
...
"_score": 0.23013961,
"_source": {
"age": 25,
},
"highlight": {
"about": [
"I love to go <em>rock</em> <em>climbing</
em>"
]
}
}
]
}
}
The highlighted fragment from the original text

• Full text queries
• Match Query

• Multi Match Query

• Common Terms Query

• Query String Query

• Simple Query String
Query

• Term level queries
• Term Query

• Terms Query

• Range Query

• Exists Query

• Missing Query

• Preﬁx Query

• Wildcard Query

• Regexp Query

• Fuzzy Query

• Type Query

• Ids Query

• Compound queries
• Constant Score Query

• Bool Query

• Dis Max Query

• Function Score Query

• Boosting Query

• Indices Query

• And Query

• Not Query

• Or Query

• Filtered Query

• Limit Query

• Joining queries
• Nested Query

• Has Child Query

• Has Parent Query

• Geo queries
• GeoShape Query

• Geo Bounding Box
Query

• Geo Distance
Query

• Geo Distance
Range Query

• Geo Polygon
Query

• Geohash Cell
Query

• Specialized
queries
• More Like This
Query

• Template Query

• Script Query

Elasticsearch 2.0 | Aggregations
{
"query": {
"match": {
"last_name": "smith"
}
},
"aggs": {
"all_interests": {
"terms": {
"ﬁeld": "interests"
}
}
}
}
...
"all_interests": {
"buckets": [
{
"key": "music",
"doc_count": 2
},
{
"key": "sports",
"doc_count": 1
}
]
}

{
"aggs" : {
"all_interests" : {
"terms" : { "ﬁeld" : "interests" },
"aggs" : {
"avg_age" : {
"avg" : { "ﬁeld" : "age" }
}
}
}
}
}
...
"all_interests": {
"buckets": [
{
"key": "music",
"doc_count": 2,
"avg_age": {
"value": 28.5
},
} {
"key": "sports",
"doc_count": 1,
"avg_age": {
"value": 25
}
}
]
}

{
"aggs" : {
“my_ip_ranges" : {
"ip_range" : {
"ﬁeld" : "ip",
"ranges" : [
{ "to" : "10.0.0.5" },
{ "from" : "10.0.0.5" }
]
}
}
}
}
{
...
"aggregations": {
"my_ip_ranges": {
"buckets" : [
{
"to": 167772165,
"to_as_string": "10.0.0.5",
"doc_count": 4
},
{
"from": 167772165,
"from_as_string": "10.0.0.5",
"doc_count": 6
}
]
}
}
}

{
"aggs" : {
"articles_over_time" : {
"date_histogram" : {
"ﬁeld" : "date",
"interval" : "month"
}
}
}
}
{
"aggs" : {
"articles_over_time" : {
"date_histogram" : {
"ﬁeld" : "date",
"interval" : "1.5h"
}
}
}
}

• Metrics Aggregations
• Avg Aggregation

• Cardinality Aggregation

• Extended Stats Aggregation

• Geo Bounds Aggregation

• Max Aggregation

• Min Aggregation

• Percentiles Aggregation

• Percentile Ranks Aggregation

• Scripted Metric Aggregation

• Stats Aggregation

• Sum Aggregation

• Top hits Aggregation

• Value Count Aggregation

• Bucket Aggregations
• Children Aggregation

• Date Histogram Aggregation

• Date Range Aggregation

• Filter Aggregation

• Geo Distance Aggregation

• GeoHash grid Aggregation

• Histogram Aggregation

• IPv4 Range Aggregation

• Missing Aggregation

• Nested Aggregation

• Range Aggregation

• Reverse nested Aggregation

• Sampler Aggregation

• Signiﬁcant Terms Aggregation

• Terms Aggregation

Elasticsearch 2.0 | Plugins
• Plugins Types
• Java plugins
• JAR ﬁles

• Must be installed on all nodes in the cluster

• Each node must be restarted

• Site plugins
• Web content: JS, HTML, CSS etc.

• Can be only on one node

• Do not require a restart

• Mixed plugins
• Both JAR ﬁles and web content
to enhance the core Elasticsearch functionality

Elasticsearch 2.0 | Plugins
• API extension Plugins
• Alerting Plugins
• Analysis Plugins
• Discovery Plugins
• Management and Site Plugins
• Mapper Plugins
• Scripting Plugins
• Security Plugins
• Snapshot/Restore Plugins
• Transport Plugins
• Integrations
to enhance the core Elasticsearch functionality

Elasticsearch 2.0 | Plugins - kopf
Web administration tool for Elasticsearch cluster
sudo [/usr/share/elasticsearch/]bin/plugin install [plugin_name]

sudo [/usr/share/elasticsearch/]bin/plugin install lmenezes/elasticsearch-kopf

sudo [/usr/share/elasticsearch/]bin/plugin install lmenezes/elasticsearch-kopf/2.x
open http://localhost:9200/_plugin/kopf

index 5k/s larger document (full tweets)
index 23k/s smaller document (tweet date, text, etc.)
index 67k/s just 140 characters and ID

sudo service elasticsearch stop

sudo service elasticsearch start

Elasticsearch 2.0 | ISCPIF Use Cases
Full-text search and Data Analytics
• Web of Science
• Text Mining and NLP
• Keyword Extractions
• Phrase Occurrence

• Phrase Co-Occurrence

• Keyword Analytics
• Date Histogram

• Signiﬁcant Terms

• N-Grams

• etc.

• Boolean
• Query String
• Aggregation
• Date histogram

• Query cache
• nop!

{
"bool": {
"must": { "match": { "title": "how to make millions" }},
"must_not": { "match": { "tag": "spam" }},
"should": [
{ "match": { "tag": "starred" }},
{ "range": { "date": { "gte": "2014-01-01" }}}
]
}
}
• title ﬁeld matches “how to make
millions”

• not marked as spam

• documents are starred or are from
2014 onward will rank higher

• Documents that match both
conditions will rank even higher

• Filters
• for binary yes/no searches

• for queries on exact values

• Exists
• just the ones with abstract != null

• Query cache
• true!

My bool vs. ﬁlter = ~500ms vs. ~50ms - ~120ms

• Query DSL: Filters
• And Filter

• Bool Filter

• Exists Filter

• Geo Bounding Box Filter

• Geo Distance Filter

• Geo Distance Range Filter

• Geo Polygon Filter

• GeoShape Filter

• Geohash Cell Filter

• Has Child Filter

• Has Parent Filter

• Ids Filter

• Indices Filter

• Limit Filter

• Match All Filter

• Missing Filter

• Nested Filter

• Not Filter

• Or Filter

• Preﬁx Filter

• Query Filter

• Range Filter

• Regexp Filter

• Script Filter

• Term Filter

• Terms Filter

• Type Filter

• Significant Terms Aggregation
• JLH score

• mutual information
• Chi square

• google normalized distance

• Percentage

• scripted
[Yang and Pedersen, "A Comparative Study on Feature Selection in Text Categorization", 1997]
(http://courses.ischool.berkeley.edu/i256/f06/papers/yang97comparative.pdf) for a study on using
significant terms for feature selection for text classification).
"script_heuristic": {

"script": "_subset_freq/(_superset_freq - _subset_freq + 1)"

}

The Elastic Platform | Make Sense of Your Data

Logs!
• Check Nginx access logs between 2015-11-23T10:23:10 and 2015-11-24T21:53:30

• Check Suricata alert >= 2 between 2015-11-23T10:23:10 and 2015-11-24T21:53:30 and with type
of DNS

• Now correlate the results!!!

Logs!

Logs!
• Old School tools
• grep/sed/awk/cut/sort

• Manually analyze the output

• Different formats

• Customized fields and details

• Not centralized

• Modern way (the right way!)
• Define endpoints (input/output)

• Correlate patterns

• Store data (searchable and visualizable)

Logs!
• Symantec Security Information Manager

• Splunk

• HP / Arcsight

• Tripwire

• NetIQ

• Quest Software

• IMB/Q1 Labs

• Novell

• graylog

• ﬂuentd

Logstash | Collect, Enrich & Transport Data
• Process Any Data, From Any Source

• Centralize data processing of all types

• Normalize varying schema and formats

• Quickly extend to custom log formats
• Easily add plugins for custom data sources

• https://github.com/elastic/logstash

input { stdin { } }

ﬁlter {

grok {

match => { "message" => "%{COMBINEDAPACHELOG}" }

}

date {

match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]

}

}

output {

elasticsearch { hosts => ["localhost:9200"] }

stdout { codec => rubydebug }

}

127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/
xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0
{

"message" => "127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/
xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"",

"@timestamp" => "2013-12-11T08:01:45.000Z",

"@version" => "1",

"host" => "cadenza",

"clientip" => "127.0.0.1",

"ident" => "-",

"auth" => "-",

"timestamp" => "11/Dec/2013:00:01:45 -0800",

"verb" => "GET",

"request" => "/xampp/status.php",

"httpversion" => "1.1",

"response" => "200",

"bytes" => "3891",

"referrer" => ""http://cadenza/xampp/navi.php"",

"agent" => ""Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0""

}

An input plugin enables a speciﬁc source of events to be read by Logstash.
• Input Plugins
• beats

• couchdb_changes

• drupal_dblog

• elasticsearch

• exec

• eventlog

• ﬁle
• ganglia
• gelf

• generator

• graphite

• github

• heartbeat

• heroku

• http

• http_poller

• irc

• imap

• jdbc

• jmx

• kafka

• log4j

• lumberjack
• meetup

• pipe

• puppet_facter

• relp

• rss

• rackspace

• rabbitmq
• redis
• salesforce

• snmptrap

• stdin
• sqlite
• s3

• sqs

• stomp

• syslog
• tcp

• twitter
• unix

• udp

• varnishlog

• wmi

• websocket
• xmpp

• zenoss

• zeromq

• Output Plugins
• boundary

• circonus

• csv

• cloudwatch

• datadog

• datadog_metric
s

• email

• elasticsearch
• elasticsearch_j
ava

• exec

• ﬁle
• google_bigquery

• google_cloud_s
torage

• ganglia
• gelf

• graphtastic

• graphite
• hipchat

• http

• irc

• inﬂuxdb

• juggernaut

• jira

• kafka
• lumberjack

• librato

• loggly
• mongodb
• metriccatcher

• nagios
• null

• nagios_nsca

• opentsdb

• pagerduty

• pipe

• riemann

• redmine

• rackspace

• rabbitmq
• redis
• riak

• s3

• sqs

• stomp

• statsd

• solr_http

• sns

• syslog
• stdout
• tcp

• udp

• webhdfs

• websocket
• xmpp

• zabbix

• zeromq
An output plugin sends event data to a particular destination.

Logstash-forwarder: syslog, auth, ufw and nginx
{

"network": {

"servers": [ "10.0.0.2:5000" ],

"timeout": 15,

"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"

},

"files": [

{

"paths": [

"/var/log/syslog",

"/var/log/auth.log"

],

"fields": { "type": "syslog" }

},

{

"paths": [

"/var/log/ufw.log"

],

"fields": {"type": "firewall"}

},

{

"paths": [

"/var/log/nginx/*.log"

],

"exlude":["*.gz", "err*.log", "*.log.*"],

"fields": { "type": "nginx-api" }

}

]

}

Logstash Server: syslog, auth, ufw and nginx
input {
lumberjack {
port => 5000
type => "logs"
ssl_certiﬁcate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
}
}

Logstash Server: syslog, auth, ufw and nginx
filter {

if [type] == "syslog" {

grok {

match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %
{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %
{GREEDYDATA:syslog_message}" }

add_field => [ "received_at", "%{@timestamp}" ]

add_field => [ "received_from", "%{host}" ]

}

syslog_pri { }

date {

match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]

}

}

}

Logstash-forwarder: syslog, auth, ufw and nginx
output {elasticsearch {
host => "10.0.0.25"
port => "9300"
cluster => "iscpif-es"}
}

Logstash: Suricata | Open Source IDS / IPS / NSM engine
• Highly Scalable
• Suricata is multi threaded

• Protocol Identification
• Suricata a Malware Command and Control Channel
hunter.

• Off port HTTP CnC channels, which normally slide
right by most IDS systems

• Thanks to dedicated keywords you can match on
protocol fields which range from http URI to a SSL
certificate identifier.

• File Identification, MD5 Checksums, and File
Extraction
• Identify thousands of file types while crossing your
network!

Suricata.yml
- eve-log:
enabled: yes
type: file #file|syslog|unix_dgram|unix_stream
filename: eve.json
types:
- alert
- http:
extended: yes
- dns
- tls:
extended: yes # enable this for extended logging information
- files:
force-magic: yes # force logging magic on all logged files
force-md5: yes # force logging of md5 checksums
- ssh

Logstash-forwarder: Suricata
{

# The network section covers network configuration :)

"network": {

"servers": [ "10.0.0.2:5000" ],

"timeout": 15,

"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"

},

"files": [{

"paths": ["/var/log/suricata/eve.json"],

"fields": { "type": "suricata" },

"sincedb_path": "/var/logstash/suricata.db",

"sincedb_write_interval": 1,

"codec": "json",

"type":"suricata"

}]

}

Logstash server: Suricata
filter {

if [type] == "suricata" {

json{

source => "message"

}

if [src_ip] {

geoip {

source => "src_ip"

target => "geoip"

database => "/etc/logstash/GeoLiteCity.dat"

add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]

add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]

}

mutate {

convert => [ "[geoip][coordinates]", "float" ]

remove_field => [ "timestamp" ]

}

}

}

}

Logstash: Suricata

Logstash: Suricata
• Logstash daily index
• index template

• easy to retire index

• close/delete

• 22 machines
• only 2 with public IP

• Logs
• between 1-3 millions /day

Logstash: Suricata
• 73 million docs < 2days!
• during Mongodump

• transferring remotely

• Watch out for Suricata
• Stream events!

• SURICATA STREAM Packet with
invalid ack

• And lots of other stream alerts!

• I disabled it! Maybe I am wrong :)

Kibana | Explore & Visualize Your Data
• Seamless Integration with Elasticsearch

• Give Shape to Your Data
• Sophisticated Analytics

• Empower More Team Members

• Flexible Interface, Easy to Share

• Easy Setup
• Visualize Data from Many Sources

• Simple Data Export

Kibana 4.2.1 | Compatible with Elasticsearch 2.x

The ELK Stack | Elasticsearch, Logstash and Kibana
Elasticsearch 1.7.3
(2.0.0)
Logstash 1.5.5
(2.0.0)
Kibana 3
(4.2.1)

The ELK Stack | Elasticsearch, Logstash and Kibana

Elasticsearch 2.0 | Elasticsearch Mapping
• Which string fields should be full text fields.

• Which fields contain numbers, dates, or geolocations.

• The format of date values.

• a simple type like string, date, long, double, boolean or ip.

• a type which supports the hierarchical nature of JSON such as object or nested.

• or a specialized type like geo_point, geo_shape, or completion.

• multi-fields
• a string field could be indexed as an analyzed field for full-text search, and as a not_analyzed
field for sorting or aggregations.

• Alternatively, you could index a string field with the standard analyzer, the english analyzer, and
the french analyzer.

Elasticsearch 2.0 | Elasticsearch Mapping
• Dynamic mapping
• Fields and mapping types do not need to be defined before being used.

• Explicit mappings
• You can create mapping types and field mappings when you create an index

• Updating existing mappings

• Existing type and field mappings cannot be updated

• Create a new index with the correct mappings and reindex your data

Elasticsearch 2.0 | Mapping Explicit
PUT my_index

{

"mappings": {

"user": {

"_all": { "enabled": false },

"properties": {

"title": { "type": "string" },

"name": { "type": "string" },

"age": { "type": "integer" }

}

},

"blogpost": {

"properties": {

"title": { "type": "string" },

"body": { "type": "string" },

"user_id": {

"type": "string",

"index": "not_analyzed"

},

"created": {

"type": "date",

"format": "yyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"

}

}

}

}

}

Elasticsearch 2.0 | Mapping Dynamic templates
PUT my_index

{

"mappings": {

"my_type": {

"dynamic_templates": [

{

"integers": {

"match_mapping_type": "long",

"mapping": {

"type": "integer"

}

}

},

{

"strings": {

"match_mapping_type": "string",

"mapping": {

"type": "string",

"ﬁelds": {

"raw": {

"type": "string",

"index": "not_analyzed",

"ignore_above": 256

}

}

}

}

}

]

}

}

}

Elasticsearch 2.0 | Mapping Dynamic templates
PUT my_index

{

"template": "logs-*",

"settings": {

"index.number_of_replicas": "0",

"index.number_of_shards": "3"

},

"mappings": {

"my_type": {

"dynamic_templates": [

{

"integers": {

"match_mapping_type": "long",

"mapping": {

"type": "integer"

}

}

},

{

"strings": {

"match_mapping_type": "string",

"mapping": {

"type": "string",

"ﬁelds": {

"raw": {

"type": "string",

"index": "not_analyzed",

"ignore_above": 256

}

}

}

}

}

]

}

}

}

Elasticsearch 2.0 | Production
• Hardware
• Memory

• 64 GB of RAM is the ideal sweet spot

• 16 GB of RAM for Heap Size and 32 GB total

• And don’t cross 30.5 GB!

• CPU
• 2-8 cores of CPU

• faster CPUs vs. more cores = choose more cores

• Disk
• SSDs (monitor the I/O)

• high-performance server disks (15k RPM)

• RAID 0 (ES is high available by replicas)

Elasticsearch 2.0 | Security
• No built-in authentication

• Do not expose Elasticsearch to the world
• Watch out for Denial of Service

• Do not give users to define index name (like ",*" )

• Turn off Dynamic Scripts (default is off)

• Control protocols (DELETE, PUT, etc.)

• Nginx (reverse proxy, SSL and auth)

• Apache

Elasticsearch 2.0 | Lots of things!
• Analyzer and tokenizer (human language)

• Log slow ops

• Index settings
• refresh interval

• ﬂush interval

• Diﬀerentiate your nodes
• Data nodes

• Master nodes

• Client nodes

• Cluster health
• Heap size

• Thread pools

• Merging time

• etc.

“Launch your
GIANT idea”
MongoDB v3.2 is coming soon. Learn more.

• Document Database
• Documents (i.e. objects)

• Embedded documents and arrays (no expensive joins!)

• Dynamic schema supports ﬂuent polymorphism.

• High Availability
• automatic failover

• data redundancy

• Automatic Scaling
• Automatic sharding
MongoDB | Key Features

MongoDB | Platforms

MongoDB | Install MongoDB on Ubuntu
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0 multiverse" | sudo
tee /etc/apt/sources.list.d/mongodb-org-3.0.list
Import the public key used by the package management system
Create a list ﬁle for MongoDB
sudo apt-get update && sudo apt-get install -y mongodb-org

sudo service mongod start
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10
Install MongoDB 3.0.7

MongoDB | SQL to MongoDB

MongoDB | SQL to MongoDB
https://docs.mongodb.org/manual/reference/sql-comparison/

MongoDB | Insert Documents

MongoDB | Query Documents
db.inventory.find( { type: "snacks" } )
db.inventory.find( {} )
db.inventory.find( { type: { $in: [ 'food', 'snacks' ] } } )
db.inventory.find( { type: 'food', price: { $lt: 9.95 } } )
db.inventory.find(
{
$or: [ { qty: { $gt: 100 } }, { price: { $lt: 9.95 } } ]
}
)
All documents
Equality
Query Operation
AND condition
OR condition

MongoDB | Query Documents
db.tweets.ﬁnd(
{
"coordinates.coordinates":
{ $near :
{
$geometry: { type: "Point", coordinates: [ 2.3325923, 48.8537095] },
$minDistance: 1,
$maxDistance: 500
}
}
}
)

MongoDB | Aggregation
{

"_id": "10280",

"city": "NEW YORK",

"state": "NY",

"pop": 5574,

"loc": [

-74.016323,

40.710537

]

}
db.zipcodes.aggregate( [

{ $group: { _id: "$state", totalPop: { $sum: "$pop" } } },

{ $match: { totalPop: { $gte: 10*1000*1000 } } }

] )
{

"_id" : "AK",

"totalPop" : 550043

}
SELECT state, SUM(pop) AS totalPop

FROM zipcodes

GROUP BY state

HAVING totalPop >= (10*1000*1000)

db.tweets.aggregate([

{

$geoNear: {

near: [ 2.348730, 48.840982 ],

distanceField: "dist.calculated",

maxDistance: 100,

includeLocs: "dist.location",

query: {"coordinates.type": "Point"},

limit: 100

}

},

{ $group: {

_id: "$user.id",

count: { $sum: 1 },

name: { $addToSet: "$user.name" },

date: { $addToSet: "$created_at" },

text: { $addToSet: "$text" },

coordinates: { $addToSet: "$coordinates" }

} },

{$sort: {"count": -1}},

{$limit: 10}

]);
MongoDB | Aggregation

MongoDB | MapReduce

MongoDB | Index
• Creating an index

• db.ships.ensureIndex({name : 1})

• Dropping an index

• db.ships.dropIndex({name : 1})

• Creating a compound index

• db.ships.ensureIndex({name : 1, operator : 1, class : 0})

• Dropping a compound index

• db.ships.dropIndex({name : 1, operator : 1, class : 0})

MongoDB | Replica Set

MongoDB | Replica Set
• Replica Set Members

• Replica Set Primary
• Accepts write operations

• Replica Set Secondary Members
• Replicate the primary’s data set and accept read operations

• Priority 0 Replica Set Members
• Priority 0 members are secondaries that cannot become the primary.

• Hidden Replica Set Members
• Invisible to applications

• Replica Set Arbiter

MongoDB | Shardings

MongoDB | 3.0

MongoDB | 3.0
• 7-10x Better Performance
• Up to 80% Less Storage
• Reduce Operational Overhead By Up to 95%

• Pluggable Storage Optimized For Your Workload

• Low Latency Across the Globe
• Enhancements That Make You More Productive

• Faster Loading and Export
• Easier Query Optimization
• Faster Debugging
• Richer Geospatial Apps
• Better Time-Series Analytics

Datacenter in France and Italy
60M/week - 8M/day - 360K/hour
MongoDB | ISCPIF Use Cases

Avg. usage of Twitter in Paris in October

24HOURS NEWS: Real-time Breaking News

24HOURS NEWS: Real-time Breaking News
• 50-100 Updates /s

• Time-series Queries

• Grid-FS

• FTS (full-text search)

• Tokenizes and stems

• Scoring

• 140 characters/small dataset!

• TTL Index

• Session store

TIMOTHY: Real-time Dashboards

HIGH THROUGHPUT: Real-time aggregations, over 50K inserts /s

Most generic
Most speciﬁc Calculating the new graph
HIGH THROUGHPUT: Real-time aggregations, over 50K inserts /s

130K
inserts /s

10K-80K
inserts /s

9K queries /s

Gephi Streaming

MongoDB | Monitoring System

MongoDB | Monitoring System Network and Cache

MongoDB | Monitoring System Hardware

Monitoring NewRelic

MongoDB | Monitoring System Objects
3.02 Billion Documents
64 Collections
195 Indexes

MongoDB | Monitoring System Objects

“in-memory data structure store”
used as database, cache and message broker

• Data structures
• strings
• hashes
• lists
• sets
• sorted sets
• bitmaps

• hyperlogs
• geospatial indexes

• Built-in
• replication

• Lua scripting

• transactions

• on-disk persistence
Redis | Key Features

Redis KEYS:
> set mykey somevalue
OK

> get mykey
“somevalue”
Redis LISTS:
> rpush mylist A
(integer) 1

> rpush mylist B
(integer) 2

> lpush mylist ﬁrst
(integer) 3
> lrange mylist 0 -1
1) "ﬁrst"
2) "A"
3) "B"
Redis | Data Structures just a little!

maziyar-beautiful-MacBook$ redis-benchmark -q -n 100000
PING_INLINE: 85178.88 requests per second

PING_BULK: 80000.00 requests per second

SET: 86580.09 requests per second

GET: 83263.95 requests per second

INCR: 83963.05 requests per second

LPUSH: 86880.97 requests per second

LPOP: 90252.70 requests per second

SADD: 84388.19 requests per second

SPOP: 92936.80 requests per second

LPUSH (needed to benchmark LRANGE): 87336.24 requests per second

LRANGE_100 (ﬁrst 100 elements): 25614.75 requests per second




MSET (10 keys): 50000.00 requests per second
Redis | How fast is Redis?

maziyar-beautiful-MacBook$ redis-benchmark -n 1000000 -t set,get -P 16 -q
PING_INLINE: 735294.12 requests per second

PING_BULK: 988142.31 requests per second

SET: 681198.88 requests per second

GET: 831255.25 requests per second

INCR: 778210.12 requests per second

LPUSH: 682593.81 requests per second

LPOP: 713775.88 requests per second

SADD: 732600.75 requests per second

SPOP: 885739.62 requests per second

LPUSH (needed to benchmark LRANGE): 656598.81 requests per second
Redis | How fast is Redis? Pipelining of 16 commands

Redis | ISCPIF Use Cases
Scientiﬁc Games

• Scientiﬁc Operations
• Occurrence

• Co-Occurrence

• Scientiﬁc Games
• Pub/Sub
• Rate Limiter (IP-based with TTL)
• Chat rooms
• TTL

Monitoring NewRelic

“Messaging that just works”
well actually it’s more than that, but OK!

RabbitMQ | Feature List
A messaging broker
• Highlights

• Reliability

• Flexible Routing

• Clustering

• Federation

• Highly Available Queues

• Multi-protocol

• Many Clients

• Management UI

• Plugin System

• For what?
• Data delivery

• Non-blocking operations

• Push notiﬁcations

• Publish / subscribe

• Asynchronous processing (work queues)

RabbitMQ | Feature List
A messaging broker
Type
Topic
Q1
Q2
Q3
climate.*
risk.*
news.*
RabbitMQ Routing

RabbitMQ | ISCPIF Use Cases
Monitoring NewRelic

Monitoring RabbitMQ Management UI

• Distributed Computations
• Parsing text ﬁles

• Scientiﬁc calculations

• Realtime Processing

• Text mining

• NLP

• Annotation

• Keyword extractions

• Job Queues
• RPC (Remote procedure call)
• Topic based routing

• Parsing
• 225 ﬁle

• 10m-20m lines

• Avg. total of 3.3 Billions

• RPC
• Post-process each document

• Output

• MongoDB

• ElasticSearch

• Redis

ISCPIF Big Data
• Multivac (Open Data Platform)
• ISCPIF APIs
• Science en Poche
• Climatique (COP21)
• Risk (AXA research fund)
• Scientiﬁc Dashboards
• Distributed Computing
• Nobel Game (scientiﬁc game)
• Twitter streaming (UN, France, Climate Change, Risk, etc)
• Instagram streaming (Paris)

Twitter
Storing
Data
Real-time
Streaming
System
Data
Analytics
Real-time
Processing
Web
Mobile
Wearable
Devices
Text
Mining
Sensor-based
devices
Mobile
Devices
Wearable
Devices
Instagram
Foursquare
Data Streams Real-time Streaming System
Web Socket
XMLJSON
Authorization
Authentication
Identiﬁcation
Flash Socket
xhr-polling
jsonp-polling
Backend
Architecture
Facebook
Files
End User
Indexing
Data
RPC
System
NLP
Annotation
Extraction
Streaming
Data
Crowd
Sourcing
Real-Time Data Stream Processing

• Multivac (Open Data)
• ISCPIF APIs
• Scientiﬁc Dashboards
• Science en Poche
• Distributed Computing
• Climatique (COP21)
• Risk (AXA research fund)
• Nobel Game (scientiﬁc game)
• Twitter streaming (UN,
France, Climate Change, Risk)
• Instagram streaming (Paris)
Python
Scala
Script
Java
Erlang
iOS
Node
JS
current projects

–just a regular Geek :)
“You are your best benchmark!”

SHOWCASE

FRANCE 2014
Real-time processing and visualizing Twitter in France

News Tracking
Real-time tracking news with highest impact of networks

Aviation Accidents
50K retweets/10min

Aviation Accidents
120K retweets/10min

Robin Williams
180K retweets/10min

#Ferguson michealBROWN
75K retweets/10min

Paris
13 Novembre

22 VMs
Distributed Systems
320K Ops /S
130K Insert /S
64K Index /S
…
8 WEB SERVERS
4 API SERVERS
Search Engine Cluster
900 million data
45%
Database
2.9 billion data
22%
120 Cores
2.5TB RAM
30 TB SSD
+8000 Lines Code
14 Web Apps
4 Mobile Apps

• Starting 2016 with CouchBase in parallel

• Graph Databases

• Spark Streaming / machine learning

• Clustering and categorizing in real-time

• Creative Hardware
• SlipStream, StratusLab and EGI Cloud
• Healthcare and Wearable devices

• Non, no drones! ;-)
What’s next for Big Data at ISCPIF

–The Blacklist: Lord Baltimore (No. 104)
“Every piece of information is worth something to somebody. And
in the hands of the wrong person, that could be deadly.”
Reddington: People love to decry big brother the NSA, the government listening in on
their most private lives, yet they all willingly go online and hand over the most intimate
details of those lives - to big data.
Elizabeth: Most people don't care that Google knows their search history.
Reddington: They know more than that. They know your habits, the banks you use, the
pills you pop, the men or women you sleep with.”

Thanks!
maziyar.panahi@iscpif.fr
25 November 2015
http://iscpif.fr/maziyar

HOW TO SCALE FROM ZERO TO BILLIONS!

Recommended

Recommended

More Related Content

What's hot

What's hot (6)

Similar to HOW TO SCALE FROM ZERO TO BILLIONS!

Similar to HOW TO SCALE FROM ZERO TO BILLIONS! (20)

Recently uploaded

Recently uploaded (20)

HOW TO SCALE FROM ZERO TO BILLIONS!