Building an ETL pipeline for Elasticsearch using Spark

Building an ETL pipeline for
Elasticsearch using Spark
* *
@2014 eXelate Inc. Confidential and Proprietary
Itai Yaffe, Big Data Infrastructure Developer
December 2015

Agenda
• About eXelate
• About the team
• eXelate’s architecture overview
• The need
• The problem
• Why Elasticsearch and how do we use it?
• Loading the data
• Re-designing the loading process
• Additional improvements
• To summarize
* *
©2011 eXelate Inc. Confidential and Proprietary

About eXelate, a Nielsen company
• Founded in 2007
• Acquired by Nielsen on March, 2015
• A leader in the Ad Tech industry
• Provides data and software services through :
• eXchange (2 billion users)
• maX DMP (data management platform)
* *

Our numbers
* *
• ~10 billion events per day
• ~150TB of data per day
• Hybrid cloud infrastructure
• 4 Data Centers
• Amazon Web Services

About the team
• The BDI (Big Data Infrastructure) team is in charge
of shipping, transforming and loading eXelate’s data
into various data stores, making it ready to be
queried efficiently
• For the last year and a half, we’ve been transitioning
our legacy systems to modern, scale-out
infrastructure (Spark, Kafka, etc.)
* *

About me
• Dealing with Big Data challenges for the last 3.5 years,
using :
• Cassandra
• Spark
• Elasticsearch
• And others…
• Joined eXelate on May 2014
• Previously : OpTier, Mamram
• LinkedIn : https://www.linkedin.com/in/itaiy
• Email : itai.yaffe@nielsen.com
* *

eXelate’s architecture overview
* *
Serving
(frontend
servers)
Incoming
HTTP
requests
ETL
ETL
ETL
DMP
applications
(SaaS)
DB
DWH

The need
* *

The need
• From the data perspective :
• ETL – collect raw data and load it into
Elasticsearch periodically
• Tens of millions of events per day
• Data is already labeled
• Query - allow ad hoc calculations based on the
stored data
• Mainly counting unique users related to a specific
campaign in conjunction with
geographic/demographic data limited by date range
• The number of permutations is huge, so real-time
queries are a must! (and can’t be pre-calculated)
* *

The problem
• We chose Elasticsearch as the data store (details to
follow)
• But… the ETL process was far from optimal
• Also affected query performance
* *

Why Elasticsearch?
• Originally designed as a text search engine
• Today it has advanced real-time analytics
capabilities
• Distributed, scalable and highly available
* *

How do we use Elasticsearch?
• We rely heavily on its counting capabilities
• Splitting the data into separate indices based on a
few criteria (e.g TTL, tags VS segments)
• Each user (i.e device) is stored as a document with
many nested document
* *

How do we use Elasticsearch?
* *
{
"_index": "sample",
"_type": "user",
"_id": "0c31644ad41e32c819be29ba16e14300",
"_version": 4,
"_score": 1,
"_source": {
"events": [
{
"event_time": "2014-01-18",
"segments": [
{
"segment": "female"
}
,{
"segment": "Airplane tickets"
}
]
},
{
"event_time": "2014-02-19",
"segments": [
{
"segment": "female"
}
,{
"segment": "Hotel reservations"
}
]
}
]
}
}

Loading the data
* *

Standalone Java loader application
• Runs every few minutes
• Parses the log files
• For each user we encountered :
• Queries Elasticsearch to get the user’s document
• Merges the new data into the document on the
client-side
• Bulk-indexes documents into Elasticsearch
* *

OK, so what’s the problem?
• Multiple updates per user per day
• Updates in Elasticsearch are expensive (basically delete +
insert)
• Merges are done on the client-side
• Involves redundant queries
• Leads to degradation of query performance
• Not scalable or high available
* *

Re-designing the loading process
• Batch processing once a day during off-hours
• Daily dedup leads to ~75% less update operations in
Elasticsearch
• Using Spark as our processing framework
• Distributed, scalable and highly available
• Unified framework for batch, streaming, machine
learning, etc.
• Using update script
• Merges are done on the server-side
* *

Elasticsearch update script
* *
import groovy.json.JsonSlurper;
added=false;
def slurper = new JsonSlurper();
def result = slurper.parseText(param1);
ctx._ttl = ttl;
ctx._source.events.each() {
item->if (item.event_time == result[0].event_time) {
def segmentMap = [:];
item.segments.each() {
segmentMap.put(it.segment,it.segment)
};
result[0].segments.each{
if(!segmentMap[it.segment]){
item.segments += it
}
};
added=true;
}
};
if(!added) {
ctx._source.events += result
}

Re-designing the loading process
* *
AWS S3
AWS Data Pipeline
AWS EMR
AWS SNS

Zoom-in
• Log files are compressed (.gz) CSVs
• Once a day :
• Files are copied and uncompressed into the EMR cluster
using S3DistCp
• The Spark application :
• Groups events by user and build JSON documents,
which include an inline udpate script
• Writes the JSON documents back to S3
• The Scala application reads the documents from S3 and
bulk-indexes them into Elasticsearch
• Notifications are sent via SNS
* *

We discovered it wasn’t enough…
• Redundant moving parts
• Excessive network traffic
• Still not scalable enough
* *

Elasticsearch-Spark plugin-in for the rescue…
* *
AWS S3
AWS Data Pipeline
AWS EMR
Elasticsearch-Spark plug-in
AWS SNS

Deep-dive
• Bulk-indexing directly from Spark using
elasticsearch-hadoop plugin-in for Spark :
// Save created RDD records to a file
documentsRdd.saveAsTextFile(outputPath)
Is now :
// Save created RDD records directly to Elasticsearch
documentsRdd.saveJsonToEs(configData.documentResource,
scala.collection.Map(ConfigurationOptions.ES_MAPPING_ID ->
configData.documentIdFieldName))
• Storing the update script on the server-side (Elasticsearch)
* *

Better…
• Single component for both processing and indexing
• Elastically scalable
• Out-of-the-box error handling and fault-tolerance
• Spark-level (e.g spark.task.maxFailures)
• Plug-in level (e.g
ConfigurationOptions.ES_BATCH_WRITE_RETRY_COUNT/
WAIT)
• Less network traffic (update script is stored in
Elasticsearch)
* *

… But
• Number of deleted documents continually grows
• Also affects query performance
• Elasticsearch itself becomes the bottleneck
• org.elasticsearch.hadoop.EsHadoopException: Could not
write all entries [5/1047872] (maybe ES was
overloaded?). Bailing out...
• [INFO ][index.engine ] [NODE_NAME]
[INDEX_NAME][7] now throttling indexing:
numMergesInFlight=6, maxNumMerges=5
* *

Expunging deleted documents
• Theoretically not a “best practice” but necessary
when doing significant bulk-indexing
• Done through the optimize API
• curl -XPOST
http://localhost:9200/_optimize?only_expunge_deletes
• curl -XPOST
http://localhost:9200/_optimize?max_num_segments=5
• A heavy operation (time, CPU , I/O)
* *

Improving indexing performance
• Set index.refresh_interval to -1
• Set indices.store.throttle.type to none
• Properly set the retry-related configuration
properties (e.g spark.task.maxFailures)
* *

What’s next?
• Further improve indexing performance, e.g :
• Reduce excessive concurrency on Elasticsearch nodes by
limiting Spark’s maximum concurrent tasks
• Bulk-index objects rather than JSON documents to avoid
excessive parsing
• Better monitoring (e.g using Spark Accumulators)
* *

To summarize
• We use :
• S3 to store (raw) labeled data
• Spark on EMR to process the data
• Elasticsearch-hadoop plug-in for bulk-indexing
• Data Pipeline to manage the flow
• Elasticsearch for real-time analytics
* *

To summarize
• Updates are expensive – consider daily dedup
• Avoid excessive querying and network traffic -
perform merges on the server-side
• Use an update script
• Store it on your Elasticsearch cluster
• Make sure your loading process is scalable and
fault-tolerant – use Spark
• Reduce # of moving parts
• Index the data directly using elasticsearch-hadoop plug-in
* *

To summarize
• Improve indexing performance – properly configure
your cluster before indexing
• Avoid excessive disk usage – optimize your indices
• Can also help query performance
• Making the processing phase elastically scalable (i.e
using Spark) doesn’t mean the whole ETL flow is
elastically scalable
• Elasticsearch becomes the new bottleneck…
* *

Questions?
Also - we’re hiring!
http://exelate.com/about-us/careers/
•DevOps team leader
•Senior frontend developers
•Senior Java developers
* *

Thank you
Itai Yaffe

Keep an eye on…
• S3 limitations :
• The penalty involved in moving files
• File partitioning and hash prefixes
* *

Building an ETL pipeline for Elasticsearch using Spark

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Building an ETL pipeline for Elasticsearch using Spark

Similar to Building an ETL pipeline for Elasticsearch using Spark (20)

More from Itai Yaffe

More from Itai Yaffe (20)

Recently uploaded

Recently uploaded (20)

Building an ETL pipeline for Elasticsearch using Spark