Evolving your analytics from batch processing to real-time processing can have a major impact on your business. However, designing and maintaining applications to continuously collect and analyze large amounts of data per hour can be extremely difficult without the right tools. Using Amazon Kinesis you can easily ingest streaming data without building any complex pipelines. In this webinar, we will show you how to build your first real-time processing application on AWS, using a combination of managed services like Amazon Kinesis and open source tools such as Apache Spark and Zeppelin. First, you will learn how to push randomly generated event sensor data into Kinesis. Once the data is available in Kinesis we are going to consume and process it using Spark streaming. Finally, we will build data visualizations with this data, using Zeppelin, an open source GUI tool which creates interactive notebooks for data explorations using Spark and can be easily installed on Amazon EMR with a simple click.
Learning Objectives:
• Understand the basics of ingesting streaming data with Amazon Kinesis Streams
• Create a real-time ingestion application
• Work with Apache Zeppelin and Spark on Amazon EMR to processing Kinesis Streams
Who Should Attend:
• Developers, Data Analysts, Data Engineers, Data Scientists and Architects
2. Agenda
1. Introduction to Amazon EMR and Kinesis
2. Real Time Processing Application Demo
3. Best Practices
3. Amazon EMR?
Easy to Use
Launch a cluster in minutes
Low Cost
Pay an hourly rate
Elastic
Easily add or remove capacity
Reliable
Spend less time monitoring
Secure
Manage firewalls
Flexible
Customize the cluster
5. Options to submit jobs to EMR
Amazon EMR
Step API
Submit a Spark
application
Amazon EMR
Airflow, Luigi, or other
schedulers on EC2
Create pipeline
to schedule job
submission or create
complex workflows
Use AWS Lambda to
submit applications to
EMR Step API or directly
to Spark on your cluster
AWS
Lambda
AWS Data
Pipeline
6. Many storage layers to choose from
Amazon DynamoDB
Amazon RDS Amazon Kinesis
Amazon Redshift
EMR File
System
(EMRFS)
Amazon S3
Amazon EMR
11. Spark 2.0 – Performance Enhancements
• Second generation Tungsten engine
• Whole-stage code generation to create optimized
bytecode at runtime
• Improvements to Catalyst optimizer for query
performance
• New vectorized Parquet decoder to increase throughput
12. Datasets & DataFrames (Spark 2.0)
Datasets
• Distributed collection of data
• Strong typing, can use Lambda
functions
• Object-oriented operations (similar to
RDD API)
• Optimized encoders for better
performance; minimize serialization
/deserialization overhead
• Compile-time type safety for robust
applications
DataFrames
• Dataset organized into named
columns
• Represented as a Dataset of rows
13. Spark SQL (Spark 2.0)
• SparkSession – replaces the old SQLContext and
HiveContext
• Seamlessly mix SQL with Spark programs
• ANSI SQL Parser and subquery support
• HiveQL compatibility and can directly use tables in
Hive metastore
• Connect through JDBC / ODBC using the Spark Thrift
server
14. Spark 2.0 – Structured Streaming
• Structured Streaming API: extension to the
DataFrame/Dataset API (instead of DStream)
• SparkSession is the new entry point for streaming
• Better merges processing on static and streaming
datasets, abstracting the velocity of the data
15. Configuring Executors – Dynamic Allocation
• Optimal resource utilization
• YARN dynamically adjusts executors based on resource
needs of Spark application
• Spark uses the executor memory and executor cores
settings in the configuration for each executor
• Amazon EMR uses dynamic allocation by default,
and calculates the default executor size to use based on
the instance family of your Core Group
16. Try different configurations to find your optimal architecture.
CPU
c3 family
c4 family
cc1.4xlarge
cc2.8xlarge
Memory
m2 family
r3 family
Disk/IO
d2 family
i2 family
General
m4 family
m3 family
m1 family
Choose your instance types
Batch Machine Spark and Large
process learning interactive HDFS
17. Zeppelin 0.6.1 – New Features
• Shiro Authentication
• Notebook Authorization
Configurations to save notebook in S3
zeppelin-env.sh:
export ZEPPELIN_NOTEBOOK_S3_BUCKET = bucket_name
export ZEPPELIN_NOTEBOOK_S3_USER = username
(optional) export ZEPPELIN_NOTEBOOK_S3_KMS_KEY_ID = kms-key-id
18. Amazon Kinesis
Streams
Build your own custom
applications that
process or analyze
streaming data
Amazon Kinesis
Firehose
Easily load massive
volumes of streaming
data into Amazon S3
and Redshift
Amazon Kinesis
Analytics
Easily analyze data
streams using
standard SQL queries
Amazon Kinesis: Streaming data made easy
Services make it easy to capture, deliver, and process streams on AWS
19. Amazon Kinesis Concepts
Streams: • Ordered sequence of data records
Data Records: • Unit of data stored in a Kinesis stream
Producers: • Anything that puts records into a Kinesis stream
• Kinesis Producer Library (KPL)
Consumers: • End-points that get records from a Kinesis stream &
processes them
• Kinesis Consumer Library (KCL)
20. Amazon Kinesis Streams features…
• Kinesis Client Library in Python, Node.JS,
Ruby…
• PutRecords API, up to 500 records or 5 MB of payload
• Kinesis Producer Library to simplify producer development
• Server-Side Timestamps
• individual max record payload up to 1 MB
• Low end-to-end propagation delay
• Stream Retention defaults to 24 hrs, but can increase to 7 days
22. Real Time Processing Application
+
Kinesis Producer
(KCL)
Amazon Kinesis
Apache Zeppelin
+
Produce Collect Process Analyze
Amazon
EMR
23. Real Time Processing Application – 5 Steps
http://bit.ly/realtime-aws
1. Create Kinesis Stream
2. Create Amazon EMR Cluster
3. Start Kinesis Producer
4. Ad-hoc Analytics
• Analyze data using Apache Zeppelin on
Amazon EMR with Spark Streaming
5. Continuous Streaming
• Spark Streaming application submitted to
Amazon EMR cluster using Step API
Amazon
Kinesis
Amazon
EMR
Amazon
EC2
(Kinesis
Producer)
(Kinesis
Consumer)
24. Real Time Processing Application
1. Create Kinesis Stream:
$ aws kinesis create-stream --stream-name spark-demo --shard-count 2
25. Real Time Processing Application
2. Create Amazon EMR Cluster with Spark and Zeppelin:
$ aws emr create-cluster --release-label emr-5.0.0
--applications Name=Zeppelin Name=Spark Name=Hadoop
--enable-debugging
--ec2-attributes KeyName=test-key-1,AvailabilityZone=us-east-1d
--log-uri s3://kinesis-spark-streaming1/logs
--instance-groups
Name=Master,InstanceGroupType=MASTER,InstanceType=m3.xlarge,InstanceCount=1
Name=Core,InstanceGroupType=CORE,InstanceType=m3.xlarge,InstanceCount=2
Name=Task,InstanceGroupType=TASK,InstanceType=m3.xlarge,InstanceCount=2
--name "kinesis-processor"
26. Real Time Processing Application
3. Start Kinesis Producer (Using Kinesis Producer Library)
a. On a host (e.g. EC2), download JAR and run Kinesis Producer:
$ wget https://s3.amazonaws.com/chayel-emr/KinesisProducer.jar
$ java –jar KinesisProducer.jar
b. Continuous stream of data records will be fed into Kinesis in CSV format:
… device_id,temperature,timestamp …
Git link: https://github.com/manjeetchayel/emr-kpl-demo
27. Real Time Processing Application
Use Zeppelin on Amazon
EMR:
• Configure Spark interpreter to use
org.apache.spark:spark-streaming-
kinesis-asl_2.11:2.0.0 dependency
• Import notebook from
https://raw.githubusercontent.com/m
anjeetchayel/aws-big-data-
blog/master/aws-blog-realtime-
analytics-using-
zeppelin/Spark_Streaming.json
• Run spark code blocks to generate
real-time analytics.
29. Best Practices - Kinesis
• Use Kinesis Producer Library
• Increase capacity utilization with Aggregation
• Setup CloudWatch alarms
• Randomized Partitions key to distribute puts across
Shards
• Deal with unsuccessful records
• Use PutRecords for higher throughput!
• Kinesis Scaling Utility for scaling Streams
30. Best Practices – Amazon EMR + Spark Streaming
• Use Step API to submit long running applications
• Run Spark application in cluster mode
• Use maxResourceAllocation
• Unqiue Spark Application name for KCL DynamoDB
metadata
• Setup CloudWatch alarms
• Use IAM roles