We will introduce key concepts for a data lake and present aspects related to its implementation. Also discussing critical success factors, pitfalls to avoid operational aspects, and insights on how AWS enables a server-less data lake architecture. Speaker: Sebastien Menant, Solutions Architect, Amazon Web Services

1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Sebastien Menant & Nam Je Cho, Enterprise Solutions Architects
Amazon Web Services
Building a Server-­less Data Lake on AWS
Technical 301

2. Agenda
• What is a Data Lake?
• Why You Need a Data Lake
• Building the Data Lake
• Demo
• Next Steps

3. What is a Data Lake?

4. Definition
“A data lake provides massive storage for
any kind of data, enormous processing
power and the ability to handle virtually
limitless concurrent tasks or jobs”
-­ Wikipedia

5. Characteristics of a Data Lake
Collect
Everything
Dive in
Anywhere
Flexible
Access

6. Why You Need a Data Lake

8. What About Modern Business Needs?

9. Big Data… and The Hadoop Ecosystem

10. But Both are Complementary
Amazon
EMR
Amazon
Redshift
But Both are Complementary

12. STORAGE
COMPUTE
COMPUTE
COMPUTE
COMPUTE
COMPUTE
COMPUTE
COMPUTE
COMPUTE
COMPUTE
Amazon
EMR
Amazon S3

13. New Business Outcomes and Capabilities
• Enable New Insights in Your Data
• Cost Savings of Compute and Storage
• Use the Right Tool for the Job
• Increase Durability of Data
• Charge Storage Costs to Owner
• Streaming and Real-­time Analysis
Retain all your data, for years!

14. Building the Data Lake

15. Beware

16. Building Blocks of the Data Lake
Storage and Ingestion
Catalogue and Search
Security
API and UI

17. Storage and Ingestion
Storage and
Ingestion
Catalogue and
Search
Security
API and UI

18. Requirements for Storage
• Multi-­year Scalable Storage Capability
• High Durability
• Store Raw Data from Any Input Sources
• Support for Any Data Type
• Low Cost

19. Amazon S3
1. Highly Scalable and Durable
2. Security and Encryption
3. Lifecycle Management
4. Event Notifications
5. Versioning
Key Services for Storage
Amazon Glacier
1. Long-­term Archival Storage
2. Lifecycle Integration with S3
3. Extremely Low-­cost
4. Vault Lock
Amazon
S3
Amazon
Glacier

20. Amazon
S3
Amazon
Glacier
Storage
and
Ingestion

21. Recommendations #1
• S3 Buckets
• Close to Users and Compute
• Select Region for Regulatory Compliance
• Naming
• Human-­readable Path
• Random Hash Prefix for Optimal Partitioning
• Format
• Structured vs Unstructured + Compression
• CSV, Parquet, ORC, JSON, XML, logs, etc
• GZIP for small files, Avro, LZO, Snappy

22. Recommendations #2
• Optimise
• Store Everything
• Use Large Files with Split-­able Format
• Lifecycle Policies for Cost-­savings
• Tagging for Cost Allocation
• Security
• Encryption
• Bucket Policies, ACL, Tagging, CloudTrail

23. Requirements for Ingestion
• Batch File Support
• Traditional ETL
• Streaming Data
• Consumption of any Dataset as a Stream
• Low Latency Analytics
• Replay-­ability from the Data Lake
• Server-­less ETL Capabilities

24. Amazon Kinesis Firehose
1. Easy to use with Agent
2. Automatic Elasticity
3. Near Real-­time
4. Simultaneous Destinations
Key Services for Ingestion
Amazon Kinesis Streams
1. Enables Custom Processing
2. Continuous Data Collection
3. Real-­time
4. API Driven for Custom Apps
Amazon
Kinesis
Streams
Amazon
Kinesis
Firehose

25. Data
Sources
Data
Sources
Data
Sources
Data
Sources
Data
Sources
S3
DynamoDB
Redshift
Amazon Kinesis
Availability
Zone
Availability
Zone
Availability
Zone
Stream
AWS Lambda
KCL App
EMR
Elasticsearch

26. Amazon
Glacier
Amazon
Kinesis
Storage
and
Ingestion
Amazon
S3

27. Recommendations
• Reminder
• Added Complexity needs Business Justification
• Select the Right Tools
• Real-­time Analysis: Apache Spark Streaming, Storm, Flink
• Firehose to Redshift for BI and Dashboards
• Tips
• AWS Lambda for ETL Transformation
• Persist Streams into S3

28. http://amzn.to/23DWr5O

29. http://amzn.to/1SRk8wG

30. Catalogue and Search
Storage and
Ingestion
Catalogue and
Search
Security
API and UI

31. Requirements for Catalogue and Search
• Metadata Index
• Automated Metadata Processing
• Discovery and Search
• Data Classification
• Server-­less and Event-­driven

32. Key Services for Catalogue and Search
1. Server-­less
2. Event Driven
3. Auto Scaling
4. Real-­time
1. NoSQL
2. Streams
3. Logstash Plugin
1. Deploy Simply
2. Easy Admin
3. Kibana
Amazon
Elasticsearch
Service
Amazon
DynamoDB
AWS
Lambda
Lambda DynamoDB Elasticsearch

33. Catalogue and Search
AWS
Lambda
Amazon
DynamoDB
Amazon
Elasticsearch

34. Recommendations
• Tips
• Start Small and Simple… add Capabilities
• File names, size, state, dates, tags, owner
• Region, versions, lineage, relationships
• Search Metadata and Object Content
• Events
• S3 Triggers Lambda
• DynamoDB Streams
• Logstash Plugin to Elasticsearch

35. http://amzn.to/23E9LUp

36. http://amzn.to/1TQVBwp

37. Security
Storage and
Ingestion
Catalogue and
Search
Security
API and UI

38. Requirements for Security
• Data Encryption at Rest
• Authentication
• Authorisation

39. AWS IAM
1. Users and Roles
2. Identity Federation
3. Multi Factor Authentication
4. Granular Permissions
Key Services for Security
AWS KMS
1. Seamless Service Integration
2. Extensive Compliance
AWS
IAM
AWS
KMS
AWS
CloudHSM
SSE-­S3

40. Security
AWS
KMS
AWS
IAM

41. Recommendations
• Start Early
• Security Needs Practice!
• Federate with your Corporate Directory
• Best Practice
• Use CloudTrail and CloudWatch
• Encrypt Where Possible
• Select Bucket Region for Regulatory Compliance
• Tips
• IAM Policies, S3 Versioning and MFA Delete
• Lambda for Data Masking

42. API and UI
Storage and
Ingestion
Catalogue and
Search
Security
API and UI

43. Requirements for API and UI
• Serve Data and Capabilities to Customers
• Programmatically
• Search Catalogue
• Run Compute
• Extend Access Control Management
• And… Use of Familiar Visualisation Tools

44. Amazon API Gateway
1. Performance at Any Scale
2. Create RESTful Frontend
3. Managed API Lifecycle
Key Services for API and UI
AWS Lambda
1. Enables Server-­less API
2. Custom Logic for Services
3. Automatic Scaling
AWS
Lambda
Amazon API
Gateway

45. API
and
UI Amazon
API Gateway
AWS
Lambda

46. Recommendations
• Tips
• Go Server-­less!
• Extend Existing AWS Services and Build Custom Logic
• Data Management, Processing and Transformations
• API Gateway for Data Access
• Serve the Data, Search and Compute via RESTful APIs
• Distribute a Custom SDK
• Extend the Solution
• Build Advanced Security Controls using Metadata Index

47. The Whole Picture…
Storage and
Ingestion
Catalogue and
Search
Security
API and UI
Storage and
Ingestion
Catalogue and
Search
Security
API and UI

48. Amazon
EMR
Amazon
RDS
Amazon
S3
Amazon
Glacier
Amazon
Kinesis
Storage
and
Ingestion
Security
AWS
KMS
AWS
IAM
API
And
UI Amazon
API Gateway
AWS
Lambda USERS
Amazon
Redshift
Catalogue and Search
AWS
Lambda
Amazon
DynamoDB
Amazon
Elasticsearch

49. A Data Lake is…
• Foundation of Data Storage and Streaming Data
• Metadata index to help Categorise and Govern
• Search Index to Enable Data Discovery
• Robust Set of Security Controls
• Governance Through Technology Not Policy
• Interface to Expose Data and Capabilities to Users

50. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
2016-­04-­28
Demo

51. Demo

52. Building Catalogue and Search
ElasticSearch
Metadata
Index
LambdaS3 Bucket Logstash
Data Flow
Data
Source
DynamoDB

53. Next Steps

54. Proof of Concept

55. Next Steps
• How to Get Started
• AWS Documentation
• Getting Started Guide
• AWS Training & Certification
• Big Data on AWS
• AWS Partner Network
• AWS Professional Services
• Big Data Specialists

56. AWS Training & Certification
Intro Videos & Labs
Free videos and labs to
help you learn to work
with 30+ AWS services
– in minutes!
Training Classes
In-­person and online
courses to build
technical skills –
taught by accredited
AWS instructors
Online Labs
Practice working with
AWS services in live
environment –
Learn how related
services work
together
AWS Certification
Validate technical
skills and expertise –
identify qualified IT
talent or show you
are AWS cloud ready
Learn more: aws.amazon.com/training

57. Your Training Next Steps:
ü Visit the AWS Training & Certification pod to discuss your
training plan & AWS Summit training offer
ü Register & attend AWS instructor led training
ü Get Certified
AWS Certified? Visit the AWS Summit Certification Lounge to pick up your swag
Learn more: aws.amazon.com/training

58. Thank You!