大數據運算媒體業案例分享 (Big Data Compute Case Sharing for Media Industry)

4. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. GB TB PB ZB EB Big Data: Unconstrained Growth 95% of the 1.2 zettabytes of data in the digital universe is unstructured 70% of this data is user- generated content Unstructured data growth is explosive Machine data/IoT will only steepen the curve Source: IDC

5. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data Gap 1990 2000 2010 2020 Generated Data Available for Analysis Data Volume Sources: Gartner: User Survey Analysis: Key Trends Shaping the Future of Data Center Infrastructure Through 2011 IDC: Worldwide Business Analytics Software 2012–2016 Forecast and 2011 Vendor Shares

7. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Plethora of Tools Amazon Glacier S3 DynamoDB RDS EMR Amazon Redshift Data PipelineAmazon Kinesis Cassandra CloudSearch Kinesis- enabled app Lambda ML SQS ElastiCache DynamoDB Streams

8. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. A complete platform for big data & analytics Retrospective analysis and reporting Here-and-now real-time processing and dashboards Predictions to enable smart applications Amazon Kinesis Amazon EC2 Amazon Redshift Amazon EMR Amazon ML Amazon EMR

10. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. http://aws.amazon.com/marketplace Big Data Case Studies Learn from other AWS customers aws.amazon.com/solutions/case-studies/big-data

11. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Simplify Big Data Processing ingest / collect store process / analyze consume / visualize Time to Answer (Latency) Throughput Cost

13. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Types of Data • Transactional • Database reads & writes (OLTP) • Cache • Search • Logs • Streams • File • Log files (/var/log) • Log collectors & frameworks • Stream • Log records • Sensors & IoT data Database File Storage Stream Storage A iOS Android Web Apps Logstash LoggingIoTApplications Transactional Data File Data Stream Data Mobile Apps Search Data Search Collect Store LoggingIoT

15. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Stream Storage A iOS Android Web Apps Logstash Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Apache Kafka Amazon Glacier Amazon Kinesis Amazon DynamoDB Amazon ElastiCache SearchSQLNoSQLCacheStreamStorageFileStorage Transactional Data File Data Stream Data Mobile Apps Search Data Database File Storage Search Collect Store LoggingIoTApplications ü

16. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Why Is Amazon S3 Good for Big Data? • Natively supported by big data frameworks (Spark, Hive, Presto, etc.) • No need to run compute clusters for storage (unlike HDFS) • Can run transient Hadoop clusters & Amazon EC2 Spot instances • Multiple distinct (Spark, Hive, Presto) clusters can use the same data • Unlimited number of objects • Very high bandwidth – no aggregate throughput limit • Highly available – can tolerate AZ failure • Designed for 99.999999999% durability • Tired-storage (Standard, IA, Amazon Glacier) via life-cycle policy • Secure – SSL, client/server-side encryption at rest • Low cost

17. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What about HDFS & Amazon Glacier? • Use HDFS for very frequently accessed (hot) data • Use Amazon S3 Standard for frequently accessed data • Use Amazon S3 Standard – IA for infrequently accessed data • Use Amazon Glacier for archiving cold data

18. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Database + Search Tier A iOS Android Web Apps Logstash Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Apache Kafka Amazon Glacier Amazon Kinesis Amazon DynamoDB Amazon ElastiCache SearchSQLNoSQLCacheStreamStorageFileStorage Transactional Data File Data Stream Data Mobile Apps Search Data Collect Store ü

20. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Best Practice — Use the Right Tool for the Job Data Tier Search Amazon Elasticsearch Service Amazon CloudSearch Cache Redis Memcached SQL Amazon Aurora MySQL PostgreSQL Oracle SQL Server NoSQL Cassandra Amazon DynamoDB HBase MongoDB Database + Search Tier

21. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What Data Store Should I Use? • Data structure → Fixed schema, JSON, key-value • Access patterns → Store data in the format you will access it • Data / access characteristics → Hot, warm, cold • Cost → Right cost

22. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data Structure and Access Patterns Access Patterns What to use? Put/Get (Key, Value) Cache, NoSQL Simple relationships → 1:N, M:N NoSQL Cross table joins, transaction, SQL SQL Faceting, Search Search Data Structure What to use? Fixed schema SQL, NoSQL Schema-free (JSON) NoSQL, Search (Key, Value) Cache, NoSQL

23. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Data / Access Characteristics: Hot, Warm, Cold Hot Warm Cold Volume MB–GB GB–TB PB Item size B–KB KB–MB KB–TB Latency ms ms, sec min, hrs Durability Low–High High Very High Request rate Very High High Low Cost/GB $$-$ $-¢¢ ¢ Hot Data Warm Data Cold Data

24. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What Data Store Should I Use? Amazon ElastiCache Amazon DynamoDB Amazon Aurora Amazon Elasticsearch Amazon EMR (HDFS) Amazon S3 Amazon Glacier Average latency ms ms ms, sec ms,sec sec,min,hrs ms,sec,min (~ size) hrs Data volume GB GB–TBs (no limit) GB–TB (64 TB Max) GB–TB GB–PB (~nodes) MB–PB (no limit) GB–PB (no limit) Item size B-KB KB (400 KB max) KB (64 KB) KB (1 MB max) MB-GB KB-GB (5 TB max) GB (40 TB max) Request rate High - Very High Very High (no limit) High High Low – Very High Low – Very High (no limit) Very Low Storage cost GB/month $$ ¢¢ ¢¢ ¢¢ ¢ ¢ ¢/10 Durability Low - Moderate Very High Very High High High Very High Very High Hot Data Warm Data Cold Data Hot Data Warm Data Cold Data

26. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AnalyzeA iOS Android Web Apps Logstash Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Apache Kafka Amazon Glacier Amazon Kinesis Amazon DynamoDB Amazon Redshift Impala Pig Amazon ML Streaming Amazon Kinesis AWS Lambda AmazonElasticMapReduce Amazon ElastiCache SearchSQLNoSQLCache StreamProcessingBatchInteractive Logging StreamStorage IoTApplications FileStorage Hot Cold War m Hot Hot ML Transactional Data File Data Stream Data Mobile Apps Search Data Collect Store Analyze ü ü

27. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Process / Analyze Analysis of data is a process of inspecting, cleaning, transforming, and modeling data with the goal of discovering useful information, suggesting conclusions, and supporting decision-making. Examples • Interactive dashboards → Interactive analytics • Daily/weekly/monthly reports → Batch analytics • Billing/fraud alerts, 1 minute metrics → Real-time analytics • Sentiment analysis, prediction models → Machine learning

28. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Spark Streaming Apache Storm AWS Lambda KCL Amazon Redshift Spark Impala Presto Hive Amazon Redshift Hive Spark Presto Impala Amazon Kinesis Apache Kafka Amazon DynamoDB Amazon S3data Hot Cold Data Temperature Processing Latency Low High Answers Amazon EMR (HDFS) Hive Native KCL AWS Lambda Data Temperature vs Processing Latency Batch

29. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Interactive Analytics Takes large amount of (warm/cold) data Takes seconds to get answers back Example: Self-service dashboards

30. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Batch Analytics Takes large amount of (warm/cold) data Takes minutes or hours to get answers back Example: Generating daily, weekly, or monthly reports

31. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Real-Time Analytics Take small amount of hot data and ask questions Takes short amount of time (milliseconds or seconds) to get your answer back • Real-time (event) • Real-time response to events in data streams • Example: Billing/Fraud Alerts • Near real-time (micro-batch) • Near real-time operations on small batches of events in data streams • Example: 1 Minute Metrics

32. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Predictions via Machine Learning ML gives computers the ability to learn without being explicitly programmed Machine Learning Algorithms: - Supervised Learning ← “teach” program - Classification ← Is this transaction fraud? (Yes/No) - Regression ← Customer Life-time value? - Unsupervised Learning ← let it learn by itself - Clustering ← Market Segmentation

33. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Analysis Tools and Frameworks Machine Learning • Mahout, Spark ML, Amazon ML Interactive Analytics • Amazon Redshift, Presto, Impala, Spark Batch Processing • MapReduce, Hive, Pig, Spark Stream Processing • Micro-batch: Spark Streaming, KCL, Hive, Pig • Real-time: Storm, AWS Lambda, KCL Amazon Redshift Impala Pig Amazon Machine Learning Streaming Amazon Kinesis AWS Lambda AmazonElasticMapReduce StreamProcessingBatchInteractiveML Analyze

34. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Real-time Analytics Producer Apache Kafka KCL AWS Lambda Spark Streaming Apache Storm Amazon SNS Amazon ML Notifications Amazon ElastiCache (Redis) Amazon DynamoDB Amazon RDS Amazon ES Alert App state Real-time Prediction KPI process store DynamoDB Streams Amazon Kinesis

35. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Interactive & Batch Analytics Producer Amazon S3 Amazon EMR Hive Pig Spark Amazon ML process store Consume Amazon Redshift Amazon EMR Presto Impala Spark Batch Interactive Batch Prediction Real-time Prediction

36. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Batch Layer Amazon Kinesis data process store Lambda Architecture Amazon Kinesis S3 Connector Amazon S3 A p p l i c a t i o n s Amazon Redshift Amazon EMR Presto Hive Pig Spark answer Speed Layer answer Serving Layer Amazon ElastiCache Amazon DynamoDB Amazon RDS Amazon ES answer Amazon ML KCL AWS Lambda Spark Streaming Storm

38. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Collect Store Analyze Consume A iOS Android Web Apps Logstash Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Apache Kafka Amazon Glacier Amazon Kinesis Amazon DynamoDB Amazon Redshift Impala Pig Amazon ML Streaming Amazon Kinesis AWS Lambda AmazonElasticMapReduce Amazon ElastiCache SearchSQLNoSQLCache StreamProcessingBatchInteractive Logging StreamStorage IoTApplications FileStorage Analysis&Visualization Hot Cold War m Hot Slow Hot ML Fast Fast Transactional Data File Data Stream Data Notebook s Predictions Apps & APIs Mobile Apps IDE Search Data ETL Amazon QuickSight

39. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Consume • Predictions • Analysis and Visualization • Notebooks • IDE • Applications & API Consume Analysis&Visualization Amazon QuickSight Notebook s Predictions Apps & APIs IDE Store Analyze ConsumeETL Business users Data Scientist, Developers

41. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Collect Store Analyze Consume A iOS Android Web Apps Logstash Amazon RDS Amazon DynamoDB Amazon ES Amazon S3 Apache Kafka Amazon Glacier Amazon Kinesis Amazon DynamoDB Amazon Redshift Impala Pig Amazon ML Streaming Amazon Kinesis AWS Lambda AmazonElasticMapReduce Amazon ElastiCache SearchSQLNoSQLCache StreamProcessingBatchInteractive Logging StreamStorage IoTApplications FileStorage Analysis&Visualization Hot Cold War m Hot Slow Hot ML Fast Fast Amazon QuickSight Transactional Data File Data Stream Data Notebook s Predictions Apps & APIs Mobile Apps IDE Search Data ETL Reference Architecture

42. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Problem Statement: • Need massive scalability and elasticity Use of AWS: • Nearly 100% of its online video service on AWS • Global use of Amazon EC2, Amazon S3, Amazon SQS, Amazon EMR, Lambda, etc. • 30-50K EC2 instances Business Benefits: • Application achieves near zero downtime • Massive scalability and elasticity • Transcoding entire library to ~60 output renditions “AWS is the market leader and has been able to create a continuous and virtuous cycle.” – Kevin McEntee, VP Content Engineering, Netflix Customer: Netflix

43. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AdRoll Builds Bidding Platform on AWS and Cuts Costs by 83% AdRoll is a global leader in digital advertising retargeting products. We’ve been able to seamlessly scale our infrastructure and reduce our fixed costs by 75% and operational costs by 83%.” Valentino Volonghi CTO, AdRoll ” “ • AdRoll manages its Real-Time Bidding platform using Amazon EC2, Amazon Dynmo DB, and Amazon S3 • Reduced annual operational costs by 83% • Reduced fixed costs by 75% • Staff now 95% focused on new product development

44. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Problem Statement: • Needed scalable, high performance, and highly available storage and big data solutions Use of AWS: • Direct Connect, S3, EMR, other AWS services • Went from ~5GB of logs per day to ~1300GB/day Business Benefits: • By moving to AWS, went from spending $50K/mo to $13K/mo on big data solutions Xfinity X1 Set Top Box Platform Customer: Comcast