Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

數位媒體雲端儲存案例和技術分享 (AWS Storage Options for Media Industry)

1,627 views

Published on

Web Applications on AWS: This session introduces AWS services that you can leverage to build a scalable web application architecture on AWS to handle large-scale flows.

Published in: Technology
  • Be the first to comment

數位媒體雲端儲存案例和技術分享 (AWS Storage Options for Media Industry)

  1. 1. ©2015,  Amazon  Web  Services,  Inc.  or  its  affiliates.  All  rights  reserved 數位媒體雲端儲存技術分享與案例 Harry Lin, AWS Solutions Architect linharry@amazon.com
  2. 2. Agenda Media Storage & Archive Media Ingest Case Study
  3. 3. Where is the problem?
  4. 4. AWS Direct Connect Elastic Load! Balancing! AWS Import/ Export Amazon S3 AWS Storage Gateway Amazon Glacier! Amazon EBS Amazon! CloudFront! Amazon CloudSearch! Amazon SNS! Amazon SQS! Amazon SWF! Amazon Elastic Transcoder! Amazon EC2! Amazon EMR! Amazon! DynamoDB! Amazon VPC! Amazon RDS! Amazon Redshift Amazon! ElastiCache AMI! Amazon! Route 53! Ingest Store DeliverProcess Amazon EC2! AWS Services Stack in a Media Workflow
  5. 5. MEDIA STORAGE & ARCHIVE
  6. 6. A Concept – The Content Lake Inspired from Data Lake (Coined by James Dixon in 2010) •  A single store of all of digital contents that you create and acquire in any form or factor •  Don’t assume any resolutions/formats •  It is up to the consumer (application consuming the content) to use the appropriate infrastructure for processing
  7. 7. Amazon S3: The Content Lake •  Designed for 99.999999999% durability •  Authentication & authorisation mechanisms ensure that data is kept secure •  Cross-region replication •  Multiple encryption options •  Object lifecycle management •  Standard, infrequent access (IA) & Reduced redundancy storage (RRS)
  8. 8. S3 Resources: Buckets and Objects
  9. 9. S3 Cross-Region Replication Automated, fast, and reliable asynchronous replication of data across AWS regions Source (Virginia) Destination (Oregon) •  Only replicates new PUTs. Once S3 is configured, all new uploads into a source bucket will be replicated •  Entire bucket or prefix based •  1:1 replication between any 2 regions •  Versioning required Use cases: •  Compliance—store data hundreds of miles apart •  Lower latency—distribute data to regional customers •  Security—create remote replicas managed by separate AWS accounts
  10. 10. S3 Event Notifications Delivers notifications to Amazon SNS, Amazon SQS, or AWS Lambda when events occur in S3 S3 Events SNS topic SQS queue Lambda function Notifications Notifications Notifications Foo() { … }
  11. 11. Amazon S3 Security Controls •  Bucket-level and object-level permissions •  Owner-only access (by default) •  Signed URLs/query string authentication •  AWS IAM policies •  Versioning (MFA delete) •  Detailed access logging ✔Access logs
  12. 12. Amazon S3 Server-Side Encryption •  Encryption& Decryption •  Key management (Encrypted by Amazon S3 master key; stored separately from your data) •  256-bit AES encryption •  User-provided keys •  Integration with AWS KMS Content to be uploaded (encryption enabled in the HTTP header) Envelop Key Encrypted stored key 101010101010 101010101010 101010101010 101010101010 101010101010 101010101010 101010101010 101010101010 Encrypted stored data Master Amazon S3 key Encryption Encryption Key generator Amazon S3
  13. 13. 1 PB raw storage 800 TB usable storage 600 TB allocated storage 400 TB application data S3 Capacity Pricing—Pay Only For What You Use! Amazon S3 No need for capacity planning or provisioning
  14. 14. Archiving Options - Amazon Glacier •  Designed for 99.999999999% durability •  AES 256 encryption at rest •  Highly scalable and reliable •  Authentication & authorisation mechanisms ensure data is kept secure •  Retrievals: •  5% of monthly average storage (pro-rated daily) free, after this 0.011 per gigabyte •  3–5 hour asynchronous retrieval aws.amazon.com/glacier
  15. 15. $0.0114 per GB-month
  16. 16. $140 per TB/year
  17. 17. 99.999999999% durability
  18. 18. Storage costs vs. Retrieval costs
  19. 19. Glacier Benefits •  Reduce cost for long-term archiving •  Leverage unlimited storage capacity •  Replace tape museums •  Improve durability
  20. 20. S3 Lifecycle Policies → Key prefix “logs/” Transition objects to Glacier 30 days after creation Delete 365 days after creation date <LifecycleConfiguration> <Rule> <ID>archive-in-30-days</ID> <Prefix>logs/</Prefix> <Status>Enabled</Status> <Transition> <Days>30</Days> <StorageClass>GLACIER</StorageClass> </Transition> <Expiration> <Days>365</Days> </Expiration> </Rule> </LifecycleConfiguration
  21. 21. Selecting the Right Object Storage Class S3 S3-IA Glacier L i f e c y c l e Available S3: 99.99% S3-IA: 99.9% Performant Low Latency High Throughput Secure SSE, client encryption, IAM integration Event Notifications SQS, SNS, and Lambda Versioning Keep multiple copies automatically Cross Region Replication Common Namespace Define storage class per object Durable 99.999999999% Scalable Elastic capacity No preset limits “Hot” Data Active and/or Temporary Data “Warm” Data Infrequently Accessed Data “Cold” Data Archive and Compliance Data
  22. 22. Selecting the Right Object Storage Class S3 S3-IA Glacier L i f e c y c l e Available S3: 99.99% S3-IA: 99.9% Performant Low Latency High Throughput ≥ 30 Days ≥ 90 Days Durable 99.999999999% Scalable Elastic capacity No preset limits $0.007/GB per month $0.0125/GB per month “Hot” Data Active and/or Temporary Data “Warm” Data Infrequently Accessed Data “Cold” Data Archive and Compliance Data ≥ 0 Days~$0.03/GB per month 3 – 5 Hrs $0.01/GB retrieval $0.01/GB retrieval < 5%
  23. 23. Amazon Elastic Block Store (EBS) •  High I/O block storage for Amazon EC2 •  Predictably scale to 1000s of IOPS per Amazon EC2 instance •  Automatic replication within the Availability Zone •  Point-in-time snapshots •  Amazon S3 durability (11-9s) •  Point-in-time snapshots across regions aws.amazon.com/ebs/
  24. 24. EBS – The SAN in the cloud
  25. 25. EBS – Persists beyond an instance
  26. 26. EBS – Attach multiple volumes to an instance
  27. 27. EBS – Snapshots to S3 are your backups
  28. 28. EBS use cases •  Persistent block storage for Amazon EC2 •  Transactional workloads •  File system for an instance—NTFS, ExtFS, and so on
  29. 29. Overview of Amazon EBS Volume Types Solid State Drives (SSD) Hard Disk Drives (HDD) ------------------------------------------------------------------Current Generation------------------------------------------------------------------- -----Previous Gen----- Volume Type Provisioned IOPS SSD (io1) General Purpose SSD (gp2) Throughput Optimized HDD (st1) Cold HDD (sc1) Magnetic Use Cases I/O-intensive NoSQL and relational databases Boot volumes, low- latency interactive apps, dev & test Big data, data warehouses, log processing Colder data requiring fewer scans per day Infrequent data Access Volume Size 4 GB - 16 TB 1 GB - 16 TB 500 GB - 16 TB 500 GB - 16 TB Up to 1TB Max IOPS*/ Volume 20,000 10,000 500 250 40 - 200 Max Throughput/ Volume 320 MB/s 160 MB/s 500 MB/s 250 MB/s 40 - 90MBps Max Throughput / Instance 800 MB/s 800 MB/s 800 MB/s 800 MB/s 800MBps Price** $0.125/GB-month $0.065/provisioned IOPS $0.10/GB-month $0.045/GB-month $0.025/GB-month $.05/GB - Month $.05/million I/O New! EBS Volumes available in all regions * io1/gp2 based on 16K I/O size, st1/sc1 based on 1 MB I/O size ** Pricing is for US East-1 as of 4/19/2016, for the latest see Amazon EBS Pricing
  30. 30. MEDIA INGEST
  31. 31. •  Reduces costs for bandwidth-heavy workloads •  Private connectivity to AWS •  Physical connection – 1 Gbps or 10 Gbps port •  Logical connections (802.1q VLANs) •  Public: To AWS cloud (Amazon S3 endpoints etc.) •  Private: To in-VPC resources •  Consistent network performance •  Compatible with all AWS services Ingest Options – AWS Direct Connect aws.amazon.com/directconnect
  32. 32. •  Rapidly move data into and out of AWS •  Portable storage device shipment to AWS •  eSATA •  USB 2.0 and 3.0 •  2.5 and 3.5 inch internal SATA hard drives •  Supports •  Amazon Elastic Block Store (EBS) •  Amazon Simple Storage Service (S3) •  Amazon Glacier Ingest Options – AWS Import/Export Disk aws.amazon.com/importexport/disk
  33. 33. 1.  Multipart upload 2.  Request rate optimization 3.  TCP window scaling 4.  TCP selective acknowledgement Ingest Options – S3 Multipart Uploads
  34. 34. Amazon S3 Transfer Acceleration Embedded WAN acceleration S3 Bucket AWS Edge Location Uploader Optimized Throughput! Move over long geographic distances Up to 6 times faster No firewall mods, no client software 54 global edge locations Change your endpoint, not your code
  35. 35. S3 Transfer Acceleration Pricing Starting at $0.04/GB transferred (+ usual bandwidth charges). Up to $0.08/GB in some regions Pay only for what you use Accelerated performance or no charge
  36. 36. CASE STUDY
  37. 37. •  Netflix is one of the leading streaming video services for movies and TV with millions of users globally •  Needed a reliable, scalable system that would not be prone to data corruption •  With AWS, Netflix can quickly deploy thousands of servers and terabytes of storage within minutes—and spin servers down when they’re not needed Netflix Uses AWS to Increase Reliability and Efficiency
  38. 38. Use of AWS: Corporate website, 40 producer websites, streaming for iPhone and iPad applications Use Amazon EC2, Amazon S3, Amazon RDS, and stream 1.5 PB/m on Amazon CloudFront Business Benefits: Launched and scaled multiple media services fast Improved server set up time from months to 15 minutes Improved Reliability to 99.99% “I can’t imagine having been able to do very many of the things that we’ve done, in the timeframe that we’ve done them, without the benefit of a platform like this.” – Jon Brendsel, VP, Products PBS Distributes its Online and Mobile Media on AWS, Launching new Media Services Fast
  39. 39. Summary: Content Lake
  40. 40. Thank You! AWS is in TW for You

×