The document discusses AWS Storage Gateway, which provides hybrid storage solutions for using AWS storage services like S3 and Glacier to store data on-premises. It describes the File, Volume, and Tape gateway types and their capabilities like standard storage protocols, caching, and security integration. Use cases include enabling cloud workloads, tiered storage, and backup/disaster recovery. Metrics and best practices for monitoring performance and data transfer are also covered.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Paul Reed – Storage Gateway Product Manager
18 April 2017
AWS Storage Gateway

2. Cloud Data Migration
Direct
Connect
Snow* data
transport
family
3rd Party
Connectors
Transfer
Acceleration
Storage
Gateway
Kinesis Firehose
AWS Storage Platform and SolutionsThe AWS Storage Portfolio
Object
Amazon GlacierAmazon S3
Block
Amazon EBS
(persistent)
Amazon EC2
Instance Store
(ephemeral)
File
Amazon EFS

3. Hybrid storage use cases and architectures
Enabling cloud workloads
Move data to AWS storage for Big Data, cloud bursting, or migration
Tiered cloud storage
Easily add AWS storage to your on-premises environment
Backup, archive, and disaster recovery
Cost effective storage in AWS with local or cloud restore

4. Storage Gateway hybrid storage solutions
Enables using standard storage protocols to access AWS storage services
AWS Storage
Gateway
Amazon EBS
snapshots
Amazon S3
Amazon Glacier
AWS Identity and Access
Management (IAM)
AWS Key Management
Service (KMS)
AWS
CloudTrail
Amazon
CloudWatch
Files
Volumes
Tapes

5. Storage Gateway – Files, volumes, and tapes
File gateway NFS (v3 and v4.1) interface
On-premises file storage backed by Amazon S3 objects
Tape gateway iSCSI virtual tape library interface
Virtual tape storage in Amazon S3 and Glacier with VTL management
Volume gateway iSCSI block interface
On-premises block storage backed by S3 with EBS snapshots

6. Storage Gateway – Common capabilities
Standard storage protocols integrate with on-premises applications
Local caching for low-latency access to frequently used data
Efficient data transfer with buffering and bandwidth management
Native data storage in AWS
Stateless virtual appliance for resiliency
Integrated with AWS management and security

7. File gateway
On-premises file storage maintained as objects in Amazon S3
Customer Premises
File
Gateway
Data stored and retrieved from your S3 buckets
One-to-one mapping from files-to-objects
File metadata stored in object metadata
Bucket access through IAM role you own and manage
Use S3 Lifecycle Policies, versioning, or CRR to manage data
GlacierS3
Standard
S3
Standard -
Infrequent
Access
HTTPS
NFS
v3 / v4.1
Application
Server

8. Enabling cloud workloads
Move data to AWS storage for Big Data, cloud bursting, or migration
“Storage Gateway has the promise to transform the way we move
data into the cloud. The NFS interface lets us easily integrate data
files from analytical instruments, and the transparent S3 storage
lets us easily connect our cloud-based applications and leverage the
powerful storage capabilities of S3.
With Storage Gateway, we can now unleash the full power of AWS
on our instrument data.”

9. S3 Bucket
Storage
Gateway
S3 Bucket
Read-only NFS client
Cross region
replication
NFS clientAWS Snowball
S3 Bucket
AWS SnowballS3 cross-region replication
Storage
Gateway
Read-only NFS client
Job
GETs/PUTs
objects
RefreshCache
Amazon
EMR
In-cloud workload
S3 Bucket
Storage
Gateway
RefreshCache RefreshCache
File Gateway – CacheRefresh

10. Application
Server
Volume Gateway
On-premises volume storage backed by Amazon S3 with EBS snapshots
Block storage in S3 accessed via the volume gateway
Data compressed in-transit and at-rest
Backup on-premises volumes to EBS snapshots
Create on-premises volumes from EBS snapshots
Up to 1PB of total volume storage per gateway
Amazon
EBS
snapshots
Storage Gateway
bucket in
Amazon S3
Customer Premises
Volume
Gateway
iSCSI HTTPS

11. Customer data center
AWS Storage
Gateway VM
Volume Gateway – Stored
Primary data stored on-premises
Asynchronous upload to AWS
Point-in-time backups stored as Amazon EBS snapshots
Up to 32 volumes, up to 16 TB each, for up to 512 TB per gateway
Amazon EBS
snapshots
Application
server
INITIATOR
TARGET
Upload
Buffer
Volume
Storage
AWS
Storage Gateway
service

12. Customer data center
AWS Storage
Gateway VM
Volume Gateway – Cached
Primary data stored in AWS
Frequently accessed data cached on-premises
Point-in-time backups stored as Amazon EBS snapshots
Up to 32 volumes, up to 32 TB each, for up to 1 PB per gateway
Amazon EBS
snapshots
Application
server
INITIATOR
TARGET
Upload
Buffer
Cache
Storage
Volume storage
backed by
Amazon S3
AWS
Storage Gateway
service

13. Tiered storage to S3, Glacier, and EBS
Easily add AWS storage to your on-premises environment
“Storage Gateway is at the core of our disaster
recovery and business continuity (BCM) processes,
handling our co-lo'd OLTP and OLAP off site data
backups, as well as our in-office BCM. It works
transparently, in a lights out way, archiving off to a
separate AWS account with a simple grandfather-father-
son snapshot plan in place”.

14. Tape gateway
Virtual tape storage in Amazon S3 and Glacier with VTL management
Virtual tape storage in S3 and Glacier accessed via tape gateway
Data compressed in-transit and at-rest
Up to 1 PB total tape storage per gateway, unlimited archive capacity
Supports leading backup applications:
Archived Tapes
stored in
Amazon Glacier
MEDIA
CHANGER
TAPE
DRIVE
Customer Premises
Tape
Gateway
Virtual Tapes
stored in
Amazon S3
Backup
Server
HTTPSiSCSI

15. Backup, archive, and disaster recovery
Cost effective storage in AWS with local or cloud restore
“Tapes are a headache, prone with hardware
failures, offsite storage costs, and constant
maintenance needs. Storage Gateway
provided the most cost-effective and simple
alternative. We even got disaster recovery by
using a bi-coastal data center”.

16. Storage Gateway – Key Benefits
Seamless integration across standard storage protocols
Low-latency access
Durability, cost, and elasticity of AWS Storage services
Efficient data transfer
Data encryption
Integrated with AWS monitoring, management, and security

17. Storage Gateway – Pricing
All gateway types $0.01 per GB of data written to AWS*
File Files stored and billed by S3
Volume $0.023 per GB-month of volume data stored
Snapshots stored and billed by EBS
Tape $0.023 per GB-month of tape data stored
$0.004 per GB-month of tape data archived
$0.01 per GB of data retrieved from archive
* Up to a maximum of $125/month. First 100GB per gateway free.

18. Under the hood

19. File Gateway – File system metadata
File system metadata persisted in object user-metadata, eg.
Configurable defaults for objects that don’t have this metadata
• E.g. objects that were already in the bucket
Changing file metadata copies the object
x-amz-meta-file-permissions: 0666
x-amz-meta-file-user-class: 4321
x-amz-meta-file-group-class: 42
x-amz-meta-file-created: 2016-10-05T20:08:45+00:00
x-amz-meta-file-last-modified: 2016-10-05T20:08:45+00:00

20. How do I monitor cache performance?
CloudWatch metrics metrics for gateway
CachePercentUsed
CacheHitPercent
CachePercentDirty
High CachePercentUsed is good
Once 100% will start to impact CacheHitPercent
May indicate the cache is too small for the working set
Writes increase CachePercentDirty
Reduces size of working cache, and can reduce CacheHitPercent
May indicate data not uploading to AWS quickly enough

21. How does Storage Gateway transfer data to
AWS?
Dirty data asynchronously uploaded to AWS
Byte-level parallel upload and download
Compression for tape and volume
Uploaded data committed to storage resource
periodically
File uses multi-part PUT
Tape and volume create periodic recovery points (internal
snapshots)
Rate of writes by application determine periodicity of
commit
High write rates will commit more frequently
Low write rates will commit based on timer

22. How do I monitor data transfer?
CloudWatch metrics for gateway and per storage resource
For volumes TimeSinceLastRecoveryPoint indicates how
long since the last commit
Application
server
Storage Gateway
WriteBytes
ReadBytes
CloudBytesUploaded
CloudBytesDownloaded

23. Monitoring and Security extended to on-
premises gateways
IAM Integration for File Gateway rolesCloudTrail Logging of API Calls

24. Getting Started
AWS Storage Gateway home page:
aws.amazon.com/storagegateway
AWS Storage Gateway documentation:
aws.amazon.com/documentation/storage-gateway/
Ready to get started? AWS Storage Gateway console:
console.aws.amazon.com/storagegateway/home
Storage Gateway

25. Thanks!
Simple, Secure, Cost-effective Hybrid Storage in AWS