The document discusses AWS Storage Gateway, which allows for hybrid cloud storage architectures by enabling on-premises access and transfer of data to AWS cloud storage services. It provides an overview of Storage Gateway's file, volume, and tape gateway types. It also discusses how Kellogg's uses Storage Gateway for backup and disaster recovery, including migrating from traditional backups to using Storage Gateway to backup to S3 and EBS. Best practices for using Storage Gateway are also covered.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Deep dive: build hybrid storage
architectures with AWS Storage Gateway
S T G 3 0 5
Asa Kalavade
General Manager
AWS Storage Gateway
Paul Reed
Principal Product Manager
AWS Storage Gateway
Abdul Mohammed
Sr. IT Architect
Kellogg Company

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
• AWS Storage Gateway and hybrid storage
• What’s new since re:Invent 2017
• How Kellogg’s is using Storage Gateway
• Deep dive and best practices
• Q&A
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
AWS Storage Gateway

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What’s the hybrid storage problem?
You have on-premises data and applications …
… that want to use storage
and services in the cloud
AWS

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hybrid storage
Backup on-premises
data and applications
Multi-protocol access
(in-cloud and on-prem
access same date)
Tier
cold data
Low-latency on-premises
access to in-cloud data
Content
Distribution
Simplify
Branch Offices

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hybrid storage requirements
Integrate with AWS Storage1. Work with existing applications
Enable access to in-cloud processing
2. Low-latency access to data
in the cloud
Integrate with management
and monitoring in the cloud
3. Integrate and operate within
enterprise environments
Access from multiple locations
4. Optimized data transfers,
resilience to network variations
In-cloudOn-premises

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hybrid storage with Storage Gateway
On-premises In-cloud
Integrates with Amazon S3, Amazon EBS,
and Amazon Glacier
1. Standard storage protocols
Notifications to enable automated data
pipelines in the cloud
2. Low-latency access via fully
managed cache
Manage through the AWS console as
another AWS Service
3. Multiple hardware and
software deployment options
Enables native AWS capabilities such as
storage classes and replication
4. Optimized data transfers
AWS Storage
Gateway

9. Hybrid cloud storage is experiencing a
renaissance, with new storage offerings
bridging the gap between on-premises
and public cloud providers
Gartner, Oct. 2018

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS Storage Gateway
Customer premises
Files
(NFS/SMB)
Volumes
(iSCSI)
Tapes
(iSCSI VTL)
AWS Storage Gateway
Integrated with IAM, KMS, CloudTrail, CloudWatch services
HTTPS
Amazon Glacier
Amazon S3
Amazon Elastic
Block Store
Gateway appliance Service

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
File, volume, and tape storage types
Store and access objects
in Amazon S3 from
file-based applications
with local caching
Block storage on-premises
backed by cloud storage
with local caching, Amazon
EBS snapshots, and clones
Drop-in replacement for
physical tape infrastructure
backed by cloud storage
with local caching
File Gateway Volume Gateway Tape Gateway

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Storage gateway
Cost effective, while providing enterprise compliance and security needs
File
Gateway
Billed as Amazon S3 storage
Volume
Gateway
$0.023/GB storage
$0.05/GB EBS snapshot storage
Tape
Gateway
$0.023/GB virtual tapes storage
$0.004/GB archived tape storage
Data written
$0.01/GB written to AWS, capped
at $125/month
* **
* **
Additional request and AWS Data Transfer pricing applies.

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Available in 17 regions globally

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Support
for S3 Z-IA
Launches since re:Invent 2017
Tape
File
All Paris
region
Dell EMC
Networker 9
Support for
content
distribution
workflows
GovCloud
region
Singapore
region
NovaStor
DataCenter
6 & 7
PCI DSS
compliance
SMB
support
KMS
support
Hardware
appliance
Refresh
cache folders
Up to 4 Gbps writes
IBM Spectrum
Protect and
Bacula
Up to 2.3 Gbps
writes, 0.6 Gbps
reads

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Some AWS Storage Gateway customers

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
File gateway: overview and use cases
Store and access objects in Amazon S3 from file-based applications with local caching
• Backup on-premises data to the cloud
• Hybrid cloud workflows
• Low-latency on-premises access to cloud storage
• Content distribution and collaboration via the cloud
Use cases
On-Premise
NFS & SMB
File Gateway
HTTPS
Application S3 bucket

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Connect using NFS v3/v4 or SMB v2/v3 protocols
Files stored as native S3 objects
Metadata is preserved as object user metadata
Object-level encryption with SSE-S3 or SSE-KMS
Fully managed local cache
Read-through, write-back, LRU managed
Notifications through Amazon CloudWatch (e.g., upload complete)
Optimized data transfers
Uploads only sends changes, downloads retrieve file parts needed
Refresh cache by prefix
Optimizes content distribution workloads
Performance increase
Client writes at up to 4 Gbps
File gateway

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tape gateway: Overview and use cases
Virtual tapes presented to on-premises backup applications
• Tape based backup with existing backup apps
• Archive to Glacier
Use cases
On-Premise
iSCSI VTL
Tape Gateway
HTTPS
Application
Tape Shelf
(Amazon Glacier)
Tape library
(Amazon S3)
Service

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Tape gateway
Emulates physical tape library through iSCSI-VTL protocol
Fully managed local cache for recent backups
Read-through, write-back, LRU managed
Virtual tapes stored in Amazon S3
Ejected virtual tapes archived read-only in Amazon Glacier
Retrieve archived tapes to library (3–5 hours)
Configurable encryption SSE-S3 or SSE-KMS
Compatible with all leading backup software
New backup apps: IBM Spectrum Protect, Bacula
Performance increase
Client writes up to 2.3 Gbps, downloads up to 0.6 Gbps

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Volume gateway: Overview and use cases
Block storage on-premises backed by cloud storage
• Backup on-premises data to the cloud
• Migration of volumes to the cloud
• DR to the cloud
Use cases
Service
On-Premise
iSCSI HTTPS
GatewayApplication Amazon EBS
snapshots

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Volume gateway
Connect using the iSCSI block protocol
Volumes stored in AWS reducing on-premises SAN footprint
Thin-provisioned (cached) or local (stored) volume types
Fully managed local cache
Read-through, write-back, LRU managed
Configurable encryption with SSE-S3 or SSE-KMS
Volume snapshots stored in Amazon EBS

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Storage Gateway is available on
a variety of form factors
New

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hardware appliance: Easy as 1, 2, 3 …
Purchase on amazon.com
with your Amazon or
Amazon business account
1U rack-mounted appliance
ships with Storage Gateway
software pre-installed
Add and remove gateways
on the appliance from the
AWS Console. Software is
automatically updated.
Simple procurement Simple installation Manage from AWS

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hardware appliance:
Configuration, redundancy & performance
Gateway configuration
• Supports file, volume (cached) and tape gateways
• Standardized cache and upload buffer configuration
Availability
• Software-based ZFS RAID
• Dual auto-switching power supplies
• Dual boot drives for software
Predictable performance
• Dedicated platform for storage gateway offers predictable throughput performance

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hardware appliance:
Configuration, redundancy & performance
Support
• Contact AWS for support; we will engage Dell for hardware support
• Ability to open support channel on local hardware console for remote troubleshooting
Availability
• Sold and ships in the United States
• Use with four AWS US public regions – US East (N. Virginia), US East (Ohio), US West
(Oregon), US West (Northern California)

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hardware appliance since launch
Industryverticals
Examples
Fromcustomers
Use case: backups
Use case: file shares
Use cases: backups, file shares

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Abdul Mohammed
Sr. Infrastructure Architect
Kellogg Company

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Who is Kellogg’s
The world's leading cereal company, 2nd largest producer of cookies, crackers, and savory snacks,
and leading North American frozen foods company
2017 net sales of ~$13 billion
100+ Year Heritage

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Business drivers
Agility Tight margins New costing model
(capex to opex)
End of life
infrastructure
Scalability

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
IT architectures are complex
Global Data Storage
Colo
Facilities
Hadoop
Data
Remote Offices & Users
Applications
File Servers
Offsite
Archives
SAP Multimedia
Data Lakes
Workstations Mobile DataUsers Branch
Offices
SQL
How to work with file servers,
databases, user directories, data
lakes, and archives both on
premises and in-cloud?

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cloud journey: Migrating SAP on Oracle + backups
• Dev and Test workloads initially
• SAP migration to AWS for APAC
• Lift & shift + upgrades
• 100 GB to 20 TB Oracle databases
• Traditional RMAN backups
• Compute costs down, backup costs the same
• Goal: Cloud-native backup
• With point-in-time recovery
• @ lower cost

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cloud native backup
For application servers, Oracle & SQL Server databases and SAP HANA
Cloud Based
• Pay for what we use,
no upfront cost
• Reduced backup infrastructure management
• Scale up as usage went up
• EBS Snapshot for daily backup
• Storage Gateway for point in time recovery
Traditional
• Costly (needed capital to standup same infrastructure
for other regions)
• Traditional backup required us to maintain multiple
master and media servers
• Needed resourcing staff to maintain this infrastructure
• Licensing

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Storage gateway simplifies backup
Remote Offices & Users
Applications
Global Data Storage
File Servers
Colo
Facilities
Hadoop
Data
Offsite
Archives
SAP Multimedia
Data Lakes
Workstations Mobile DataUsers Branch
Offices
SQL
How to work with file servers,
databases, user directories, data
lakes, and archives both on premises
and in-cloud?
Storage Gateway works in
multiple environments across
all types of workloads.

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Initial architecture: Cached volume gateway + Amazon EBS
Transaction log backup & recovery to maintain low RPO
Needed backup storage mount across DB servers
Multiple client writes concurrently
Created a single 192 TB volume using six 32 TB cached volumes
At peak (bulk loading) we were backing up 350GB/hour of
transactional logs (Oracle RMAN or SAP HANA) to Storage gateway
2X faster jobs vs. traditional backup
RTO in <10 minutes with Python scripts
Cached volume snapshot every 2 hours and copied to DR region
However, cross-region copy process pushed the boundary of RPO
Storage Gateway

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Final solution: file gateway for logs + Amazon EBS snapshots
Transaction log backup & recovery to maintain low RPO
Replicated S3 bucket to meet RPO
AWS does the heavy lifting; Amazon S3 cross-region
replication (CRR)
Achieved RPO of 15 minutes
2.5X faster than volume gateway backup (no NFS layer)
Same RTO of <10 minutes
Set and forget infrastructure
Encryption on by default
S3 life cycle rule for cleanup
S3 bucket
Storage Gateway

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
On-premises SQL Server backup
Microsoft SQL Server native backup
for on premise system via SMB
Access control via Active Directory
Archival backup for systems that will
be decommissioned
S3 Lifecycle rules that moves backup’s
to Glacier for long-term retention
Native SQL Server restore to Amazon
EC2 or RDS
Amazon S3
SQL Servers in US data centers
AWS Storage Gateway
Amazon RDS
Amazon EC2
West East

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Additional Storage Gateway use cases
Hadoop
• HDFS write to S3 via S3A connector
• Needed POSIX compliant file system
• Tried open-source S3FS connector; lots of
locking issues
• File Gateway to the rescue
• Mounted specific folder/data set to third
party applications
Backup target expansion
• Running out of backup space for on-premise
system
• Deployed file gateway to mount S3 bucket
• Instant access to unlimited space for backup target

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Best practices, tips & tricks
Use IO1 volume type for upload buffer
Separate out disk for upload and cache volume for
on-prem. gateway
Keep an eye on CachePercentDirty in CloudWatch Metrics
Use Amazon EC2 instance type that have high throughput
For gateway running on Amazon EC2, enable instance recovery
Rebalance backup job to optimize throughput
Keep an eye on cross-region copy of snapshot; this can
turn into full
Refresh cache for Oracle RMAN backup catalog sync
IO1
Amazon CloudWatch

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Outcomes
Move from legacy backup to Volume Gateway cut monthly cost from 50%
Switch to File Gateway further optimized cost and brought it down to 90%
90%
Reduction in
monthly backup costs
40%
Reduction of cash
outlay over 5 years
25-40%
Reduction in
server resources

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Future
SAP on
X & U instances
Ask:
High availability
Larger
cloud footprint
Consolidating
gateways
Storage Gateway
hardware appliance

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Objects in Amazon S3 accessed as files on-premises
Each file share maps to an Amazon S3 bucket
• Up to 10 file shares per gateway
Each file share accessed by either NFS or SMB
• Share access restricted by IP (NFS) and password / AD (SMB)
Each file is an object in Amazon S3
• Files access restricted by POSIX permissions
• File metadata stored in S3 object user metadata
All operations are performed against local cache
• Read-through / write-back
Devices
Users
Applications
On-Premise
File Gateway

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Objects in Amazon S3 accessed as files on-premises
AWS
AWS region
Objects in your
S3 bucket
NFS or
SMB client
File
share
Data
cache
Metadata
cache
File gateway
Some of your data
cached on gateway
Up to 16 TB
All of your data durably
stored in Amazon S3

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Objects in Amazon S3 accessed as files on-premises
AWS
AWS region
Objects in your
S3 bucket
NFS or
SMB client
File
share
Data
cache
Metadata
cache
File gateway
Parts of files cached based on what the client reads/writes
Data cached until space needed for more recently accessed files

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Objects in Amazon S3 accessed as files on-premises
AWS
AWS region
Objects in your
S3 bucket
NFS or
SMB client
File
share
Data
cache
Metadata
cache
File gateway
Gateway assumes IAM role to PUT/GET from your bucket
Configure storage class, encryption with AWS KMS, MIME
type, requester pays, …

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Cache management and file to object conversions
AWS
AWS region
Objects in your
S3 bucket
NFS or
SMB client
File
share
Data
cache
Metadata
cache
File gateway
• File writes asynchronously uploaded to S3 objects
• Small writes aggregated into S3-sized PUTs
• Parallel MultipartPUTs
• Validation and automatic error handling
• CopyPUTs to optimize change objects
• File reads synchronously cached from S3
• Byte-range GETs from S3 with predictive pre-
fetching
s3:ListBucket /foo/*
C:> dir D:foo
$ ls /foo/*

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Active workloads with notifications
AWS
AWS region
Objects in your
S3 bucket
NFS or
SMB client
File
share
Data
cache
Metadata
cache
File gateway
• NotifyWhenUploaded and RefreshCache APIs
• Used to trigger workloads
1. NotifyWhen
Uploaded API
2. Amazon
CloudWatch
Events
3. Trigger
processing
Amazon
QuickSight
Amazon
Rekognition
AWS
Lambda
4. RefreshCache
API
2nd
gateway

49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content distribution: Separate folders
nfs:/bucket/blue/
nfs:/bucket/green/
File gateway A
File gateway B
s3:/bucket
Each gateway mounts the bucket with
a unique prefix
(+) Safe multi-write; no object collisions
(-) Can’t share between A and B

50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content distribution: Single writer, multiple readers
File gateway A
(reader)
File gateway B
(reader)
Single writable file share, multiple read-only file shares
Read-only call RefreshCache periodically or with UploadNotification API
Can scope RefreshCache to refresh by folder to improve performance
S3 bucket
File gateway A
(writer)

51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Content distribution: Cross Region Replication
Use CRR for replicating your files to buckets in different regions
You can use Cross Region Replication and RefreshCache to distribute content
across regions (e.g. regional and branch offices)
File gateway A
(writer)
S3 bucket File gateway A
(reader)
S3 bucket

52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Best practices: Optimizing performance

53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Using CloudWatch to monitor your on-premises
gateway like an AWS service
AWS
S3 bucket

54. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Best practices: Leveraging Amazon S3 storage classes
Use lifecycle management for write-once workloads (e.g., archival)
Use Amazon S3 Standard for frequently used data
• Reading files in S3 Infrequent Access Classes (S-IA and Z-IA) costs retrieval fees
• Modifying files in S-IA and Z-IA creates new objects and could lead to deletion fees
• Glacier retrieval times leads to IO errors
• Modifying objects in Glacier not supported: retrieve files first

55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Best practices: Leveraging Amazon S3 storage classes
Use bucket versioning to maintain copies of your files
Setup version lifecycle retention policy
• Each time you change a file, you’ll get a new version
To recover/restore an older version
• Use the S3 API/Console to replace the current object with the old object
• Then call RefreshCache to update the file in the gateway

56. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

57. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Storage Gateway in summary
Feature Benefit
Standard storage protocols No changes to existing applications
Fully managed cache Low latency access to frequently used data
Automatic and optimized data transfer Minimizes network bandwidth
Automated Hybrid workflows Leverage in-cloud processing
Multiple Deployment options Supports multiple configurations
Security and compliance Meets enterprise customer needs
Managed from AWS console and CLI Simplified management and monitoring
Pay as you go pricing Cost effective

58. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS data transfer & hybrid storage
Online
data transfer
Hybrid
storage
Offline
data transfer
Load streaming
data into
Amazon S3
Ship static data
into and out of
Amazon S3
Access AWS
storage from
on-premises
Edge locations
for S3 enabled
applications
Online
transfer of
active data
AWS
DataSync
AWS
Transfer
for SFTP
Managed file
transfers into
Amazon S3
NEW
Storage and
compute in
disconnected
environments
NEW

59. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How do I get started?
Visit aws.amazon.com/
Drop by one of the demo booths
See one of us

60. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Asa Kalavade
kalavade@amazon.com
Paul Reed
paulreed@amazon.com

61. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.