Amazon Elastic Block Store (Amazon EBS) provides persistent block level storage volumes for use with Amazon EC2 instances. In this technical session, we conduct a detailed analysis of the differences among the three types of Amazon EBS block storage: General Purpose (SSD), Provisioned IOPS (SSD), and Magnetic. We discuss how to maximize Amazon EBS performance, with a special eye towards low-latency, high-throughput applications like databases. We discuss the performance implications of our new larger and faster SSD volumes (up to 16 TB with increased max throughput levels), as well as Amazon EBS encryption. Throughout, we share tips for success.

1. ©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved
Amazon Elastic Block Store
Deep Dive
Dougal Ballantyne, HPC Solutions Architect

2. What we’ll cover
Amazon EBS overview
• Volumes
• Snapshots
Performance
Encryption
Q&A

3. EBS overview

4. For most builders AWS is get in and go!
Source: http://www.trucksplanet.com/catalog/model.php?id=1020

5. A “normal” hard drive

6. EBS =

7. What is EBS?
• Network block storage
• Designed for five nines of availability
• Attaches to Amazon EC2 within the same
Availability Zone
• Provides point-in-time snapshots to
Amazon S3

8. More about EBS
• It’s a service!
• It’s independent of EC2
• It has regional and AZ availability goals
– All EBS volumes are designed for 99.999% availability
• Over 1.5 million volumes are created per day

9. A few definitions…
• IOPS: Input/output operations per second (#)
• Throughput: Read/write rate to storage (MB/s)
• Latency: Delay between request and completion (ms)
• Capacity: Volume of data that can be stored (GB)
• Block size: Size of each I/O (KB)

10. EBS volume types
• General Purpose (SSD)
• Provisioned IOPS (SSD)
• Magnetic
When performance matters, use SSD-backed volumes

11. EBS SSD volumes
• Applies to both General Purpose and
Provisioned IOPS
• IOPS measured up to 256 KB
• Single-digit ms latency
• Designed for 99.999% availability

12. EBS General Purpose volumes (SSD)
New default volume type for EBS
Every volume can burst up to 3,000 IOPS
• Larger volumes can burst for longer periods
3 IOPS per GB baseline performance,
maximum of 10,000 IOPS
99% performance consistency
Up to 160 MB/s throughput

13. General Purpose (SSD) – Burst & baseline
16 KB I/O size

14. (2) Max I/O credit per bucket is 5.4M
(1) Always accumulating 3
IOPS per GB per second
(3) You can spend up to
3000 IOPS per second
Understanding General Purpose (SSD) bursting
Baseline performance = 3 IOPS per GB

15. Minutes to empty a full I/O credit bucket for various volume sizes
The larger the volume, the longer it takes to empty the I/O credit bucket
1 TB or larger volume will never exhaust its I/O credit bucket

16. Minutes to empty a full IO credit bucket for various sizes
The larger the volume, the longer it takes to empty the I/O credit bucket
1 TB or larger volume will never exhaust its I/O credit bucket

17. General Purpose (SSD) volumes example
Microsoft Windows 30 GB boot volume:
• Gets initial I/O credit of 5.4M
• Could burst for up to 30 mins @ 3000 IOPS
• Always accumulating 90 I/O credits per
second

18. Improved instance boot time
m3.medium
Volume type Boot time Access time OS
GP2 3:31 4:33 Windows Server
2012
Magnetic 4:30 7:16 Windows Server
2012
GP2 0:36 0:45 CentOS6
Magnetic 0:57 1:16 CentOS6
40% Reduction in boot times by using General Purpose SSD

19. 1 TB PIOPS volume with 4K IOPS = $526.40 per month per volume
GP2 1 TB volume with 3000 IOPS = $102.40
GP2 2 x 500 GB volume at 3K, Burst to 6K = $102.40
80% cost savings, 50% more peak I/O with
General Purpose SSD
Database volume

20. Guidelines for sizing General Purpose (SSD)
volumes
Generic boot, developer, test/dev, and web apps:
Provision GB required for your applications
Database apps:
1. Calculate the IOPS required in steady state
2. Perform this calculation: (steady state IOPS) ÷ 3 = GB to
provision
Note: I/O bursts will support:
• Database load or table scan operations
• Spike in I/O workload
20

21. EBS PIOPS (SSD) volumes
• Best for I/O intensive databases that require highest
consistency
• Throughput up to 320 MB/sec
• Provision up to 20,000 IOPS per volume
(supports IOPS:GB ratio of 30)
• Designed for 99.9% performance consistency

22. EBS Magnetic volumes
• Best for cold workloads (rarely accessed data that needs
always-on access)
• IOPS: ~100 IOPS steady-state, with best-effort bursts
• Throughput: variable by workload, best effort to 10s of MBs
• Latency: Varies, reads typically ~20-40 ms, writes typically
~2-10 ms

23. EBS volume types - summary
General Purpose (SSD) Provisioned IOPS (SSD) Magnetic
Recommend use cases
Boot volumes
Small to med DBs
Dev and test
I/O-intensive workloads
Large DBs
Cold storage
Storage media SSD-backed SSD-backed Magnetic-backed
Volume size 1 GB - 16 TB 4 GB - 16 TB 1 GB - 1 TB
Max IOPS per volume 10,000 IOPS 20,000 IOPS ~100 IOPS
Burst < 1 TB to 3000 IOPS baseline baseline
Read and write peak throughput 160 MB/s 320 MB/s ~50-90 MBps
Max IOPS per node (16k) 48,000 48,000 48,000
Peak throughput node 800 MB/s 800 MB/s 800 MB/s
Latency (random read) 1-2 ms 1-2 ms 20-40 ms
API Name gp2 io1 standard
Price* $.10/GB-month
$.125/GB-month
$.065/provisioned IOPS
$.05/GB-month
$.05/ 1M I/O

24. Why is General Purpose SSD the default?
High baseline level of performance
Burst to higher level of IOPS
Single, capacity-based pricing dimension
• Makes forecasting very easy
• Eliminates sizing complexity
Attractive price/gigabyte/price/IOPS density

25. Always use General Purpose (SSD) for boot volumes

26. Migrating to General Purpose (SSD) volumes
Change volume type during launch
Use EBS snapshots
You may be able to resize the file system
Use General Purpose (SSD) sizing guide

27. Benefits of using EBS snapshots
More durable than an EBS volume
• Stored in Amazon S3
Differential (space-efficient)
• First snapshot is a clone
• Pay only for what you use
Availability Zone-independent
• Clone into any AZ
Can be copied efficiently across regions

28. Tagging snapshots
Use tags to add
metadata to snapshots:
• Type (daily, weekly)
• Version
• Instance Id
• Volume Id
• Application stack

29. Performance

30. Queuing theory – Little’s law
Little’s law is the foundation for performance
tuning theory
• Mathematically proven by John Little in 1961
L = A * W
L = Queue length = average number of requests waiting
A = Arrival rate = the rate of requests arriving
W = Wait time = average wait time
EBS performance is related to this law

31. Performance optimization is measured by:
IOPS: Read/write I/O rate (IOPS)
Latency: Time between I/O submission
and completion (ms)
Throughput: Read/write transfer rate
(MB/s); throughput = IOPS X I/O size

32. Four key components of performance optimization
1. EC2 instance
2. I/O
4. EBS
3. Network
link

33. Tools available for performance tuning:
1. EC2 instance: Network bandwidth (Mbps)
2. EBS-optimized instance: EC2 instance option (On/Off)
3. Workload: Block size, read/write ratio, serialization
4. Queue depth: The number of outstanding I/Os
5. RAID: Stripe volumes to maximize performance
6. Pre-warming: Eliminate first-touch penalty

34. 1. EC2 instance
Compute-optimized – C3,C4
Memory-optimized – R3
General-Purpose – M3
EBS
EC2
Select the EC2 instance that has the right network,
RAM, and CPU resources for your applications

35. 2. EBS-optimized instance
Most instance families support the EBS-optimized flag
EBS-optimized instances now support up to 4 GB/s
• Drive 32,000 16K IOPS or 500 MB/s
EC2 *.8xlarge instances support 10 Gb/s network
Max IOPS per node supported is ~48,000 IOPS @ 16K I/O

36. Use EBS-optimized instances for
consistent EBS performance

37. 3. Workload
I/O size:
• 4 KB to 64 MB
I/O pattern:
• Sequential and random
I/O type:
• Read and write
I/O concurrency:
• Number of concurrent I/O
EBS SSD-backed volumes measure I/O size up to 256 KB
EBS SSD-backed volumes deliver same performance for read and write

38. EBS IOPS and throughput limits
20,000 IOPS
PIOPS volume
20,000 IOPS
320 MB/s
throughput
You can achieve 20,000 IOPS when
driving smaller I/O operations
You can achieve up to 320 MB/s
when driving larger I/O operations

39. EBS IOPS and throughput limits
8,000 IOPS
PIOPS volume
8,000 IOPS
320 MB/s
throughput
8,000 x 64 KB=512 MB/s
1,250 x 256 KB = 320 MB/s
8,000 X 8 KB = 64 MB/s
8,000 X 16 KB = 128 MB/s
16,000 x 8 KB = 128 MB/s
8,000 x 32 KB = 256 MB/s

40. Block (I/O) size determines whether your
application is IOPS bound or throughput bound

41. 4. Queue depth
An I/O operation
EBS
After it’s gone, it’s gone
EC2
Queue depth is the pending I/O for a volume

42. Monitoring EBS volumes
Important Amazon
CloudWatch metrics:
• IOPS and bandwidth
• Latency
• Queue depth

43. I/O latency
• Elapsed time between I/O submission and its completion
time
• Performance requirements may be driven by IOPS or
latency or both
• There is an interdependency among IOPS, queue depth,
and latency

44. Latency: What does it look like?

45. Latency: Introducing the boxplot

46. Latency: baseline

47. Latency: General Purpose SSD volume

48. Latency: Latest generation instance

49. EC2: Instance comparison
m2.4xlarge
CPU: Intel Xeon
vCPU: 8
Memory: 68.4 GiB
Price: $0.98/hour
r3.2xlarge
CPU: Intel Xeon E5-2670 v2
vCPU: 8
Memory: 61 GiB
Enhanced Networking
Price: $0.70/hour
* All pricing from us-east-1

50. Random read latency
0.075
35.1
0
5
10
15
20
25
30
35
40
1 4 8 12 16 20 24 28 32
LatencyTP90(ms)
Queue depth
Random read latency across various queue depths
Latency (TP90)
Read latency linearly increases with increase in queue depth

51. 0.075
35.1
2.09
1,865
4,152
3,851
-
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
4,500
0
5
10
15
20
25
30
35
1 4 8 12 16 20 24 28 32
LatencyTP90(ms)
Queue depth
16 KB random read IOPS, latency across various queue depths
Latency (TP90) Avg Read IOPS
IOPS
Queue depth of 1 has the lowest latency, but also has the lowest IOPS
Random read latency

52. 0.075
35.1
2.09
1,865
4,152
3,851
-
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
4,500
0
5
10
15
20
25
30
35
1 4 8 12 16 20 24 28 32
LatencyTP90(ms)
Queue depth
16 KB random read IOPS, latency across various queue depths
Latency (TP90) Avg Read IOPS
IOPS
Queue depth between 4 and 8 has the optimal IOPS and latency performance
Random read latency

53. 0.075
35.1
2.09
1,865
4,152
3,851
-
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
4,500
0
5
10
15
20
25
30
35
1 4 8 12 16 20 24 28 32
LatencyTP90(ms)
Queue depth
16 KB random read IOPS, latency across various queue depths
Latency (TP90) Avg Read IOPS
IOPS
Higher queue depths have negative impact on IOPS and latency
Random read latency

54. Random write latency
0.08
7.71
845
4,152
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
4,500
0
1
2
3
4
5
6
7
8
9
10
1 4 8 12 16 20 24 28 32
LatencyTP90(ms)
Queue depth
16 KB random write IOPS, latency across various queue depths
Latency (TP90) AvgIOPS
IOPS
Write latency queue depth and IOPS interaction is similar to that of read latency

55. Optimal queue depth to achieve lower latency and highest IOPS
is typically between 4-8; ~1 queue depth per 500 IOPS
EBS-optimized instances provide consistent latency experience
Use SSD volumes with latest-generation EC2 instances

56. 5. RAID
Increases performance, or capacity, or both
Over 320 MB/sec or 20K IOPS, striping needed
Don’t mix volume types
Typically RAID 0 or LVM stripe
Avoid RAID for redundancyEBS
EC2

57. Maximum performance per instance
How should you think about taking snapshots on a striped volume?
• Quiesce file systems and take snapshot
• Unmount file system and take snapshot
• Use OS-specific tools
12×400 GB PIOPS, pre-warmed, RAID 0 LVM, stripe size 128 KB, attached to CR1 instance

58. Use stripe size of 128 KB or 256 KB

59. 6. Pre-warming
• Eliminates first-access penalty
• Typically 5%, extreme worst case of 50% performance reduction in IOPS and latency
when volumes are used without pre-warming:
– Performance is as provisioned when all the chunks are accessed
• Recommendations before benchmarking:
– For new volumes:
• Linux: DD write
• Windows: NTFS full format
– Takes roughly an hour to pre-warm 1 TB PIOPS/General Purpose (SSD) volumes
• Always check latest documentation
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-prewarm.html

60. Use large block size to speed up your pre-warming
Example: sudo dd if=/dev/xvdf
of=/dev/xvdf conv=notrunc bs=1M

61. Final tips
• Try to use Ext4 or XFS
• Alignment can matter; check tools use 4k

62. Workload/
software
Typical block
size
Random/
Seq?
Max EBS @ 500
MB/s instances
Max EBS @
1 GB/s instances
Max EBS @ 10 GB/s
instances
Oracle DB Configurable:2 KB
–16 KB
Default 8 KB
random ~7,800 IOPS ~15,600 IOPS ~96,000 IOPS
Microsoft SQL
Server
8 KB w/ 64 KB
extents
random ~7,800 IOPS ~15,600 IOPS ~80,000 IOPS
MySQL 16 KB random ~4,000 IOPS ~7,800 IOPS ~48,000 IOPS
PostgreSQL 8 KB random ~7,800 IOPS ~15,600 IOPS ~96,000 IOPS
MongoDB 4 KB serialized ~15,600 IOPS ~31,000 IOPS ~96,000 IOPS
Apache
Cassandra
4 KB random ~15,600 IOPS ~31,000 IOPS ~96,000 IOPS
GlusterFS 128 KB sequential ~500 IOPS ~1,000 IOPS ~6,000 IOPS
Cheat sheet sample: Storage workloads on AWS

63. EBS-optimized instance
Four key components: balanced (Oh, YEAH!!)
EC2
A “boatload” of I/O
Right-sized EBS

64. Tools available for tuning
EC2 Instance: Network transfer rate (Mbps)
EBS Optimized: EC2 instance option (On/Off)
Workload: Block size, read/write ratio, serialization
Queue Depth: The number of outstanding I/Os (#)
RAID: Stripe volumes to maximize performance
Pre-warm: Eliminates first touch penalty

65. Encryption

66. Why encrypt data volumes?
Security:
Protects against someone who might gain unauthorized physical access to the volume
Can help with internal or external compliance efforts:
• Chief Information Security Officer wants encryption to protect sensitive corporate
data
• 3rd-party auditors want to see evidence that sensitive customer data is encrypted
Ease of use and operating cost reduction:
Unlike open-source or 3rd-party solutions, such as Trend Micro SecureCloud, SafeNet
ProtectV, etc., EBS encryption offers:
• “Checkbox” encryption at no extra cost
• Automated, secure key management

67. AWS KMS
A service that simplifies encrypting data and managing keys
Allows customers to create, use, and manage encryption keys from within
their own applications and supported AWS services (Amazon S3, EBS,
Amazon Redshift)
Key management functions include:
• Create, enable, disable, rotate, and define usage policy on master keys
• Generate a data key that can be exported from the service after it’s
encrypted by a master key
• Audit use of master keys in AWS CloudTrail
Available in 9 commercial regions

68. Use Create Volume in the EC2 Console

69. How AWS services integrate with AWS KMS
2-tiered key hierarchy using envelope
encryption
Unique data key encrypts customer data
AWS KMS master keys encrypt data keys
Benefits of envelope encryption:
• Limits risk of a compromised data key
• Better performance for encrypting large data
• Easier to manage a small number of master
keys than millions of data keys
Master key(s)
Data key 1
S3 object EBS
volume
Amazon
Redshift
cluster
Data key 2 Data key 3 Data key 4
Custom
application
KMS

70. Summary
Use encryption if
you need it
Take snapshotsSelect the right
instance for your
workload
Select the right
volume for your
workload

71. Q&A

72. SAN FRANCISCO