This advanced technical session is ideal for customers that are looking to maximise the performance of AWS Elastic Block Store (EBS) storage to support workloads with demanding IO performance requirements. If you need to run high IO workloads on EBS such as NoSQL or RBDMS systems then attend this session to find out how to optimise your EBS configuration to enable this.

1. © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Maximizing Amazon EC2
& Amazon EBS Performance
Carlos Conde – Head of AWS Evangelism EMEA

2. A “Normal” Hard Drive

3. A few definitions…

4. So.. what storage workloads?

5. What is Amazon EBS?
Very flexible service with lots of choice
– Used with Amazon EC2 instances
– Attach/detach/copy/delete volumes
– Point-in-time snapshots of volumes -> Amazon S3
– Automatically replicated within its Availability Zone to protect
from component failure
– Paying a low price for only what you provision

6. EBS =

7. EC2 EBS

8. Amazon EBS Standard
Amazon Elastic
Block Storage
(EBS)
IOPS: ~100 IOPS steady-state, with best-effort bursts
Throughput: variable by workload, best effort to 10s of MB/s.
Latency: Varies, reads typically <20 ms writes typically <10 ms
Capacity: As provisioned, up to 1 TB

9. EBS STANDARD
≠
EBS PIOPS

10. EBS PIOPS
Amazon Elastic
Block Storage
(EBS)
IOPS: Within 10% of up to 4000 IOPS,
99.9% of a given year, as provisioned.
Throughput: 16 KB per I/O = up to 64 MB/s, as provisioned.
Latency: low and consistent. Second / IOPS
Capacity: As provisioned, up to 1 TB
*

11. ❶ Select a new type of Provisioned IOPS volume
❸ Specify the number of I/O operations per
second your application needs, up to 4000
IOPS per volume. The volume will deliver the
specified I/O operations per second.
❷ Specify the volume capacity

12. I/O Characteristics
• I/O size
– 4 KB to 64 MB
• I/O pattern
– Sequential and random
• I/O type
– Read and write
• PIOPS always measures I/O in terms of
16 KB or smaller
• PIOPS delivers same number of IOPS for
sequential and random I/O
• PIOPS delivers same number of IOPS for
reads or writes
PIOPS is optimized for database workloads

13. EC2 EBS
16 KBytes

14. A bigger I/O
EBS
>16 KB? = more than one I/O
EC2

15. EC2
A smaller I/O
EBS
< 16 KB? = no “bonus” random I/Os

16. EC2
An I/O
EBS
Once it’s gone, it’s gone.

17. EC2
an I/O
EBS
Just because EC2 sends more work
doesn’t mean there’s enough
IOPS to handle it!

18. EC2 EBS
Just because Amazon EC2 sends more
work doesn’t mean there’s enough
bandwidth to handle it!

19. EC2
An I/O
Without more bandwidth,
more EBS volumes or higher PIOPS won’t help!

20. EC2
An I/O
EBS
Adding bandwidth alone
or increasing instance size won’t help!
EBS-Optimized

21. USE EBS PIOPS
WITH
EBS-OPTIMIZED EC2 INSTANCES

22. BALANCE
IOPS / BANDWIDTH / THROUGHPUT

23. EBS-Optimized
EC2

24. EBS-Optimized
Network interference tests:
No impact on IOPS or
Amazon EBS throughput
Row Labels AvgBW AvgIOPs
m3.2xlarge (EBS-optimized)
no network load
random
read 57,542 3,596
write 61,713 3,857
rw (70/30) 66,997 4,186
sequential
read 61,708 3,856
write 61,651 3,853
rw (70/30) 66,996 4,187
with network load-test1
random
read 59,835 3,739
write 63,407 3,962
rw (70/30) 68,859 4,303
sequential
read 61,736 3,858
write 63,360 3,959
rw (70/30) 68,859 4,302

25. Architecture – What Else Uses Network?

26. Architecting for Performance
IOPS consistency requires
EBS-optimized instances
Maximum throughput delivered by
Amazon EBS is limited by Amazon
EC2 bandwidth
EBS throughput =
EBS IOPS × Block size
Ex: 64 MB/s = 4000 IOPS × 16 KB
Instance
vCPU
EBS
Optimized
Max MB/s
Max 16k IOPS
t1 micro
1
No
32MB/s
2000
m1.small
1
No
64MB/s
4000
m1.medium
1
No
64MB/s
4000
m1.large
2
Yes
64MB/s
4000
m1.xlarge
4
Yes
128MB/s
8000
m3.xlarge
4
Yes
64MB/s
4000
m3.2xlarge
8
Yes
128MB/s
8000
c1.medium
2
No
32MB/s
2000
c1.xlarge
8
Yes
128MB/s
8000
cc2.8xlarge
32
NA
800MB/s
50,000
m2.xlarge
2
No
64MB/s
4000
m2.2xlarge
4
Yes
64MB/s
4000
m2.4xlarge
8
Yes
128MB/s
8000
cr1.8xlarge
32
NA
800MB/s
50,000
hi1.4xlarge
16
NA
800MB/s
50,000
cg1.4xlarge
16
NA
800MB/s
50,000

27. Smaller I/O (4 KB, 16 KB)
• Why are 4 KB I/O size in
sequential operations
driving greater than 4000
IOPS?
• Why is m1.large and
m3.xlarge IOPS at 16 KB
less than 4000 IOPS?
• Database needs 5000 ops/
second. How many IOPS do
I need to provision?
• What happens when
customers want to burst
beyond provisioned IOPS?
IOPS and BW
performance at QD
8
m1.large M3.xlarge m3.2xlarge
AvgIOPs
Avg BW
( KB) AvgIOPs
Avg BW
( KB) AvgIOPs
Avg
BW
( KB)
Write
sequential
4K 4146 16,587 5997 23,990 7767 31,068
16K 3712 59,402 4157 55,461 4153 60,332
Write random
4K 4082 16,329 4433 17,733 4178 16,712
16K 3713 59,422 3743 53,813 4153 60,332
Read
sequential
4K 5301 21,205 9232 36,929 13450 53,802
16K 3533 56,535 4796 56,824 4153 60,332
Read random
4K 4538 18,154 5864 23,457 4177 16,711
16K 3510 56,168 3583 51,246 4153 60,332
Results for 400 GB volume with 4000 IOPS at QD 8; EBS-optimized instances

28. Larger I/O (128 KB, 512 KB)
• Why am I seeing only 462
IOPS on a volume?
• Why there is no difference
in performance for
random and sequential
workloads?
• How should I configure
500 MB/s read or write
throughput using PIOPS
volumes
IOPS and BW
performance at QD 8
m1.large M3.xlarge m3.2xlarge
AvgIOPs
Avg
BW(KB) AvgIOPs
Avg
BW(KB) AvgIOPs
Avg
BW(KB)
Write sequential
128K 462 59,268 462 59,145 522 66,843
512K 115 59,292 115 59,278 130 66,804
Write random
128K 462 59,265 462 59,241 522 66,843
512K 115 59,291 115 59,272 130 66,843
Read sequential
128K 455 58,240 454 58,225 522 66,843
512K 113 58,003 114 58,589 130 66,843
Read random
128K 455 58,236 454 58,215 522 66,843
512K 113 57,960 114 58,496 130 66,805
4000, 16 KB read/write per second, or 2000 32
KB read/write per second, or 1000 64 KB read/
write per second…
Results for 400 GB volume with 4000 IOPS at QD 8

29. Write Latency
• Database applications care
about latency as much as IOPS
delivered
• There is an Interdependency
among IOPS, queue depth, and
latency
• Current guidance is queue
depth of 1 for every 200 IOPS,
but if latency-bound and write-
heavy, 1:500 – 1:1000 is better.
1 4 8 12 16 20 24 28 32
AvgIOPS ( Count) 845 4152 4153 4177 4152 4176 4177 4177 4151
AvgTP90 ( ms) 3.13 1.47 2.03 3.56 3.62 5.54 6.18 7.48 7.71
845
4152
3.13
1.47
2.03
3.56 3.62
5.54
6.18
7.48
7.71
0
1
2
3
4
5
6
7
8
9
0
500
1000
1500
2000
2500
3000
3500
4000
4500
WriteIOPS
16 KBk random WRITE- M3.2Xlarge EBS-
optimized
L
a
t
e
n
c
y
QD

30. Read Latency
• Reads can take advantage of a
deeper queue
• Current guidance is queue
depth of 1 for every 250 IOPS
• EBS-optimized provides
predictable latency
1 4 8 12 16 20 24 28 32
AvgIOPS ( Count) 1864 4153 4153 4177 4120 2800 1965 1213 1089
AvgTP90 ( ms) 0.68 1.46 2.15 3.43 3.88 5.18 91.14 93.18 93.70
1864
4153
4120
1965
0.68 1.46 2.15 3.43 3.88 5.18
91.14
93.18 93.70
0
10
20
30
40
50
60
70
80
90
100
0
500
1000
1500
2000
2500
3000
3500
4000
4500
ReadIOPS
16 KB random READ - M3.2Xlarge EBS-optimized
L
a
t
e
n
c
y
QD

31. • Performance requirements may be driven by IOPS or latency or both
• Recommendation is to start with queue depth of 4 and tune based on
IOPS and latency requirement
– Some customers may need lowest possible latency; this can be achieved at queue
depth of 1 or 2
• Very high queue depths ( >24) may decrease IOPS count as well as
increase latency
Architecting for Performance: Latency

32. • Typically 5%, extreme worst case of 50% performance
reduction in IOPS and latency when volumes are used without
pre-warming
– Performance is as provisioned when all the chunks are accessed
• Recommendation if testing or you have spare setup time:
– Write to every 4 MB block before using new volumes
• Linux: DD
• Windows: NTFS Full format
– Takes roughly an hour to pre-warm 1TB 4KB PIOPS volume
– Be warned, can take up to a day for a 1 TB standard EBS volume
Pre-warming EBS volumes

33. What about RAID?
Amazon Elastic
Block Storage
(EBS)
Amazon Elastic
Compute Cloud
(EC2)
• Don’t do
RAID 5, 6
Amazon Elastic
Block Storage
(EBS)
Amazon Elastic
Compute Cloud
(EC2)

34. Architecting for Performance: RAID
Stripe number of volumes to
drive higher IOPS and
throughput
– RAID 0 or RAID 10
Taking snapshots on a striped
volume:
– Quiesce file systems and
take snapshot
– Unmount file system and
take snapshot
– Use OS-specific tools
12×400 GB PIOPS volumes, pre-warmed,
RAID 0 LVM, Stripe size 64 KB, attached to
CR1 instance
IO
Pa&ern
Block
Sizes
Thread
Write
IOPS
Write
BW
(MB)
Read
IOPS
Read
BW
(MB)
Sequen>al
4K
8
33,500
134
48,250
193
16K
8
13,875
222
48,063
769
1M
1
247
247
823
823
Random
4K
8
35,250
141
48,250
193
16K
8
13,875
222
42,125
674
1M
1
496
496
795
795

35. Workload/
Software
Typical block
size
Random
/Seq?
Typical
correct
EBS type
Max EBS @
500 MB/s
Max EBS @
1 GB/s
Max EBS @ CC/H
Oracle DB Configurable:2 KB –
16 KB
Default 8 KB
random PIOPS RAID 0, 2 × 4000
PIOPS:
~7800 IOPS
RAID 0, 4 × 4000 PIOPS:
~15,600 IOPS
RAID 0, 24 × 4000 PIOPS:
~96,000 IOPS
Microsoft
SQL Server
8 KB w/ 64 KB
extents
random PIOPS RAID 0, 2 × 4000
PIOPS:
~7800 IOPS
RAID 0, 4 × 4000 PIOPS:
~15,600 IOPS
RAID 0, 20 × 4000 PIOPS:
~80,000 IOPS
MySQL 16 KB random PIOPS 1 x 4000 PIOPS:
~4000 IOPS
RAID 0, 2 × 4000 PIOPS:
~7,800 IOPS
RAID 0, 12 × 4000 PIOPS:
~48,000 IOPS
PostgreSQL 8 KB random PIOPS RAID 0, 2 × 4000
PIOPS:
~7,800 IOPS
RAID 0, 4 × 4000 PIOPS:
~15,600 IOPS
RAID 0, 24 × 4000 PIOPS:
~96,000 IOPS
IBM DB/2 Configurable:2 KB
–-256 KB
Default 32 KB
random PIOPS 1 x 4000 PIOPS:
~2000 IOPS
RAID 0, 4 × 4000 PIOPS:
~8,000 IOPS
RAID 0, 12 × 4000 PIOPS:
~24,000 IOPS
NuoDB 8 KB random PIOPS RAID 0, 2 × 4000
PIOPS:
~7800 IOPS
RAID 0, 4 × 4000 PIOPS:
~15,600 IOPS
RAID 0, 24 × 4000 PIOPS:
~96,000 IOPS
Clustrix 4 KB random PIOPS RAID 0, 4 × 4000
PIOPS:
~15,600 IOPS
RAID 0, 8 × 4000 PIOPS:
~31,000 IOPS
RAID 0, 24 × 4000 PIOPS:
~96,000 IOPS
GenieDB 16 KB random PIOPS 1 × 4000 PIOPS:
~4000 IOPS
RAID 0, 2 × 4000 PIOPS:
~7,800 IOPS
RAID 0, 12 × 4000 PIOPS:
~48,000 IOPS
NewSQLRDBMS
Cheat Sheet! Storage Workloads on AWS

36. Workload/
Software
Typical
block
size
Random
/Seq?
Typical
correct
EBS type
Max EBS @
500 MB/s
Max EBS @ 1 GB/s Max EBS @ CC/H
MongoDB 4 KB serialized PIOPS RAID 0, 4 × 4000
PIOPS:
~15,600 IOPS
RAID 0, 8 × 4000
PIOPS:
~31,000 IOPS
RAID 0, 24 × 4000
PIOPS:
~96,000 IOPS
Cassandra 4 KB random PIOPS RAID 0, 4 × 4000
PIOPS:
~15,600 IOPS
RAID 0, 8 × 4000
PIOPS:
~31,000 IOPS
RAID 0, 24 × 4000
PIOPS:
~96,000 IOPS
CouchBase 4 KB random PIOPS RAID 0, 4 × 4000
PIOPS:
~15,600 IOPS
RAID 0, 8 × 4000
PIOPS:
~31,000 IOPS
RAID 0, 24 × 4000
PIOPS:
~96,000 IOPS
Hadoop /
Hbase
64 KB random PIOPS 1 × 4000 PIOPS:
~1000 IOPS
RAID 0, 2 × 4000
PIOPS:
~2000 IOPS
RAID 0, 12 × 4000
PIOPS:
~12,000 IOPS
Riak /
LevelDB
4 KB random PIOPS RAID 0, 4 × 4000
PIOPS:
~15,600 IOPS
RAID 0, 8 × 4000
PIOPS:
~31,000 IOPS
RAID 0, 24 × 4000
PIOPS:
~96,000 IOPS
NoSQL
Cheat Sheet! Storage Workloads on AWS

37. Cheat Sheet! Storage Workloads on AWS
Workload/
Software
Typical
block
size
Random
/Seq?
Typical
correct
EBS type
Max EBS @
500 MB/s
Max EBS @ 1 GB/s Max EBS @ CC/H
AeroSpike 128 KB random PIOPS 1 × 4000
PIOPS:
~500 IOPS
RAID 0, 2 x 4000
PIOPS:
~1000 IOPS
RAID 0, 12 x 4000 PIOPS:
~6,000 IOPS
(768MB/s)
Vertica 1 MB sequential PIOPS 1 × 4000
PIOPS:
~68 IOPS
RAID 0, 2 × 4000
PIOPS:
~125 IOPS
RAID 0, 12 × 4000 PIOPS:
~750 IOPS
(768MB/s)
TeraData 128 KB sequential PIOPS 1 × 4000
PIOPS:
~500 IOPS
RAID 0, 2 × 4000
PIOPS:
~1000 IOPS
RAID 0, 12 × 4000 PIOPS:
~6,000 IOPS
(768MB/s)
GlusterFS 128 KB sequential PIOPS 1 × 4000
PIOPS:
~500 IOPS
RAID 0, 2 × 4000
PIOPS:
~1000 IOPS
RAID 0, 12 × 4000 PIOPS:
~6,000 IOPS
(768MB/s)
OrangeFS 256 KB sequential PIOPS 1 × 4000
PIOPS:
~250 IOPS
RAID 0, 2 × 4000
PIOPS:
~500 IOPS
RAID 0, 12 × 4000 PIOPS:
~3,000 IOPS
(768MB/s)
Riak (CS) 1 MB sequential PIOPS 1 × 4000
PIOPS:
~68 IOPS
RAID 0, 2 × 4000
PIOPS:
~125 IOPS
RAID 0, 12 × 4000 PIOPS:
~750 IOPS
(768MB/s)
DNFileSystemWarehouse

38. Performance – Extra-large Production Scale
• Leverage SSD instance type
(hi1.4xlarge)
o 2 × 1 TB SSD storage (ephemeral
storage)
o Perfect for replicas
• If replicas on SSD instance types, disable
integrity features such as fsync and
full_page_writes on those hosts to
improve performance

39. STABLE
Testing Random 4 KB Reads
EBS PIOPS+ SSD

40. What about Capacity Cost?
cc2.8xlarge
48x
1TB
EBS
VS.
hs1.8xlarge hs1.8xlarge
$7312 on-demand,
$6128 effective 3 YR reserved
$6734 on-demand,
$2408 effective 3 YR reserved
If >43TB, or > 800MB/s, choose hs1
If 3 year, and >18TB, choose hs1

41. Performance / Stability Tips
• Ext4 or XFS (understand journal impact!)
• nobarrier, noatime, noexec, nodiratime
• Raise file descriptor limits
• Set read-aheads low
• Amazon CloudWatch metrics
• SNAPSHOT SNAPSHOT SNAPSHOT

42. From 1000 to 98,000: What does that mean?

43. So.. all storage workloads!

44. © 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.
Maximizing Amazon EC2
& Amazon EBS Performance
Carlos Conde – Head of AWS Evangelism EMEA
Thank you!