More Related Content Similar to Optimizing Network Performance for Amazon EC2 Instances (CMP308-R1) - AWS re:Invent 2018 (20) More from Amazon Web Services (20) Optimizing Network Performance for Amazon EC2 Instances (CMP308-R1) - AWS re:Invent 20182. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Optimizing Network Performance
for Amazon EC2 Instances
C M P 3 0 8
Vishvesh Sahasrabudhe
Product Manager, EC2 Networking
Nick Matthews
Principal Solutions Architect
nickpowpow
3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Network performance
Packets in Packets out
?
4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Network performance: Let’s look
Packets in Packets out
?
5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What to expect
Performance concepts
Networking performance features
• Nitro
• Elastic Network Adapter
Architectural performance
• Bandwidths and limits
Testing
6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Basic concepts
Bandwidth Latency Throughput Jitter
8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Bandwidth delay product
10 Gbps
500 milliseconds
Data
9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Bandwidth delay product
10 Gbps
500 milliseconds
1 second RTT
ACK
(To make math easy)
10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Data Data DataData
Bandwidth delay product
10 Gbps
1 second RTT
Bandwidth per second = How much
data is sent in one round trip
1. Larger packets (MTU)
2. Send more packets (TCP window size)
11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
TCP and latency
Latency is a major factor for:
• Cross-Region connectivity
• Instance-to-instance for clustering (HPC)
• Amazon ElastiCache, memcached
Tune it:
• Set larger MTUs
• TCP tuning
• Increase network efficiency
Make friends with physics:
• AWS Region choice: 10’s of ms
• Availability Zone affinity: milliseconds
• Placement groups: <1 ms
Options
12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Packets per second (PPS) and
Maximum transmission unit (MTU)
Each packet has processing overhead
• Small packets such as real time systems or transactions
Large packets increase the overall performance
• Jumbo MTU of 9001 available within VPC or VPC peers
1448 B
Payload
8949 B Payload
Jumbo MTUs increase the usable data per packet
13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
0
20
40
60
80
100
0% 2% 4% 6% 8% 10%
Loss Rate
Impact of loss on TCP throughput
14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Loss is visible as TCP retransmissions
$ netstat -s | grep retransmit
58496 segments retransmitted
52788 fast retransmits
135 forward retransmits
3659 retransmits in slow start
392 SACK retransmits failed
15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Example application
High transaction rate HTTP service
Jack Jill
16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Test setup
• 1 test server instance, 1 test client instance
• 80 ms RTT
• HTTP, not HTTPS
• 1500 MTU
• 200k requests for a 10k object
$ ab -n 200000 -c 200 http://server/10k
Goal: Minimize latency, use TCP tuning to speed it up
17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Results – HTTP application with 200k requests
Test P50 latency Avg BW
Initial congestion window—3 packets 321 ms 12.550 Mbps
Initial congestion window—10 packets 241 ms 16.765 Mbps
Initial congestion window—16 packets 161 ms 22.518 Mbps
79%
Increase!
18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Examples of TCP tuning
Initial congestion window:
ip route change 10.16.16.0/24 dev eth0
proto kernel scope link initcwnd 16
TCP receive window:
sysctl net.ipv4.tcp_rmem
TCP congestion control algorithms:
net.ipv4.tcp_available_congestion_control
TCP retransmission timer:
ip route change 10.16.16.0/24 dev eth0 proto kernel
scope link rto_min 50ms
19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Nitro system
• Lightweight hypervisor
• Nitro card
• Nitro security chip
Benefits of Nitro – Enhanced networking
• Over 20x increase in PPS performance
• Reduces instance-to-instance latencies
• Support SR-IOV with PCIe passthrough, to get the hypervisor out of the way and for consistent
performance
• Intel ixgbvef and Amazon Elastic Compute Cloud (Amazon EC2) Elastic Network Adapter (ENA)
Networking on AWS
21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of Enhanced Networking
(Latency)
0
50
100
150
200
250
300
1 2
RoundTripLatency(µs)
Series1 Series2 Series3 Series4
22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Evolution of Nitro
CC2 instance (pre-Nitro)
Xen hypervisor
EC2
Instance
EC2
Instance
EC2
instanceXen
Dom0
23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Evolution of Nitro
C3 instance (2013)
Xen hypervisor
EC2
Instance
EC2
Instance
EC2
instanceXen
Dom0 ixgbevf
Intel 82599
24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Evolution of Nitro
C4 instance (2015)
Xen hypervisor
EC2
Instance
EC2
Instance
EC2
instanceXen
Dom0 ixgbevf
Intel 82599
EBS Optimized
25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Evolution of Nitro
C5 instance (2017)
Nitro hypervisor
EC2
Instance
EC2
Instance
EC2
instance
ENA
26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Evolution of Nitro
bare metal instances (2018)
EC2
instance
27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
ENA driver capabilities
• Future proof: support for up to 400 Gbps
• Increased queues per device
• Lower latency and jitter
• Broad operating system AMI support
• Linux—Amazon Linux, Amazon Linux 2, RHEL 7.4+, SLES 12+, Ubuntu 14.04, 16.04, 18.04,
CentOS 7 (1711), and in the Linux kernel since 4.9
• Windows Server 2008 R2, 2012, 2012 R2, 2016, 2019
• FreeBSD 11
• DPDK support starting in 16.04, PMD support
• Open Source: https://github.com/amzn/amzn-drivers
ENA v2.0 – Latest release in upstream kernel
• Low Latency Queues – Improved average and tail latencies
• Rx Checksum Offload – Reduces host CPU usage
28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of Nitro enhanced networking
(latency)
0
50
100
150
200
250
300
1 2
Series1 Series2 Series3 Series4 Series5
29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of Nitro enhanced networking
(Network and Storage bandwidth)
0
20
40
60
80
100
120
1 2 3 4
Gbps
Series1 Series2
30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of Nitro enhanced networking
(instance bandwidth)
T3 instances – Up to 5 Gbps network
performance
Smaller sizes of C5, M5, R5 - up to 10
Gbps network performance
Larger instance sizes have sustained
10 or 25 Gbps
NEW – Smaller sizes of C5n – up to 25
Gbps network performance
NEW – C5n instances have sustained
50 or 100 Gbps
Model vCPU
Mem
(GiB)
Network
Performance
(Gbps)
c5.large 2 4 Up to 10
c5.xlarge 4 8 Up to 10
c5.2xlarge 8 16 Up to 10
c5.4xlarge 16 32 Up to 10
c5.9xlarge 36 72 10
c5.18xlarge 72 144 25
Model vCPU
Mem
(GiB)
Network
Performance
(Gbps)
t3.nano 2 6Up to 5
t3.micro 2 12Up to 5
t3.small 2 24Up to 5
t3.medium 2 24Up to 5
t3.large 2 36Up to 5
t3.xlarge 4 96Up to 5
t3.2xlarge 8 192Up to 5
Model vCPU
Mem
(GiB)
Network
Performance
(Gbps)
c5n.large 2 4 Up to 25
c5n.xlarge 4 8 Up to 25
c5n.2xlarge 8 16 Up to 25
c5n.4xlarge 16 32 Up to 25
c5n.9xlarge 36 72 50
c5n.18xlarge 72 144 100
31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
• Microservices architecture – 100s of microservice
• Request cache – persist data for request duration
• 3 zones/region – read local, write regional
• Millions of requests per second requiring high packet rate performance at
cache layer
• Average read latency < 250 µS
High performance caching
32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
How did Netflix benefit?
Migration to Nitro instances
• M4 to C5
Improved PPS lead to reduction in instance count
• 10 m4.large 1 c5.xlarge
Improved latency performance while reducing costs
33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AMI Check
• ixgbevf enhanced networking:
aws ec2 describe-image-attribute --image-id ami_id
--attribute sriovNetSupport
• ENA enhanced networking:
aws ec2 describe-image-attribute --image-id ami_id
--attribute enaSupport
ENA support is required for AMIs launching on Nitro based instance
types (C5, M5, R5, T3…)
34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Beyond enhanced networking with DPDK
Intel Data Plane Development Kit (DPDK) is a set of C libraries and
drivers for fast packet processing
DPDK provides:
• Lower latency due to kernel and TCP stack bypass
• More control of packet processing
• Lower CPU overhead
35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Non-DPDK application
Nitro hypervisor
EC2
Instance
EC2
Instance
Instance
Kernel
App
36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DPDK application
Nitro hypervisor
EC2
Instance
EC2
Instance
Instance
Kernel
DPDK app
37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
DPDK on AWS
Supported on enhanced networking instances:
• Intel ixgbevf and AWS Elastic Network Adapter (ENA)
DPDK can reduce latency while increasing control of packet
processing
• Kernel and TCP stack bypass, and PMD instead of interrupts reduce
latency
• DPDK does not change any underlying limits or maximums
38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Network performance scenarios
The sources and destinations matter:
• Within a placement group
• Over a VPN connection
• Using AWS Direct Connect
• Using Elastic Load Balancing
• Accessing AWS services such as Amazon Simple Storage
Service (Amazon S3)
40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
VPC Performance
No limits per VPC
Availability Zone Availability Zone
No limits for any
Availability Zone or subnet
IGW
No limits for any Internet
Gateway
Bandwidth Limits: NAT gateway
Each NAT Gateway can
sustain 10 Gbps, and
scales horizontally
beyond 10 Gbps
41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
VPC Peering Performance
No limits for VPC peering within a Region
Availability Zone Availability Zone
Internet gateway
Bandwidth Limits:
Availability Zone Availability Zone
Internet gateway
VPC peering
42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Placement group
Instance Bandwidth Limits
Availability Zone B
Availability Zone A
Amazon S3
Placement Group – 10 Gbps
flows
Placement group:
5 Gbps aggregate per instance
with 5 Gbps flows
25 Gbps
25Gbps
Everything else:
Between instances:
25 Gbps within the VPC or peered
VPC
To Amazon S3:
25 Gbps to VPC Endpoints and
public IPs in the same Region
43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Virtual private gateway performance
~1.25 Gbps aggregate per VPN tunnel
Availability Zone
Bandwidth limits:
Virtual private
gateway
Virtual private network (VPN)
AWS Direct Connect
AWS Direct Connect is only limited
by physical port speeds
44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Virtual private gateway performance
5 Gbps limit for egress traffic
per instance
Availability Zone
Bandwidth limits:
Virtual private
gateway
Virtual private network (VPN)
AWS Direct Connect
45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Aviatrix Insane Mode
25G Forwarder
instance
VPN
ENI Route
Up to 5 Gbps Up to 5 Gbps
Cumulatively
up to 25 Gbps
High-speed VPN
C5 PPS improvements
• Forwarder performance
46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Network Load Balancer performance
Auto Scaling Group
Network Load
Balancer
Network Load Balancer
• L4 load balancer
• TCP
• High performance
• Availability Zone locality
• One IP per Availability Zone
Bandwidth limits:
Several gigabits of capacity at
launch
Scales horizontally
47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS HyperPlane performance
AWS Region
VPC A VPC B VPC A VPC B VPC A VPC B
AWS HyperPlane
Attachments
48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Distributed load testing (BWMG), executed in a loop
bees attack --url '<NLB>' --number 10000000 --concurrent 100000 –keepalive
Performance graph shows no errors, and content was served fine
49. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
50. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of testing
Many potential performance factors:
• Flow location, type, size
• Instance family, type
• Application profile
Packet sizes, protocols, CPU/memory/disk usage
AWS recommendation: test it yourself
51. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Performance testing: How?
With your application and environment
• Load testing
• Transaction testing
• “Race track numbers”
Benchmark testing
• Scenario testing: intra-VPC, internet traffic, VPN
• Bandwidth, latency, jitter, throughput, packets per second
52. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Testing tips
Use multiple flows
#iperf –c 192.168.1.1 –P 10
Try UDP as well
#iperf –c 192.168.1.1 –u –p <port number>
Distributed testing tools
• https://github.com/newsapps/beeswithmachineguns
Network benchmark testing
• https://aws.amazon.com/premiumsupport/knowledge-
center/network-throughput-benchmark-linux-ec2/
Operating system configuration for enhanced networking
• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/en
hanced-networking-os.html
53. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Summary
Understand latency and TCP dynamics
• Tune and reduce latency
Understand what AWS offers to help
• Enhanced networking, jumbo MTU, instance families
Use the right architecture
• Know the limits, distribute the load
Test it out
54. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Nick Matthews
@nickpowpow
Vishvesh Sahasrabudhe
55. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.