2. Tempesta Technologies
Custom software development in:
●
high performance network traffic processing
e.g. WAF mentioned in Gartner magic quadrant
https://www.ptsecurity.com/ww-en/products/af/
●
Databases
e.g. MariaDB SQL System-Versioned Tables
https://mariadb.com/kb/en/library/system-versioned-tables/
https://mariadb.com/conference/session/querying-data-previous-
point-time
Developing Tempesta FW – open source Linux
Application Delivery Controller (ADC)
3. Tempesta FW:
Application Delivery Controller (ADC)
https://www.netdevconf.org/2.1/session.html?krizhanovsky
HTTP(S) reverse proxy
filtering
●
HTTP DDoS mitigation
●
Web Application Firewall
built into the TCP/IP stack
up to 1.8M HTTP RPS
on 4 cores
4. Disclamer
We’re sceptic about QUIC…
https://github.com/tempesta-tech/tempesta/issues/724
...but I did my best to figure out why QUIC is good
The talk isn’t about QUIC benefits, there are many other talks (see
references)
...instead it’s about how does it work
Not a comprehensive description…
...instead, just how to learn and debug the protocol
5. Why QUIC?
QUIC is ~7% of Internet traffic, 98% of them to Google
Has QUIC: Google, Amazon, Fastly, LiteSpeed Technologies
Adopting: CloudFlare, Mellanox (UDP offload on NICs)
https://www.netdevconf.org/0x12/session.html?udp-segmentation-offload
Middleboxes slowly learn about QUIC
Highlights:
●
Performance: no OSI layers - each layer knows about each other
●
A UDP-based TCP replacement
●
no head-of-line blocking
●
0-RTT handshakes
6. Current state
Still in draft state (IETF meeting 103, Nov 6-7, 2018)
$ grep TBD *.txt|wc -l
23
Several server implementations
Chrome seems the only usable client
Wireshark knows about QUIC
7. Head-of-line blocking (the long story)
HTTP/2 solves HTTP/1 HoL blocking
...so no need many TCP connections
...so 1 TCP connection introduces HoL
blocking
...so multi-stream QUIC replaces TCP
BTW: SCTP solves HoL problem for 11 years
https://en.wikipedia.org/wiki/Stream_Control_Tr
ansmission_Protocol#Features
SCTP is implemented by many libraries and OS
kernels
8. Why not SCTP and/or DTLS?
QUIC FAQ for Geeks
https://docs.google.com/document/d/1lmL9EF6qKrk7gbazY8bIdvq3Pno2
Xj_l_YShP40GLQE/edit
SCTP and DTLS were not designed to minimize latency, and this is
significantly apparent even during the connection establishment phases.
Several of the techniques that QUIC is experimenting with would be
difficult technically to incorporate into existing standards. As an
example, each of these other protocols require several round trips to
establish a connection, which is at odds with our target of 0-RTT
connectivity overhead.
Middleboxes alre also against updating standards
9. Why not TCP Fast Open + TLS 1.3?
TLS 1.3 (used by QUIC anyway): 0/1-RTT handshakes
TCP Fast Open (RFC 7413, default in Linux): 0-RTT
●
does not detect duplicate SYN segments (RFC 7413 6.1)
●
1st
data segment size <= MSS (RFC 7413 6.2)
●
TCP HoL blocking still exists
source: https://lwn.net/Articles/508865/
11. QUIC & TLS 1.3
draft-ietf-quic-applicability-03.txt
draft-ietf-quic-tls-16.txt: ”Rather than a strict layering,
these two protocols are co-dependent”
TLS record = QUIC packet (no need for TLS dynamic records)
12. QUIC in the wild
Chrome, ver. >= 63
google-chrome
--enable-quic
# tcpdump -i wlp1s0 -X -s0 -nn -vvv
udp port 443 and host www.google.com
17. Connection upgrade: HTTP header Alt-Svc
draft-ietf-quic-http-16.txt, RFC 7838
ALTSVC HTTP/2 frame also can be used
Format seems changing
"headers": [
":status: 200",
"accept-ranges: bytes",
"content-type: image/png",
...
"alt-svc: quic=":443"; ma=2592000; v="44,43,39,35""
],
18. Fallback to TCP
Some middleboxes frop UDP (draft-ietf-quic-applicability-03.txt)
# iptables -A OUTPUT -p udp --dport 443 -j DROP
# tcpdump -i wlp1s0 -nn -q host www.google.com
192.168.1.67.54512 > 64.233.165.103.443: tcp 78
192.168.1.67.54512 > 64.233.165.103.443: tcp 46
…
19. Packet headers
Long header – to establish connection contexts
●
Initial, handshake, retry
Short header – after that
Version negotiation – for unsupported version in ClientHello
20. QUIC long header
{D,S}CIL = {Destination,Source} Connection ID Length
Version=0: version negotiation w/ list of supported versions
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+
|1| type(7) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Version (32) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|DCIL(4)|SCIL(4)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Destination Connection ID (0/32..144) ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Source Connection ID (0/32..144) ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length (i) ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Packet Number (8/16/32) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
21. QUIC short header
Connection ID
●
Survive NAT rebindings of UDP ports
●
Connection migration (draft-deconinck-quic-multipath-01.txt)
Bits description: “this section should be removed”
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+
|0|K|1|1|0|R R R|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Destination Connection ID (0.144) ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Packet Number (8/16/32) ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
24. QUIC handshake
Crypto tags
●
CCS – Common Certificate Set
●
AEAD – authentication & encryption algorithm
●
KEXS – key exchange method
●
…and many others
25. QUIC: 0-RTT resumption
draft-ietf-quic-applicability-03.txt: 2 data copies are possible (~TFO)
0-RTT (draft-ietf-quic-tls-16.txt)
●
Protection with earlier or handshake keys
ClientHello
(0-RTT Application Data) -------->
ServerHello
{EncryptedExtensions}
{Finished}
<-------- [Application Data]
{Finished} -------->
[Application Data] <-------> [Application Data]
26. Streams
A “message” abstraction
Separate control streams from application data streams
A packet may include frames for different streams
A UDP datagram may include several packets
Length-encoded integers (in most significant bits)
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+
|Stream Type (8)|
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length (i) ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type (8) | Frame Payload (*) ...
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
27. Stream life
Somewhat close to TCP connections
RST → RST_STREAM frame
FIN → CONNECTION_CLOSE or
APPLICATION_CLOSE frames
PING → keepalive probe
BLOCKED,STREAM_BLOCKED →
zero window announcement
o
| Create Stream (Sending)
| Create Bidirectional Stream (Receiving)
v
+-------+
| Ready | Send RST_STREAM
| |-----------------------.
+-------+ |
| |
| Send STREAM / |
| STREAM_BLOCKED |
| |
| Create Bidirectional |
| Stream (Receiving) |
v |
+-------+ |
| Send | Send RST_STREAM |
| |---------------------->|
+-------+ |
| |
| Send STREAM + FIN |
v v
+-------+ +-------+
| Data | Send RST_STREAM | Reset |
| Sent |------------------>| Sent |
+-------+ +-------+
| |
| Recv All ACKs | Recv ACK
v v
+-------+ +-------+
| Data | | Reset |
| Recvd | | Recvd |
+-------+ +-------+
29. HTTP headers compression
QPACK (~HPACK in HTTP/2, draft-ietf-quic-qpack-03.txt)
●
Shares Huffman encoding tables among asynchronous streams
●
Static & dynamic tables
●
Uses designated stream for the synchronization
30. QUIC: packet loss
draft-ietf-quic-recovery-16.txt
RTO, TLP, Fast & early retransmit, {S,F}ACK loss
recovery
ACK frames: for packets, retransmitted packet has a
new number
●
retransmitted frames with the same offset and
length (like TCP)
●
like TCP: delayed piggybacked ACKs
●
Like SACK: ACK blocks (no reneging)
Explicit Congestion Notification (ECN) [RFC3168]
32. QUIC: congestion control
Chromium is a monster: BBR, CUBIC, PRR, slow start etc.
draft-ietf-quic-recovery-16.txt
NewReno (cwnd in bytes), slow start
Packets pacing
●
Packetization delays to bundle multiple frames
Frames
MAX_DATA, MAX_STREAM_DATA
33. Security considerations
Volumetric DDoS: opaque UDP traffic just like UDP flood
●
Middlebox filtration: ClientHello + Connection ID tracking
Aplification DDoS: minimal packet length for ClientHello
Stream fragmentation & reassembly → memory overcommit
34. QUIC in the kernel
User-space for rapid prototyping
The sendfile() problem (solved by kTLS for TLS)
● setsockopt(sd, SOL_UDP, UDP_ULP, "quic", sizeof("quic") - 1);
setsockopt(sd, SOL_QUIC, QUIC_CTX, &quic_ctx, sizeof(quic_ctx));
●
recvmsg() / sendmsg() - read/write frames vector
High CPU usage (ACK & Ko copies to user-space)
System wide memory accounting for all processes and connections
NIC acceleration (crypto offload, UDP segmentation offload)
First simple implementation (TBD): handshakes are in user-space
36. References: documents
Current IETF drafts: https://datatracker.ietf.org/wg/quic/
QUIC Working Group: https://quicwg.org/
Discussions & open issues: https://github.com/quicwg/base-drafts/issues
Known implementations:
https://github.com/quicwg/base-drafts/wiki/Implementations
37. References: good to read & watch
“The QUIC Transport Protocol:Design and Internet-Scale Deployment”,
https://static.googleusercontent.com/media/research.google.com/en//pubs/archiv
e/46403.pdf
“QUIC: Developing and Deploying a TCP Replacement for the Web”,
https://www.netdevconf.org/0x12/session.html?quic-developing-and-deploying-a-
tcp-replacement-for-the-web
QUIC FAQ for Geeks,
https://docs.google.com/document/d/1lmL9EF6qKrk7gbazY8bIdvq3Pno2Xj_l_Y
ShP40GLQE/edit
https://en.wikipedia.org/wiki/QUIC
Thoughts on how to support QUIC in (lib)curl,
https://github.com/curl/curl/wiki/QUIC