Enabling Googley microservices with HTTP/2 and gRPC.

Alex Borysov
Software Engineer
Enabling Googley microservices with HTTP/2 and gRPC.
A high performance, open source, HTTP/2-based RPC framework.

Google Cloud Platform
Alex Borysov
• Software engineer / technical leader experienced in large
scale software development.
• 10+ years of software engineering experience.
• Active gRPC user.
Software Engineer @ Google
@aiborisov

Tens of billions requests
per second.
Google Scale

Why gRPC?
Photo taken by Andrey Borisenko.

gRPC vs JSON/HTTP for Google Cloud Pub/Sub
Publishing 50KB messages at
maximum throughput from a
single n1-highcpu-16 GPE VM
instance, using 9 gRPC channels.
More impressive than the
almost 3x increase in
throughput, is that it took only
1/4 of the CPU resources.
11x difference per CPU3x increase in throughput
https://cloud.google.com/blog/big-data/2016/03/announcing-grpc-alpha-for-google-cloud-pubsub

What is gRPC?
gRPC stands for gRPC Remote Procedure Calls.
A high performance, open source, general purpose
standards-based, feature-rich RPC framework.
Open sourced version of Stubby RPC used in Google.
Actively developed and production-ready, current
version is 1.0.1.

1990s vs 2016
1997 2016
Home Internet
Dial-up connection
22,8 Kbps, 33.6Kbps,
56Kbps
Fiber optic
100 - 1,000Mbps
# of active web sites ~ 1.1 million ~ 1 billion
Internet access Wired PCs
PCs, laptops, mobiles,
IoT, etc.
Internet sites Text and image pages Multimedia applications

1997 vs 2016 - World Wide Web Consortium

History of HTTP
1991 1993 1995 1997 1999
HTTP/0.9
2001 2003 2005 2007 2009 2011 2013
HTTP/1.0
HTTP/1.1
2015 2017
?

HTTP/1.1
Initially released in 1997 (RFC 2068)
Updated in June 1999 (RFC 2616)
More updates in 2014 (RFC 7230, RFC 7231, RFC 7232, RFC 7233, RFC 7234,
RFC 7235)

11
New TCP connection per HTTP
connection.
# of parallel HTTP requests = # of TCP
connections.
SSL / TLS overhead.
H1 Limited Parallelism

13
H1 Head-of-line Blocking
HTTP/1.1
HOL
blocking

14
HTTP metadata (headers).
Not compressed plain text headers for
each and every HTTP request.
H1 Protocol Overhead

15
HTTP metadata (headers).
Not compressed plain text headers for
each and every HTTP request.
Workarounds: cookies, sessions,
concatenation, sprinting, etc.
H1 Protocol Overhead

16
Limited parallelism.
Head-of-line blocking.
High protocol overhead.
HTTP/1.x

HTTP/2
Released 2015.
First new version of HTTP since HTTP/1.1, published in RFC 2068 in 1997.

HTTP/2
Released 2015.
First new version of HTTP since HTTP/1.1, published in RFC 2068 in 1997.
Design goals:
• Reduce latency.
• Minimize protocol overhead.
• Add support for request prioritization
• Add support for server push.

HTTP/2
HTTP/2 is extending, not replacing, the previous HTTP standards.
The application semantics of HTTP are the same:
• HTTP header fields
• HTTP Methods
• Request-response
• Status codes
• URIs

HTTP/2
The application semantics of HTTP are the same:
• HTTP Methods
• Status codes
• URIs
HTTP/2 modifies how the data is formatted (framed) and transported between
the client and server.

HTTP/2
The application semantics of HTTP are the same::
• HTTP Methods
• Status codes
• URIs
HTTP/2 modifies how the data is formatted (framed) and transported between
the client and server.

• Single TCP connection.
• No Head-of-line blocking.
• Binary framing layer.
• Request –> Stream.
• Header Compression.
HTTP/2 in One Slide
Transport(TCP)
Application (HTTP/2)
Network (IP)
Session (TLS) [optional]
Binary Framing
HEADERS Frame
DATA Frame
HTTP/2
POST: /upload
HTTP/1.1
Host: www.javaday.org.ua
Content-Type: application/json
Content-Length: 27
HTTP/1.x
{“msg”: “Welcome to 2016!”}

HTTP/2 Binary Framing
• Stream is a bidirectional flow of bytes within an established connection,
which may carry one or more messages.
• Message is a complete sequence of frames that map to a logical request or
response message.
• Frame is the smallest unit of communication in HTTP/2, each containing a
frame header, which at a minimum identifies the stream to which the frame
belongs: HEADERS for metadata, DATA for payload, RST_STREAM
SETTINGS, PUSH_PROMISE, PING, GOAWAY, WINDOW_UPDATE, etc.

HTTP/2 breaks down the
HTTP protocol
communication into an
exchange of
binary-encoded frames,
which are then mapped to
messages that belong to a
stream, and all of which
are multiplexed within a
single TCP connection.
Binary Framing
Stream 1 HEADERS
Stream 2
:method: GET
:path: /kyiv
:version: HTTP/2
:scheme: https
HEADERS
:status: 200
:version: HTTP/2
:server: nginx/1.10.1
...
DATA
<payload>
Stream N
Request
Response
TCP

HTTP/2 Request and Response Multiplexing
Interleave multiple requests and responses in parallel without blocking on any one.
Use a single TCP connection to deliver multiple requests and responses in parallel.
Enable flow-control, server push, etc.
Stream 1
HEADERS
Stream 2
DATA
Stream 3
HEADERS
Stream 3
DATA
Stream 1
DATA
Stream Y
HEADERS
Stream X
DATA
Requests
Responses
HTTP/2 connection
Client Server

26
Client and server maintain
and update an indexed list
of previously seen header
fields.
Indexes are sent for already
seen headers.
Values are encoded with a
static Huffman code.
HPACK
:method GET
:scheme HTTPS
:host myhost.com
:path /image
custom_header some_value
Request #1

27
fields.
seen headers.
HPACK
:method GET
:scheme HTTPS
:host myhost.com
:path /image
Request #1
:method GET
:scheme HTTPS
:host myhost.com
:path /image
HEADERS Frame

28
fields.
seen headers.
HPACK
:method GET
:scheme HTTPS
:host myhost.com
:path /image
:method GET
:scheme HTTPS
:host myhost.com
:path /resource
Request #1 Request #2
:method GET
:scheme HTTPS
:host myhost.com
:path /image
HEADERS Frame

29
fields.
seen headers.
HPACK
:method GET
:scheme HTTPS
:host myhost.com
:path /image
:method GET
:scheme HTTPS
:host myhost.com
:path /resource
:method GET
:scheme HTTPS
:host myhost.com
:path /image
HEADERS Frame

30
fields.
seen headers.
HPACK
:method GET
:scheme HTTPS
:host myhost.com
:path /image
:method GET
:scheme HTTPS
:host myhost.com
:path /resource
:method GET
:scheme HTTPS
:host myhost.com
:path /image
HEADERS Frame
:path /resource
+ indexes for already seen values
HEADERS Frame

31
Static table with 61 common values.
• single byte 0x82 means add
“:method” == “GET”
Dynamic table is initially empty, updated
based on sent values.
Values are encoded with a static
Huffman code.
HPACK

HTTP/1.x vs HTTP/2
http://http2.golang.org/gophertiles
http://www.http2demo.io/

HTTP/1.x vs HTTP/2

HTTP/1.x vs HTTP/2 6 connections 1 connection

HTTP/2 in the Wild
• Apache Tomcat 8.5+
• Apache HTTP Server
2.4.17+ (C)
• NGINX (C)
• Jetty
• Netty
• Undertow
• Vert.x
• OkHttp (Android)
• and more
• curl 7.43.0+
• Wireshark
• jmeter
• HTTP/2 Test
• h2i
• h2load
• and more
• Chrome
• Firefox
• Safari*
• Opera
• Edge
• IE 11*
• Android Browser
• Chrome for Android
• iOS Safari
*only HTTP/2 over TLS (https)
Browsers
• JEP 110: HTTP/2
Client.
• JEP 244: TLS ALPN
Extension.
JavaImplementations Tools
https://github.com/http2/http2-spec/wiki/Implementations
https://github.com/http2/http2-spec/wiki/Tools
http://caniuse.com/#feat=http2

History of HTTP
1991 1993 1995 1997 1999
HTTP/0.9
2001 2003 2005 2007 2009 2011 2013
HTTP/1.0
HTTP/1.1
2015 2017

History of HTTP
1991 1993 1995 1997 1999
HTTP/0.9
2001 2003 2005 2007 2009 2011 2013
HTTP/1.0
HTTP/1.1
SPDY
HTTP/2.0
2015 2017

gRPC and HTTP/2
Google SPDY –> HTTP/2.
Google Stubby –> gRPC.

gRPC and HTTP/2
Google SPDY –> HTTP/2.
Google Stubby –> gRPC.
gRPC is an HTTP/2-based RPC framework.
Because performance matters.
Cloud Pub/Sub: 11x throughput improvement [per CPU].

Service Definition

Service Definition (weather.proto)

syntax = "proto3";
service Weather {
rpc GetCurrent(WeatherRequest) returns (WeatherResponse);
}

Service Definition
syntax = "proto3";
option java_multiple_files = true;
option java_package = "ua.org.javaday.grpcexample";
package grpcexample;
service Weather {
}

Service Definition - Request Message
message WeatherRequest {
Coordinates coordinates = 1;
message Coordinates {
fixed64 latitude = 1;
fixed64 longitude = 2;
}
}

Service Definition - Response Message
message WeatherResponse {
Temperature temperature = 1;
float humidity = 2;
}
message Temperature {
float degrees = 1;
Units units = 2;
enum Units {
FAHRENHEIT = 0;
CELSIUS = 1;
KELVIN = 2;
}
}

syntax = "proto3";
service Weather {
}
message WeatherRequest {
Coordinates coordinates = 1;
message Coordinates {
fixed64 latitude = 1;
fixed64 longitude = 2;
}
}
float humidity = 2;
}
message Temperature {
float degrees = 1;
Units units = 2;
enum Units {
FAHRENHEIT = 0;
CELSIUS = 1;
KELVIN = 2;
}
}

Generated Classes
$ ls -1 build/generated/source/proto/main/java/ua/org/javaday/grpcexample/
Temperature.java
TemperatureOrBuilder.java
WeatherProto.java
WeatherRequest.java
WeatherRequestOrBuilder.java
WeatherResponse.java
WeatherResponseOrBuilder.java

Generated Classes
$ ls -1 build/generated/source/proto/main/java/ua/org/javaday/grpcexample/
Temperature.java
TemperatureOrBuilder.java
WeatherProto.java
WeatherRequest.java
WeatherRequestOrBuilder.java
WeatherResponse.java
WeatherResponseOrBuilder.java
$ ls -1 build/generated/source/proto/main/grpc/ua/org/javaday/grpcexample/
WeatherGrpc.java

First Law of Distributed Object Design:
don't distribute your objects.
Martin Fowler
http://martinfowler.com/articles/distributed-objects-microservices.html
“ ”

The Fallacies of Distributed Computing
The network is reliable
Latency is zero
Bandwidth is infinite
The network is secure
https://blogs.oracle.com/jag/resource/Fallacies.html
Topology doesn't change
There is one administrator
Transport cost is zero
The network is homogeneous

Services not Objects, Messages not References.
Promote the microservices design philosophy of
coarse-grained message exchange between
systems while avoiding the pitfalls of distributed
objects and the fallacies of ignoring the network.
gRPC Principles & Requirements
http://www.grpc.io/blog/principles

Implement gRPC Server
public class WeatherService extends WeatherGrpc.WeatherImplBase {
@Override
public void getCurrent(WeatherRequest request,
StreamObserver<WeatherResponse> responseObserver) {
}
}

@Override
}
}
Abstract class!

@Override
WeatherResponse response = WeatherResponse.newBuilder()
.setTemperature(Temperature.newBuilder().setDegrees(70).setUnits(FAHRENHEIT))
.setHumidity(.65f).build();
responseObserver.onNext(response);
responseObserver.onCompleted();
}
}

Start gRPC Server
void start() throws IOException {
Server grpcServer = NettyServerBuilder.forPort(8090)
.addService(new WeatherService()).build()
.start();
Runtime.getRuntime().addShutdownHook(new Thread(grpcServer::shutdown));
grpcServer.awaitTerminaton();
}

gRPC Clients
ManagedChannel grpcChannel = NettyChannelBuilder.forAddress("localhost", 8090).build();
WeatherGrpc.WeatherStub client = WeatherGrpc.newStub(grpcChannel);

gRPC Clients
WeatherGrpc.WeatherBlockingStub blockingClient = WeatherGrpc.newBlockingStub(grpcChannel);

gRPC Clients
WeatherGrpc.WeatherBlockingStub blockingClient = WeatherGrpc.newBlockingStub(grpcChannel);
WeatherGrpc.WeatherFutureStub futureClient = WeatherGrpc.newFutureStub(grpcChannel);

gRPC Async Client
WeatherRequest request = WeatherRequest.newBuilder()
.setCoordinates(Coordinates.newBuilder().setLatitude(420000000)
.setLongitude(-720000000)).build();
client.getCurrent(request, new StreamObserver<WeatherResponse>() {
@Override
public void onNext(WeatherResponse response) {
logger.info("Current weather for {}: {}", request, response);
}
@Override
public void onError(Throwable t) { logger.info("Cannot get weather for {}", request); }
@Override
public void onCompleted() { logger.info("Stream completed."); }
});

gRPC Sync Client
WeatherResponse response = blockingClient.getCurrent(request);

gRPC ListenableFuture Async Client
ListenableFuture<WeatherResponse> future = futureClient.getCurrent(request);

ListenableFuture<WeatherResponse> future = futureClient.getCurrent(request);
Futures.addCallback(future, new FutureCallback<WeatherResponse>() {
@Override
public void onSuccess(@Nullable WeatherResponse result) {
}
@Override
public void onFailure(Throwable t) {
logger.info("Cannot get current weather for {}", request);
}
});

Futures.transformAsync(futureClient.getCurrent(request), response -> {
return ...;
});

Service Definition - Response Message
float humidity = 2;
}

Adding New Field
float humidity = 2;
Wind wind = 3;
}
message Wind {
Speed speed = 1;
float direction = 2;
}
message Speed {
float value = 1;
Units units = 2;
enum Units {
MPH = 0;
MPS = 1;
KNOTS = 2;
KMH = 3;
}
}

Adding New [Micro]Services
Weather
Weather
Temp
Wind
Humidity

Service Definitions
service Weather {
}
service TemperatureService {
rpc GetCurrent(Coordinates) returns (Temperature);
}
service HumidityService {
rpc GetCurrent(Coordinates) returns (HumidityResponse);
}
service WindService {
rpc GetCurrent(Coordinates) returns (Wind);
}

Blocking Stub Dependencies
private final TemperatureServiceBlockingStub temperatureService;
private final HumidityServiceBlockingStub humidityService;
private final WindServiceBlockingStub windService;
public WeatherService(TemperatureServiceBlockingStub temperatureService,
HumidityServiceBlockingStub humidityService,
WindServiceBlockingStub windService) {
this.temperatureService = temperatureService;
this.humidityService = humidityService;
this.windService = windService;
}
...

Blocking Stub Dependencies
...
@Override
Temperature temperature = temperatureService.getCurrent(request.getCoordinates());
HumidityResponse humidity = humidityService.getCurrent(request.getCoordinates());
Wind wind = windService.getCurrent(request.getCoordinates());
WeatherResponse response = WeatherResponse.newBuilder().setTemperature(temperature)
.setHumidity(humidity.getValue()).setWind(wind).build();
responseObserver.onNext(response);
}
}

72
Blocking Stubs
Temp
Humidity
Wind

Async Stub Dependencies
private final TemperatureServiceFutureStub tempService;
private final HumidityServiceFutureStub humidityService;
private final WindServiceFutureStub windService;
public WeatherService(TemperatureServiceFutureStub tempService,
HumidityServiceFutureStub humidityService,
WindServiceFutureStub windService) {
this.tempService = tempService;
}
...

@Override
public void getCurrent(WeatherRequest request, StreamObserver<WeatherResponse> responseObserver) {
Coordinates coordinates = request.getCoordinates();
Builder responseBuilder = WeatherResponse.newBuilder();
ListenableFuture<List<Builder>> responsesFuture = Futures.allAsList(
Futures.transform(tempService.getCurrent(coordinates), responseBuilder::setTemperature),
Futures.transform(humidityService.getCurrent(coordinates),
(HumidityResponse humidity) -> responseBuilder.setHumidity(humidity.getValue())),
Futures.transform(windService.getCurrent(coordinates), responseBuilder::setWind)
);
...

@Override
...
Futures.addCallback(responsesFuture, new FutureCallback<List<Builder>>() {
@Override
public void onSuccess(@Nullable List<Builder> result) {
responseObserver.onNext(responseBuilder.build());
}
@Override
public void onFailure(Throwable t) { responseObserver.onError(t);}
});
}
}

Netty Transport
.addService(new WeatherService()).build().start();

Netty: Asynchronous and Non-Blocking IO
Netty-based transport:
• Multiplexes conn-s on the event loops: EpollEventLoopGroup, NioEventLoopGroup.
• Decouples I/O waiting from threads.
• gRPC uses Netty event loops for both client and server transports.
Request
Event
Loop
Worker thread
Worker thread
Worker thread
Worker thread
Worker thread
Worker thread
Worker thread
Event
Loop
Event
Loop
Event
Loop
Callback
Request
Callback
N
e
t
w
o
r
k
N
e
t
w
o
r
k

Panta rhei, "everything flows".
Heraclitus
~2,400 years Before HTTP/1.x
https://en.wikipedia.org/wiki/Heraclitus
“ ”

Streaming Service Definitions
service WeatherStreaming {
rpc GetCurrent(WeatherRequest) returns (stream WeatherResponse);
}
service TemperatureServiceStreaming {
rpc GetCurrent(Coordinates) returns (stream Temperature);
}
service HumidityServiceStreaming {
rpc GetCurrent(Coordinates) returns (stream HumidityResponse);
}
service WindServiceStreaming {
rpc GetCurrent(Coordinates) returns (stream Wind);
}

Bidirectional Streaming Service Definitions
rpc GetCurrent(stream WeatherRequest) returns (stream WeatherResponse);
}
rpc GetCurrent(stream Coordinates) returns (stream Temperature);
}
rpc GetCurrent(stream Coordinates) returns (stream HumidityResponse);
}
rpc GetCurrent(stream Coordinates) returns (stream Wind);
}

Bidirectional Streaming Service Definitions
rpc Observe(stream WeatherRequest) returns (stream WeatherResponse);
}
rpc Observe(stream Coordinates) returns (stream Temperature);
}
rpc Observe(stream Coordinates) returns (stream HumidityResponse);
}
rpc Observe(stream Coordinates) returns (stream Wind);
}

gRPC Service Definitions
Unary RPCs where the
client sends a single
request to the server
and gets a single
response back, just like
a normal function call.
The client sends a
request to the server
and gets a stream to
read a sequence of
messages back.
The client reads from
the returned stream
until there are no more
messages.
The client send a
sequence of messages
to the server using a
provided stream.
Once the client has
finished writing the
messages, it waits for
the server to read them
and return its response.
Client streaming
Both sides send a
sequence of messages
using a read-write
stream. The two
streams operate
independently. The
order of messages in
each stream is
preserved.
BiDi streamingUnary Server streaming

Bidirectional Streaming Service
public class WeatherStreamingService extends WeatherStreamingGrpc.WeatherStreamingImplBase {
private static final Logger logger = LogManager.getLogger();
private final TemperatureServiceStreamingStub temperatureService;
private final HumidityServiceStreamingStub humidityService;
private final WindServiceStreamingStub windService;
public WeatherStreamingService(TemperatureServiceStreamingStub temperatureService,
HumidityServiceStreamingStub humidityService,
WindServiceStreamingStub windService) {
this.temperatureService = temperatureService;
}
...

...
@Override
public StreamObserver<WeatherRequest> observe(StreamObserver<WeatherResponse> responseObserver) {
...

...
@Override
StreamObserver<Coordinates> temperatureClientStream =
temperatureService.observe(newStreamObserver(responseObserver,
(builder, temp) -> builder.setTemperature(temp)));
StreamObserver<Coordinates> humidityClientStream =
humidityService.observe(newStreamObserver(responseObserver,
(builder, humidity) -> builder.setHumidity(humidity.getValue())));
StreamObserver<Coordinates> windClientStream =
windService.observe(newStreamObserver(responseObserver,
(builder, wind) -> builder.setWind(wind)));
List<StreamObserver<Coordinates>> clientStreams =
Arrays.asList(temperatureClientStream, humidityClientStream, windClientStream);
...

...
@Override
...
...

...
@Override
...
return new StreamObserver<WeatherRequest>() {
@Override
public void onNext(WeatherRequest request) {
clientStreams.forEach(s -> s.onNext(request.getCoordinates()));
}
@Override public void onError(Throwable t) { clientStreams.forEach(s -> s.onError(t)); }
@Override public void onCompleted() { clientStreams.forEach(s -> s.onCompleted()); }
};
}
...

...
@Override
StreamObserver<Coordinates> temperatureClientStream =
temperatureService.observe(newStreamObserver(responseObserver,
(builder, temp) -> builder.setTemperature(temp)));
...
}
private <T> StreamObserver<T> newStreamObserver(StreamObserver<WeatherResponse> responseObserver,
BiFunction<Builder, T, Builder> convert) {
return new DelegatingStreamObserver<T, WeatherResponse>(responseObserver, t -> {
return convert.apply(WeatherResponse.newBuilder(), t).build();
});
}
...

96
Messaging applications.
Games / multiplayer tournaments.
Moving objects.
Sport results.
Stock market quotes.
Smart home devices.
You name it!
BiDi Streaming Use-Cases

gRPC Speaks Your Language
● Java
● Go
● C/C++
● C#
● Node.js
● PHP
● Ruby
● Python
● Objective-C
● MacOS
● Linux
● Windows
● Android
● iOS
Service definitions and client libraries Platforms supported

Interoperability
Java
Service
Python
Service
GoLang
Service
C++
Service
gRPC
Service
gRPC
Stub
gRPC
Stub
gRPC
Stub
gRPC
Stub
gRPC
Service
gRPC
Service
gRPC
Service
gRPC
Stub

gRPC API in Google Cloud
A massively scalable
NoSQL database
service.
A fully-managed
real-time messaging
service.
Speech to text
conversion powered
by machine learning.
Cloud Speech APICloud Bigtable Cloud PubSub

Some of the gRPC Adopters
Your
company could
be here

Netflix deprecated its RPC framework in favor of gRPC:
“Our team has instead building an RPC solution on top
of gRPC. We are doing this transition for two main
reasons: multi-language support and better
extensibility/composability through request
interceptors. That’s our current plan moving forward.”
https://github.com/Netflix/ribbon
Some of the gRPC Adopters

Transports:
Client-side
• Netty
• OkHttp (Android)
• InProcess
Server-side
• Netty
• InProcess
gRPC Java
HTTP/2
RPC Client-Side App RPC Server-side App
Service Definition
(extends generated definition)
ServerCall handler
Transport (Netty)
Channel
Transport (Netty, OkHttp)
Stub
Future
Stub
Blocking
Stub
ServerCallClientCall

Coverage & Simplicity
The stack should be available on every popular
development platform and easy for someone to build
for their platform of choice. It should be viable on
CPU & memory limited devices.

Tell Me Moar !!1

Payload Agnostic

Payload Agnostic - Thrift Support Example
namespace java io.grpc.examples.thrift.helloworld
struct HelloRequest {
1:string name
}
struct HelloResponse {
1:string message
}
service Greeter {
HelloResponse sayHello(1:HelloRequest request);
}
https://github.com/grpc/grpc-java/tree/master/examples/thrift/src/main/java/io/grpc/examples/thrift/helloworld

Microservices
The
D'Artagnan
[Our]
Service
Not
D'Artagnan
Service
Not
D'Artagnan
Service
Not
D'Artagnan
Service
Not
D'Artagnan
Service
Not
D'Artagnan
Service
Not
D'Artagnan
Service
Not
D'Artagnan
Service

109
Things will break.
Don’t pretend you can eliminate every
possible source of failure.
Integration points are #1 killer of
software systems.
Fault Tolerance?

Use time-outs!
Captain
“ ”* Original photo taken by Alexandr Gusew.

Use time-outs!
Because your code cannot wait forever.
Captain
“ ”* Original photo taken by Alexandr Gusew.

Timeouts?
GW
A1 A2 A3
B1 B2 B3
C1 C2 C3
200 ms
?
?
?
? ?
? ?
? ?

Default Timeout For Every Service?
GW
A1 A2 A3
B1 B2 B3
C1 C2 C3
200 ms
200 ms
200 ms
200 ms
200 ms 200 ms
200 ms 200 ms
200 ms 200 ms

Gateway
20 ms 10 ms 10 ms
FastFastFast
30 ms 40 ms 20 ms
130 ms
200 ms
timeout
200 ms
timeout
200 ms
timeout
200 ms
timeout

Gateway
20 ms 30 ms 10 ms
130 ms 100 ms 20 ms
FastFairFair
200 ms
timeout
200 ms
timeout
200 ms
timeout
200 ms
timeout

20 ms 30 ms 10 ms
130 ms 100 ms 20 ms
310 ms
FastFairFair
200 ms
timeout
200 ms
timeout
200 ms
timeout
200 ms
timeout

Gateway
30 ms 80 ms 100 ms
SlowFair
210 ms
200 ms
timeout
200 ms
timeout
200 ms
timeout
200 ms
timeout

30 ms 80 ms 100 ms
SlowFair
210 ms
200 ms
timeout
200 ms
timeout
200 ms
timeout
200 ms
timeout

80 ms 100 ms
SlowFair
200 ms
timeout
200 ms
timeout
200 ms
timeout
200 ms
timeout
Still working!

80 ms 100 ms
90 ms
SlowSlowFair
200 ms
timeout
200 ms
timeout
200 ms
timeout
200 ms
timeout

80 ms 100 ms
90 ms
SlowSlowFair
270 ms
200 ms
timeout
200 ms
timeout
200 ms
timeout
200 ms
timeout

200 ms
timeout
200 ms
timeout
200 ms
timeout
80 ms 100 ms
90 ms
SlowSlow
270 ms
200 ms
timeout

Tune Timeout For Every Service?
GW
A1 A2 A3
B1 B2 B3
C1 C2 C3
200 ms
190 ms
190 ms
190 ms
110 ms 50 ms
110 ms 50 ms
110 ms 50 ms

Gateway
200 ms
timeout
190 ms
timeout
110 ms
timeout
30 ms 80 ms 25 ms
FastFair
55 ms
Fast
30 ms
50 ms
timeout

Gateway
200 ms
timeout
190 ms
timeout
110 ms
timeout
30 ms 80 ms 25 ms
Fair
55 ms
Fast
30 ms
50 ms
timeout

Timeouts for Real World?
GW
A1 A2 A3
B1 B2 B3
C1 C2 C3
200 ms

Timeout Propagation
Gateway
90 ms
timeout =
200 - 40
ms
timeout =
160 - 90 - 20
ms
timeout =
200 ms
40 ms
20 ms
20 ms 60 ms
timeout
reached!
230 ms

130
gRPC Java does not support timeouts.
Timeouts in gRPC

131
gRPC Java does not support timeouts.
gRPC supports deadlines instead!
gRPC Deadlines
WeatherResponse response =
client.withDeadlineAfter(200, MILLISECONDS)
.getCurrent(request);

132
First-class feature in gRPC.
Deadline is an absolute point in time.
Deadline indicates to the server how
long the client is willing to wait for an
answer.
RPC will fail with DEADLINE_EXCEEDED
status code when deadline reached.
gRPC Deadlines

gRPC Deadline Propagation
Gateway
90 ms
Now =
1476600000000
Deadline =
1476600000200
40 ms
20 ms
20 ms 60 ms
withDeadlineAfter(200, MILLISECONDS)
Now =
1476600000040
Deadline =
1476600000200
Now =
1476600000150
Deadline =
1476600000200
Now =
1476600000230
Deadline =
1476600000200
DEADLINE_EXCEEDED DEADLINE_EXCEEDED DEADLINE_EXCEEDED DEADLINE_EXCEEDED

134
Deadlines are automatically propagated!
Can be accessed by the receiver!
gRPC Deadlines
Context context = Context.current();
context.getDeadline().isExpired();
context.getDeadline()
.timeRemaining(MILLISECONDS);
context.getDeadline().runOnExpiration(() ->
logger.info("Deadline exceeded!"), exec);

135
Deadlines are expected.
What about unpredictable cancellations?
• User cancelled request.
• Caller is not interested in the result any
more.
• etc
Cancellation?

Cancellation?
GW
Busy Busy Busy
Busy Busy Busy
Busy Busy Busy
RPC
Active RPC Active RPC
Active RPC
Active RPC Active RPCActive RPC
Active RPC

Cancellation?
GW
Busy Busy Busy
Busy Busy Busy
Busy Busy Busy
Active RPC
Active RPC Active RPCActive RPC
Active RPC

Cancellation Propagation
GW
Idle Idle Idle
Idle Idle Idle
Idle Idle Idle

139
Automatically propagated.
RPC fails with CANCELLED status code.
Cancellation status be accessed by the
receiver.
Server (receiver) always knows if RPC is
valid!
gRPC Cancellation

140
Automatically propagated.
RPC fails with CANCELLED status code.
Cancellation status be accessed by the
receiver.
Server (receiver) always knows if RPC is
valid!
gRPC Cancellation

gRPC Cancellation
...
@Override
ServerCallStreamObserver<WeatherResponse> streamObserver =
(ServerCallStreamObserver<WeatherResponse>) responseObserver;
streamObserver.isCancelled();
...

gRPC Cancellation
...
@Override
streamObserver.setOnCancelHandler(() -> {
cleanupCallResources();
logger.info("Call cancelled by client!");
});
...

gRPC Context
...
@Override
Context.current().getDeadline();
Context.current().isCancelled();
Context.current().cancellationCause();
...

gRPC Context
...
@Override
Context.current().addListener(context -> {
cleanupCallResources();
logger.info("Call cancelled by client!");
}, executor)
...

More Control?

BiDi Streaming - Slow Client
Fast Server
Request
Responses
Slow Client
CANCELLED
UNAVAILABLE
RESOURCE_EXHAUSTED

BiDi Streaming - Slow Server
Slow Server
Request
Response
Fast Client
CANCELLED
UNAVAILABLE
RESOURCE_EXHAUSTED
Requests

Client-Side Flow-Control
Server
Request, count = 4
Client
2 Responses
4 Responses
Count = 2

Server-Side Flow-Control
Server
Request
Client
Response
count = 2
2 Requests
count = 3
3 Requests

Flow-Control (Client-Side)
CallStreamObserver<WeatherRequest> requestStream =
(CallStreamObserver) client.observe(new ClientResponseObserver<WeatherRequest, WeatherResponse>() {
@Override
public void beforeStart(ClientCallStreamObserver requestStream) {
requestStream.disableAutoInboundFlowControl();
}
@Override
public void onNext(WeatherResponse response) { processResponse(response); }
@Override
public void onError(Throwable e) { logger.error("Error on weather request.", e); }
@Override
public void onCompleted() { logger.info("Stream completed."); }
});
requestStream.onNext(request);
requestStream.request(3);

Flow-Control (Client-Side)
...
@Override
streamObserver.setOnReadyHandler(() -> {
while (streamObserver.isReady()) {
streamObserver.onNext(calculateWeather());
}
});
...

156
Flow-control helps to balance
computing power and network
capacity between client and server.
gRPC supports both client- and
server-side flow control.
Disabled by default.
Flow-Control

Interceptors
Client Server
Request
Response
Client
interceptors
Server
interceptors

Client Interceptor
public class HeaderClientInterceptor implements ClientInterceptor {
@Override
public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(MethodDescriptor<ReqT, RespT> method,
CallOptions callOptions, Channel next) {
}
}

Client Interceptor
public class HeaderClientInterceptor implements ClientInterceptor {
private static Metadata.Key<String> customHeadKey =
Metadata.Key.of("custom_header_key", ASCII_STRING_MARSHALLER);
@Override
public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(MethodDescriptor<ReqT, RespT> method,
CallOptions callOptions, Channel next) {
return new SimpleForwardingClientCall<ReqT, RespT>(next.newCall(method, callOptions)) {
@Override
public void start(Listener<RespT> responseListener, Metadata headers) {
// put custom header
headers.put(customHeadKey, "customRequestValue");
super.start(responseListener, headers);
}
};
}
}

Server Interceptor
public class HeaderServerInterceptor implements ServerInterceptor {
@Override
public <ReqT, RespT> ServerCall.Listener<ReqT> interceptCall(ServerCall<ReqT, RespT> call,
Metadata requestHeaders, ServerCallHandler<ReqT, RespT> next) {
}
}

Server Interceptor
public class HeaderServerInterceptor implements ServerInterceptor {
private static final Logger logger = LogManager.getLogger();
private static Metadata.Key<String> customHeadKey =
Metadata.Key.of("custom_header_key", ASCII_STRING_MARSHALLER);
@Override
public <ReqT, RespT> ServerCall.Listener<ReqT> interceptCall(ServerCall<ReqT, RespT> call,
Metadata requestHeaders, ServerCallHandler<ReqT, RespT> next) {
String customHeaderValue = requestHeaders.get(customHeadKey);
logger.info("header received from client: {}", customHeaderValue);
return next.startCall(call, requestHeaders);
}
}

Metadata Exchange
Metadata is first-class feature in gRPC.
Lets the client provide information associated with the call to the server and
vice versa in the form of a list of key-value pairs.
Keys are strings; values can be either strings or binary data.

Metadata Exchange
Metadata is first-class feature in gRPC.
Lets the client provide information associated with the call to the server and
vice versa in the form of a list of key-value pairs.
Keys are strings; values can be either strings or binary data.
Can be sent in headers or trailers.
http://www.grpc.io/docs/guides/concepts.html#metadata

Service Discovery & Load Balancing
HTTP/2
RPC Client-Side App RPC Server-side App
Service Definition
ServerCall handler
Transport
Channel
Transport
Stub
Future
Stub
Blocking
Stub
ServerCallClientCall

HTTP/2
RPC Client-Side App
Channel
Transport
Stub
Future
Stub
Blocking
Stub
ClientCall
Instance
#1
Instance
#N
Instance
#2
RPC Server-side Apps

HTTP/2
RPC Client-Side App
Channel
Stub
Future
Stub
Blocking
Stub
ClientCall
Instance
#1
Instance
#1
Instance
#N
Instance
#2
Tran #1 Tran #2 Tran #N

Channel is a logical connection to a service, which is a collection of servers.
NameResolver is a pluggable component that resolves a service name and
return addresses to the caller. It has no knowledge of load-balancing. There will
be different NameResolvers to support different name-systems, e.g., DNS,
ZooKeeper etc.
LoadBalancer is a pluggable component that receives resolved addresses from
NameResolver and selects a Transport for Channel when asked.

ManagedChannel grpcChannel = NettyChannelBuilder.forTarget("WeatherSrv")
.nameResolverFactory(new DnsNameResolverProvider())
.loadBalancerFactory(RoundRobinLoadBalancerFactory.getInstance())
.build();

ManagedChannel grpcChannel = NettyChannelBuilder.forTarget("WeatherSrv")
.nameResolverFactory(new DnsNameResolverProvider())
.loadBalancerFactory(RoundRobinLoadBalancerFactory.getInstance())
.build();
Design documents:
• Load Balancing in gRPC:
https://github.com/grpc/grpc/blob/master/doc/load-balancing.md
• gRPC Java Name Resolution and Load Balancing: http://tiny.cc/grpc-java-lb

Layered Architecture
HTTP/2
RPC Client-Side App
Channel
Stub
Future
Stub
Blocking
Stub
ClientCall
Tran #1 Tran #2 Tran #N
Service Definition
ServerCall handler
Transport
ServerCall
NameResolver LoadBalancer
Pluggable
Load
Balancing
and
Service
Discovery

Layered
Key facets of the stack must be able to evolve
independently. A revision to the wire-format should
not disrupt application layer bindings.

Testing Support
In-process server transport: fully-featured, high performance, and useful in testing.
In-process client side channel: fully-featured, high performance, and useful in
testing.

Testing Support - InProcess
WeatherServiceAsync weatherService =
new WeatherServiceAsync(tempService, humidityService, windService);
Server grpcServer = InProcessServerBuilder.forName("weather")
.addService(weatherService).build();

.addService(ServerInterceptors.intercept(weatherService, interceptor)).build();

.addService(ServerInterceptors.intercept(weatherService, interceptor)).build();
Channel grpcChannel = InProcessChannelBuilder.forName("weather").build();
WeatherBlockingStub stub =
WeatherGrpc.newBlockingStub(grpcChannel).withDeadlineAfter(100, MILLISECONDS);

WeatherBlockingStub stub =
WeatherGrpc.newBlockingStub(grpcChannel).withDeadlineAfter(100, MILLISECONDS);
@Test
public void returnWeatherResponseWhenAllDependentServicesReply() {
WeatherRequest request = WeatherRequest.newBuilder().setCoordinates(
Coordinates.newBuilder().setLatitude(420000000).setLongitude(-720000000)).build();
WeatherResponse result = stub.getCurrent(request);
...

Testing Support
testing.
MetadataUtils for testing metadata headers and trailer.

Testing Support - Metadata
@Test
public void sendCustomHeaderMetaData() {
AtomicReference<Metadata> headers = new AtomicReference<>();
AtomicReference<Metadata> trailers = new AtomicReference<>();
WeatherBlockingStub metaCapturingStub =
MetadataUtils.captureMetadata(stub, headers, trailers);
metaCapturingStub.getCurrent(request);
Metadata headersMeta = headers.get();
assertThat(headersMeta.get(customHeadKey)).isEqualTo(expectedCustomHeadValue);
// same can be done for trailers
}

Testing Support
testing.
MetadataUtils for testing metadata headers and trailer.
grpc-testing - additional test utility functions useful for writing unit and integration
tests: https://github.com/grpc/grpc-java/tree/master/testing

Distributed Tracing
A B C
D E F
G H J

Distributed Tracing
Paper: Dapper, a Large-Scale Distributed Systems Tracing Infrastructure:
http://research.google.com/pubs/pub36356.html
Implementation: Zipkin is a distributed tracing system: zipkin.io (JVM, Go, C#,
Python, Ruby).
Brave is a pure-Java distributed tracing implementation compatible with Zipkin
(no Scala required): https://github.com/openzipkin/brave
Brave-gRPC integrates Braves with gRPC.

Distributed Tracing with Zipkin/Brave
// server-side
.addService(ServerInterceptors.intercept(new WeatherService(),
new BraveGrpcServerInterceptor(brave("wind"))))
.build();

// server-side
.addService(ServerInterceptors.intercept(new WeatherService(),
new BraveGrpcServerInterceptor(brave("wind"))))
.build();
// client-side
Channel windChannel = NettyChannelBuilder.forAddress(windserverAddress()).build();
WindServiceStub windClient = WindServiceGrpc.newStub(
ClientInterceptors.intercept(windChannel,
new BraveGrpcClientInterceptor(brave("weather_to_wind"))));

Weather
Weather
Wind
Conversion
Units
Temp
Humidity

192
gRPC Gateway

193
Migration.
Testing.
Swagger / OpenAPI
tooling.
gRPC Gateway
https://github.com/grpc-ecosystem/grpc-gateway

More Features
Monitoring:
• gRPC Prometheus.
• More Monitoring APIs are coming.
Built-in authentication mechanisms:
• SSL/TLS.
• Token-based authentication with Google.
• Authentication API.
Payload compression:
https://github.com/grpc/grpc/blob/master/doc/compression.md

Growing Community and Ecosystem
https://github.com/grpc-ecosystem
Polyglot is a universal grpc command line client.
grpc-gateway generates a reverse-proxy server which translates a RESTful
JSON API into gRPC.
OpenTracing is a set of consistent, expressive, vendor-neutral APIs for
distributed tracing and context propagation.
Prometheus monitoring support for grpc-java and grpc-go.

Pluggable
Large distributed systems need security,
health-checking, load-balancing and failover,
monitoring, tracing, logging, and so on.
Implementations should provide extensions points
to allow for plugging in these features and, where
useful, default implementations.

Try It Out!
http://grpc.io
gRPC on Github: https://github.com/grpc
gRPC Java on Github: https://github.com/grpc/grpc-java
Google group: grpc-io@googlegroups.com
gRPC Java quickstart: http://www.grpc.io/docs/quickstart/java.html
gRPC Java tutorial: http://www.grpc.io/docs/tutorials/basic/java.html
gRPC contribution: https://github.com/grpc/grpc-contrib

Takeaways
HTTP/2 is a high performance production-ready multiplexed
bidirectional protocol.
gRPC (http://grpc.io):
• HTTP/2 transport based, open source, general purpose
standards-based, feature-rich RPC framework.
• Bidirectional streaming over one single TCP connection.
• Netty transport provides asynchronous and non-blocking I/O.
• Deadline and cancellations propagation.
• Client- and server-side flow-control.
• Layered, pluggable and extensible.
• Supports 10 programming languages.
• Build-in testing support.
• Production-ready (current version is 1.0.1) and growing ecosystem.

Questions?

Images Used
Special thanks for the provided photos:
• Andrey Borisenko
• Alexandr Gusew
Photo from https://www.google.com/about/datacenters page:
• https://www.google.com/about/datacenters/gallery/#/tech/12
Photos licenced by Creative Commons 2.0 https://creativecommons.org/licenses/by/2.0/ :
• https://www.flickr.com/photos/13800911@N08/3557747851/ by DirectDish
• https://www.flickr.com/photos/marcalandavis/45657810/ by Marc Davis
• https://www.flickr.com/photos/garryknight/5754661212/ by Garry Knight
• https://www.flickr.com/photos/sodaigomi/21128888345/ by sodai gomi
• https://www.flickr.com/photos/zengame/15972170944/ by Zengame
• https://www.flickr.com/photos/londonmatt/18496249193/ by Matt Brown
• https://www.flickr.com/photos/gazeronly/8058105980/ by torbakhopper
U.S. Government Works:
• https://www.flickr.com/photos/cbpphotos/8653133856/ by Josh Denmark
Free of known restrictions:
https://commons.wikimedia.org/wiki/File:He_can%27t_fix_guns_in_the_air%5E_Build_%60em_right%5E_Keep_%60em_firing%5E_-_NARA_-_535050.jpg

Enabling Googley microservices with HTTP/2 and gRPC.

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Enabling Googley microservices with HTTP/2 and gRPC.

Similar to Enabling Googley microservices with HTTP/2 and gRPC. (20)

More from Alex Borysov

More from Alex Borysov (19)

Recently uploaded

Recently uploaded (20)

Enabling Googley microservices with HTTP/2 and gRPC.