An intro to the gRPC API for the containerd project by Phil Estes at the containerd summit

1. An API Intro:
Driving containerd
operations with
gRPC

2. WHO AM I?
Phil Estes
Senior Technical Staff, Office of the CTO, IBM Cloud
Maintainer, Docker engine
Maintainer, containerd
Contributor, OCI/runc
Docker Captains program member
Blog: https://integratedcode.us
Twitter: @estesp

3. 1.
What is gRPC?

4. YOU PROBABLY ALREADY HAVE AN IDEA OF WHAT A CLIENT IS
▸ Command-line client (e.g. runc)
▸ REST/HTTP client (e.g. docker -> dockerd)
▸ Web/HTTP client (e.g. curl, your web browser)
gRPC is simply a client-server method execution model
built on the long-standing “remote procedure call”
(RPC) style interface.
gRPC is an open source project created by Google and a
full toolkit is available for many platforms:
https://github.com/grpc/grpc | https://grpc.io

5. “In gRPC a client application can directly call methods
on a server application on a different machine as if it
was a local object, making it easier for you to create
distributed applications and services. As in many RPC
systems, gRPC is based around the idea of defining a
service, specifying the methods that can be called
remotely with their parameters and return types.
FROM: http://www.grpc.io/docs/guides/ :

6. IMPORTANT CONCEPTS: SERVICES & PROTOCOL BUFFERS
▸ A gRPC API has service definitions.
▸ These service definitions have rpc methods.
▸ These rpc methods have “protobuf” defined
message content (request & response)

7. “Protocol buffers are a flexible, efficient, automated
mechanism for serializing structured data – think XML,
but smaller, faster, and simpler. You define how you
want your data to be structured once, then you can use
special generated source code to easily write and read
your structured data to and from a variety of data
streams and using a variety of languages. You can even
update your data structure without breaking deployed
programs that are compiled against the "old" format.
FROM:
https://developers.google.com/protocol-buffers/docs/overview :

8. OK, FINE, BUT WHY gRPC+PROTOBUFS OVER JSON+HTTP/etc?
Code Generation
No more writing
clients by hand!
Generators convert
.proto files into
language-specific
bindings for client
and server.
Speed & Size
Protobuf binary
packed format
marshalled from
clients<-->servers is
smaller than JSON or
XML and better
performing for
serialization &
deserialization.
Commonality
gRPC and protobufs
are growing in
popularity. gRPC is
used in Docker
Swarm, Kubernetes,
etcd, and adopted by
companies like
CoreOS, Netflix,
Square and others.

9. MORE gRPC BENEFITS
▸ gRPC Connection support & options are rich and
completely built-in to the implementation
▹ Less time writing or finding server code to
handle special socket options, HTTP2,
authentication, streaming, etc.
gRPC Connect Example:

10. 2.
How is
containerd
using gRPC?

11. MAJOR API COMPONENTS
▸ Current gRPC services:
▹ Execution, Shim, and Content
▸ Go 1.8 provides plugin support
▹ Expect gRPC services to be pluggable
Execution
- Create
- Start
- Delete
- List
- Events
Shim
- Create
- Start
- Delete
- Exec
- Pty
- Events
- State
- Pause
- Resume
- Exit
Content
- Info
- Read
- Status
- Write

12. SERVICE API/PROTOBUF DEFINITIONS
▸ Each service has a .proto file with all definitions
▸ Common definitions (e.g. types) can be included
▸ Protobuf compiler generates appropriate
language binding (e.g. Go code)
api/services/shim/shim.proto

13. CODE GENERATION: FROM PROTO -> GO SOURCES
*WARNING: NOT FOR THE FAINT OF HEART!
commit 83e7610194170c794752088d54b0c3673b311811
Author: Stephen J Day <stephen.day@docker.com>
Date: Tue Feb 14 16:48:36 2017 -0800
cmd/ctrd-protobuild: create proper command for building protos
After trying to explain the complexities of developing with protobuf, I
have now created a command that correctly calculates the import paths
for each package and runs the protobuf command.
▸ make protos uses go generate tooling*
▸ <service>.pb.go files are created with the Go bindings
▸ Only required when changing (add/remove/modify) API
or message/type definitions

14. 3.
Create your own
gRPC service
and/or client

15. `CTR LIST` AS AN EXAMPLE: SERVER SIDE (GENERATED)
▸ Generated execution.pb.go has interface for
server as well as handler which calls List method
func _ContainerService_List_Handler(srv interface{}, ctx context.Context, dec
func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
…
return srv.(ContainerServiceServer).List(ctx, in)
…
}
api/services/execution/execution.pb.go

16. `CTR LIST` AS AN EXAMPLE: SERVER SIDE (IMPLEMENTED)
▸ We implement the ContainerServiceServer.List
method to perform the operation and return the
appropriate gRPC message response:

17. `CTR LIST` AS AN EXAMPLE: CLIENT SIDE
▸ The client code to call the gRPC method is already
created for us during protobuf generation
▸ We only need to connect our client command to this
generated client stub:
var listCommand = cli.Command{
Name: "list",
Action: func(context *cli.Context) error {
containers, err := getExecutionService(context)
response, err := containers.List(gocontext.Background(),
&execution.ListRequest{})
for _, c := range response.Containers {
// print container information
}
return nil
},
}
cmd/ctr/list.go

18. TAKING THE
NEXT STEP
CONSUME CONTAINERD VIA gRPC CLIENT
HELP IMPROVE/DEFINE NEW AREAS OF
CORE CONTAINERD API
ADD SERVICE VIA GO 1.8 PLUGINS + gRPC
PROTO SERVICE DEFINITION

19. 4.
Summary
Expected that consumers of containerd
will use the gRPC API. The ctr tool is a
development/admin resource.
gRPC benefits outweigh any added
complexity with protobufs for defining and
implementing the API layer.
The API is under heavy development
and will stabilize leading up to the
1.0 release of containerd. Post 1.0,
gRPC versioning will allow us to
retain backwards compatibility.

20. CREDITS
Special thanks to all the people who made and released these
awesome resources for free:
▸ Simple line icons by Mirko Monti
▸ E-commerce icons by Virgil Pana
▸ Streamline iconset by Webalys
▸ Presentation template by SlidesCarnival