Go fit perfectly inside containers, you can ship apps as tiny images on k8s, distributing them across the globe. Gianluca will show how InfluxData debugs containers running on Kubernetes to allow sysadmins and developers to troubleshoot and replicate issues using core dump, debuggers, and logs.
Go applications are perfect to be run inside a container. You can build a single binary, a tiny Docker image and you can ship them on your Kubernetes cluster. A successful production environment requires stability and simplicity, it needs to be easy to troubleshoot and operators need to be able to get all the information developers will need to fix a bug. During this talk, Gianluca will share what influxData is doing to allow developers and system administrator to work together, understanding problems running live at scale on Kubernetes and how to escalate them down to Software Engineer using logs, delve, gdb, core dumps, and traces to replicate and fix issues.
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Kubernetes debug like a pro
1. ~ @gianarb - https://gianarb.it ~
Debug like a pro on
Kubernetes
Golab - Florence - 2018
2. ~ @gianarb - https://gianarb.it ~
Gianluca Arbezzano
Site Reliability Engineer @InfluxData
● http://gianarb.it
● @gianarb
What I like:
● I make dirty hacks that look awesome
● I grow my vegetables 🍅🌻🍆
● Travel for fun and work
3. 1. Yo n !
Your team knows
and use Docker for
local development
and testing
2. Kub te !
Everyone speaks
about kubernetes.
3. Hir !
You don’t know why
but you hired a
DevOps that kind of
know k8s.
3. Ex i m !
You are moving
everything and
everyone to
kubernetes
11. K8s as code: From YAML to code (golang)
1. You have the ability to use Golang autocomplete as documentation, reference for every
kubernetes resources
2. You feel less a YAML engineer (great feeling btw)
3. Code is better than YAML! You can reuse it, compile it, embed it in other projects.
12. K8s as code: From YAML to code (golang)
Tiny cli
to make
the
migration
to golang
Some
manual
refactoring
13. K8s as code: From YAML to code (golang)
Tiny cli
to make
the
migration
to golang
Some
manual
refactoring
● Continue to improve our CI to validate that YAML and Go file are the same,
and the resources in Kubernetes are like the Go file.
● Maybe we will be able to remove the YAML at some point.
14. Examples
● Everything has an API because you should USE it TO make something good!
(cURL is good but you can make something better)
● Some of our tools:
○ Backup and Restore Operator for Persistent Volumes
○ We have a service to create runtime isolated environment to allow devs to test or product
people to have a safe environment to demo, try. We also use it for the integration and smoke
tests.
○ We have a tool to replicate environment locally on minikube to install and configure all the
dependencies
21. Golang and Kubernetes: OpenCensus
● Open Source project sponsored by Google
● It is a SPEC plus a set of libraries in different languages to instrument your
application
● To collect metrics, traces and events.
22. Golang and Kubernetes: OpenCensus
Common
Interface to
get stats and
traces from
your app
Different
exporters to
persist your
data
23. gianarb.it ~ @gianarb
# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="post",code="200"} 1027 1395066363000
http_requests_total{method="post",code="400"} 3 1395066363000
# Escaping in label values:
msdos_file_access_time_seconds{path="C:DIRFILE.TXT",error="Cannot find file:n"FILE.TXT""}
1.458255915e9
# Minimalistic line:
metric_without_timestamp_and_labels 12.47
# A weird metric from before the epoch:
something_weird{problem="division by zero"} +Inf -3982045
# A histogram, which has a pretty complex representation in the text format:
# HELP http_request_duration_seconds A histogram of the request duration.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.05"} 24054
http_request_duration_seconds_bucket{le="0.1"} 33444
http_request_duration_seconds_bucket{le="0.2"} 100392
http_request_duration_seconds_bucket{le="0.5"} 129389
http_request_duration_seconds_bucket{le="1"} 133988
http_request_duration_seconds_bucket{le="+Inf"} 144320
http_request_duration_seconds_sum 53423
http_request_duration_seconds_count 144320
27. gianarb.it ~ @gianarb
More to read
● OpenMetrics: https://github.com/OpenObservability/OpenMetrics
● OpenMetrics mailing list: https://groups.google.com/forum/#protectkern-.
1667emrelaxforum/openmetrics
● WIP branch for Python library
https://github.com/prometheus/client_python/tree/openmetrics
● Thanks RICHARD for your work! I got some slides from here:
https://promcon.io/2018-munich/slides/openmetrics-transforming-the-prometh
eus-exposition-format-into-a-global-standard.pdf
41. Golang and Kubernetes: pprof
● It is the Golang native profiler
● You can use it via the `go pprof` command
● `import "runtime/pprof"` writes runtime profiling data
● `import "net/http/pprof"` serves via HTTP server runtime profiling data
42. Golang and Kubernetes: pprof
package main
import (
"log"
"net/http"
_ "net/http/pprof"
)
func main() {
log.Println(http.ListenAndServe("localhost:6060", nil))
}
43. $ go tool pprof http://localhost:6060/debug/pprof/heap
Fetching profile over HTTP from http://localhost:6060/debug/pprof/heap
Saved profile in
/home/gianarb/pprof/pprof.main.alloc_objects.alloc_space.inuse_objects.inuse_space.009.pb.gz
File: main
Type: inuse_space
Time: Oct 15, 2018 at 4:22pm (CEST)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) png
Generating report in profile001.png
44.
45. Golang and Kubernetes: pprof
https://github.com/influxdata/influxdb/blob/4cbdc197b8117fee648d62e2e5be75c6575352f0/services/httpd/pprof.go
// handleProfiles determines which profile to return to the requester.
func (h *Handler) handleProfiles(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/debug/pprof/cmdline":
httppprof.Cmdline(w, r)
case "/debug/pprof/profile":
httppprof.Profile(w, r)
case "/debug/pprof/symbol":
httppprof.Symbol(w, r)
case "/debug/pprof/all":
h.archiveProfilesAndQueries(w, r)
default:
httppprof.Index(w, r)
}
}
46. Golang and Kubernetes: pprof
https://github.com/influxdata/influxdb/blob/4cbdc197b8117fee648d62e2e5be75c6575352f0/services/httpd/pprof.go
// archiveProfilesAndQueries collects the following profiles:
// - goroutine profile
// - heap profile
// - blocking profile
// - mutex profile
// - (optionally) CPU profile
//
// It also collects the following query results:
//
// - SHOW SHARDS
// - SHOW STATS
// - SHOW DIAGNOSTICS
//
// All information is added to a tar archive and then compressed, before being
// returned to the requester as an archive file.