Advanced debugging techniques of Go code

Advanced debugging
techniques
of Go code
F0SD3M
02022020

Andrii Soldatenko
• Gopher, OSS
contributor
• father of 👶 👶
• debuggers fan
• Speaker at many
conferences
@a_soldatenko

via prints> 🐞 ?
package main
import (
"fmt"
)
func main() {
const name, age = "Andrii", 22
fmt.Println(name, "is", age, "years old.")
}

@a_soldatenko
using duck debugging?

@a_soldatenko
- delve
- GDB
using debuggers:

@a_soldatenko
💡Interesting fact:
Go version prior 1.0
included debugger OGLE

Current versions of Go
produce DWARF
Debugging Standard
http://dwarfstd.org/doc/dwarf-2.0.0.pdf

Disable DWARF generation
go build -ldflags=-w
but with no loss of
functionality

delve: 101
- dlv debug - Compile and begin debugging
main package in current directory, or the
package speciﬁed.
- dlv test - Compile test binary and begin
debugging program.DWARF speciﬁcation

Debug CLI util 🌊
.
├── github.com/me/foo
├── cmd
│ └── foo
│ └── main.go
├── pkg
│ └── baz
│ ├── bar.go
│ └── bar_test.go
$ dlv debug github.com/me/foo/cmd/foo -- -arg1
valueType
(dlv) break main.main
Breakpoint 1 set at 0x49ecf3 for main.main() ./test.go:5
(dlv) continue

@a_soldatenko
Debugging specific
unit test
go test -test.run TestFibonacciBig
dlv test -- -test.run TestFibonacciBig

@a_soldatenko
Debugging unit tests:
example
dlv test -- -test.run TestFibonacciBig
(dlv) b main_test.go:6
Breakpoint 1 set at 0x115887f for github.com/andriisoldatenko/
debug_test.TestFibonacciBig() ./main_test.go:6
(dlv) c
> github.com/andriisoldatenko/debug_test.TestFibonacciBig() ./
main_test.go:6 (hits goroutine(17):1 total:1) (PC: 0x115887f)
1: package main
2:
3: import "testing"
4:
5: func TestFibonacciBig(t *testing.T) {
=> 6: var want int64 = 55
7: got := FibonacciBig(10)
8: if got.Int64() != want {
9: t.Errorf("Invalid Fibonacci value for N: %d, got: %d,
want: %d", 10, got.Int64(), want)
10: }
11: }
(dlv)

Debugging unit tests:
breakpoint
dlv test github.com/andriisoldatenko/debugging-containerized-go-
applications/
Type 'help' for list of commands.
(dlv) b TestCloud1
Breakpoint 1 set at 0x113efaf for github.com/andriisoldatenko/
debugging-containerized-go-applications.TestCloud1() ./
hello_test.go:16
(dlv) c
TestMy1
TestYour1
> github.com/andriisoldatenko/debugging-containerized-go-
applications.TestCloud1() ./hello_test.go:16 (hits goroutine(33):1
total:1) (PC: 0x113efaf)
11:
12: func TestYour1(t *testing.T) {
13: fmt.Println("TestYour1")
14: }
15:
=> 16: func TestCloud1(t *testing.T) {
17: fmt.Println("TestCloud1")
18: }

@a_soldatenko
Conditional breakpoint 🚑
package main
import "fmt"
func main() {
nums := []int{2, 3, 4}
for i, num := range nums {
if num == 3 {
fmt.Println(“BUG!!! index:”, i)
}
}
}
t.me/golang_for_two

(dlv) b b2 hello.go:8
Breakpoint b2 set at 0x10b71e2 for main.main() ./hello.go:8
(dlv) cond b2 num == 3
(dlv) c
> [b2] main.main() ./hello.go:8 (hits goroutine(1):1 total:
1) (PC: 0x10b71e2)
3: import "fmt"
4:
5: func main() {
6: nums := []int{2, 3, 4}
7: for i, num := range nums {
=> 8: if num == 3 {
9: fmt.Println("index:", i)
10: }
11: }
12: }
(dlv) p num
3
(dlv) n
Conditional breakpoint 🚑

@a_soldatenko
How to set variable?
4:
=> 5: func main() {
6: a := 1
7: b := 2
8: fmt.Println(a, b)
9: }
(dlv) set a = 10
t.me/golang_for_two

Call function
(dlv) call myFunc(localVar, 4, "foo")
- runtime: support for debugger function calls (https://golang.org/cl/
109699)
- Add ability to safely call functions (go-delve/delve/issues/119)

Call function
(dlv) call t()
> main.dummy() ./hello.go:18 (hits
goroutine(1):1 total:1) (PC:
0x10b732d)
13: }
14: }
15:
16:
17: func dummy(){
=> 18: fmt.Println("func call")
19: }

Pretty print with go-spew
// to illustrate the points.
type ctfileParseState struct {
header *ctfileHeader
bar string
}
ctfh := ctfileHeader{1297035375, 82, 33060, 4, "test"}
c := ctfileParseState{&ctfh, "bar"}
fmt.Printf("%+vn", c)
{header:0xf84003f480 bar:bar}
spew.Printf("%+vn", c)
{header:<*>(0xf84003f480){beacon:1297035375 numShas:82 mode:
33060 typ:4 filename:test} bar:bar}
https://github.com/davecgh/go-spew

@a_soldatenko
Debugging containerized
go apps
t.me/golang_for_two

Step 1: Dockerfile 🐳
$ cat Dockerfile
FROM golang:1.13
WORKDIR /go/src/app
COPY . .
RUN go get -u github.com/go-delve/delve/cmd/dlv
CMD ["app"]
$ docker build -t my-golang-app .
$ docker run -it --rm my-golang-app bash

@a_soldatenko
Step 2: operation not
permitted 🐳
$ docker run -it --rm my-golang-app bash
$root@03c1977b1063:/go/src/app# dlv debug
main.go
could not launch process: fork/exec /go/src/app/
__debug_bin: operation not permitted

Step 3: 🐳
$ docker run -it --rm
—security-opt="apparmor=unconfined"
—cap-add=SYS_PTRACE my-golang-app bash
root@7dc3a7e8b3fc:/go/src/app# dlv debug
main.go
Type 'help' for list of commands.
(dlv)
https://docs.docker.com/engine/security/apparmor/

@a_soldatenko
Remote debugging
containerized go apps 🎨

# Final stage
FROM alpine:3.7
# Port 8080 belongs to our application, 40000 belongs to Delve
EXPOSE 8080 40000
# Allow delve to run on Alpine based containers.
RUN apk add --no-cache libc6-compat
WORKDIR /
COPY --from=build-env /server /
COPY --from=build-env /go/bin/dlv /
# Run delve
CMD ["/dlv", "--listen=0.0.0.0:40000", "--
headless=true", "--api-version=2", "debug", "go/
src/hello", "--log"
Example of Dockerfile

#!/usr/bin/env bash
docker run
-p 8080:8080
-p 40000:40000
--security-opt="apparmor=unconfined"
--cap-add=SYS_PTRACE hello-debug
Docker run ⬆

$cat ~/.dlv/config.yml
substitute-path:
- {from: "/go/src/hello/", to: "/Users/
andrii/workspace/src/github.com/
andriisoldatenko/debugging-containerized-
go-applications"}
Let’s fix it to more readable
🔧

Configure Delve
$HOME/.dlv/conﬁg.yml
Or
(dlv) conﬁg -list

dlv connect localhost:40000
(dlv) c
> main.main() /go/src/hello/hello.go:5 (hits
goroutine(1):1 total:1) (PC: 0x49b3d8)
1: package main
2:
3: import "fmt"
4:
=> 5: func main() {
6: a := 1
7: b := 2
8: fmt.Println(a, b)
9: }
(dlv)
Delve connect

@a_soldatenko
LIVE DEMO B 🏁
./start.sh

@a_soldatenko
GDB issues on OSX

@a_soldatenko
Starting program: /x/y/foo
Unable to find Mach task port for process-id 28885: (os/kern) failure
(0x5).
(please check gdb is codesigned - see taskgated(8))
https://sourceware.org/gdb/wiki/PermissionsDarwin
check gdb is codesigned 💥

Loading Go Runtime support.
(gdb) b main.main
Breakpoint 1 at 0x10b7130: file .../hello.go, line
5.
(gdb) c
The program is not being run.
(gdb) run
Starting program: .../hello
[New Thread 0x1803 of process 94492]
💡GDB: If you use macOS
10.13 "High Sierra"

• Unlink current gdb: brew unlink gdb 
• Install gdb 8.0.1: brew install https://
raw.githubusercontent.com/Homebrew/
homebrew-core/
9ec9fb27a33698fc7636afce5c1c16787e9ce3f3/
Formula/gdb.rb 
• Optional: avoid upgrade gdb with: brew pin gdb
💡GDB: If you use macOS
10.13 "High Sierra"

@a_soldatenko
💡GDB: hm…..
(gdb) b main.main
Breakpoint 1 at 0x10b70f0
(gdb) c
The program is not being run.
(gdb) run
Starting program: /Users/andrii/workspace/src/
github.com/andriisoldatenko/debugging-
containerized-go-applications/hello
[New Thread 0xf03 of process 5994]
^C^Z
[2] + 5911 suspended gdb hello

@a_soldatenko
💡GDB: does’t understand
comprised DWARF!
go build -ldflags=-compressdwarf=false
-gcflags=all="-N -l" -o hello hello.go

@a_soldatenko
💡 GDB: pretty print!
go build -ldflags=-compressdwarf=false -
gcflags=all="-N -l" -o hello hello.go
➜ go git:(master) find . -name 'runtime-gdb.py'
./src/runtime/runtime-gdb.py
(gdb) source /Users/andrii/work/go/src/runtime/
runtime-gdb.py
Loading Go Runtime support.
➜ debugging-containerized-go-applications git:
(master) ✗ strings hello | grep gdb
/usr/local/Cellar/go/1.12.7/libexec/src/runtime/
runtime-gdb.py

@a_soldatenko
💡 GDB: finally it works!
(gdb) n
6 a := 1
(gdb) n
7 b := 2
(gdb) n
8 fmt.Println(a, b)
(gdb) n
1 2
9 }
(gdb)

@a_soldatenko
Conclusion GDB: 📝
- GDB does not understand Go programs
well.
- The stack management, threading, and
runtime contain aspects that differ enough
from the execution model GDB expects
- GDB can be useful in some situations (e.g.,
debugging Cgo code

@a_soldatenko
Conclusion
- Debugging is fun and always useful 😁
- console interactive debuggers is sometimes looks old,
but helps a lot especially in cloud environments
- UI clients for debuggers still slow… 🐌

@a_soldatenko
Future Reading 🐝:
- Internal Architecture of Delve - slides
- DWARF speciﬁcation DWARF
- https://golang.org/doc/gdb
- https://github.com/go-delve/delve/blob/
master/Documentation/
- (dlv) help
- source code of go

Advanced debugging techniques of Go code

Recommended

Recommended

More Related Content

More from Andrii Soldatenko

More from Andrii Soldatenko (6)

Recently uploaded

Recently uploaded (20)

Advanced debugging techniques of Go code