Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Debugging concurrency programs in go

Recently the interest in concurrent programming has grown dramatically. Unfortunately, parallel programs do not always have reproducible behavior. Even when they are run with the same inputs, their results can be radically different. In this talk I’ll show how to debug concurrency programs in Go.

I’ll start from showing how you can debug your gorotines using delve and gdb debuggers. Then I’ll try to visualize goroutines using different scenarios, sometimes it helps to better understand how things work. Next part of the topic will be about dumping a goroutine stack trace of your application while it’s running and inspect what each goroutine is doing. And I’ll demonstrate how to debug leaking goroutines by tracing the process of how the scheduler runs goroutines on logical processors which are bound to a physical processor via the operating system thread that is attached.

As a bonus i’ll cover debugging tips on how to find deadlocks and how to avoid race conditions in your application.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to comment

  • Be the first to like this

Debugging concurrency programs in go

  1. 1.  Debugging concurrent programs in Go
  2. 2. Andrii Soldatenko • Gopher, OSS contributor • father of 👧 👶 • debuggers fan • Speaker at many conferences @a_soldatenko
  3. 3. Concurrency Programming is Challenging!
  4. 4. Debugging concurrent programs is Hard!
  5. 5. @a_soldatenko Debugging sequential programs dlv test -- -test.run TestFibonacciBig (dlv) b main_test.go:6 Breakpoint 1 set at 0x115887f for github.com/andriisoldatenko/ debug_test.TestFibonacciBig() ./main_test.go:6 (dlv) c > github.com/andriisoldatenko/debug_test.TestFibonacciBig() ./ main_test.go:6 (hits goroutine(17):1 total:1) (PC: 0x115887f) 1: package main 2: 3: import "testing" 4: 5: func TestFibonacciBig(t *testing.T) { => 6: var want int64 = 55 7: got := FibonacciBig(10) 8: if got.Int64() != want { 9: t.Errorf("Invalid Fibonacci value for N: %d, got: %d, want: %d", 10, got.Int64(), want) 10: } 11: } (dlv)
  6. 6. @a_soldatenko Debugging concurrent programs package main import ( "fmt" "time" ) func say(s string) { for i := 0; i < 5; i++ { time.Sleep(100 * time.Millisecond) fmt.Println(s) } } func main() { go say("world") say("hello") } gowayfest git:(master) ✗ go run main.go hello world world hello hello world world hello world hello gowayfest git:(master) ✗ go run main.go world hello hello world hello world world hello hello world
  7. 7. What is a goroutine? https://tpaschalis.github.io/goroutines-size/
  8. 8. Do you know why gorotines?
  9. 9. How can I debug concurrent Go program?
  10. 10. Playground GOMAXPROCS is 1
  11. 11. visualize goroutines🐞 ? func main() { c := coloredgoroutine.Colors(os.Stdout) fmt.Fprintln(c, "Hi, I am go routine", goid.ID(), "from main routine") count := 10 var wg sync.WaitGroup wg.Add(count) for i := 0; i < count; i++ { i := i go func() { fmt.Fprintln(c, "Hi, I am go routine", goid.ID(), "from loop i =", i) wg.Done() }() } wg.Wait() }
  12. 12. visualize goroutines🐞 ? https://github.com/xiegeo/coloredgoroutine
  13. 13. visualize goroutines🐞 ? https://divan.dev/posts/go_concurrency_visualize/
  14. 14. Print scheduling events? $ GODEBUG=schedtrace=5000 <binary>
  15. 15. scheduling events
  16. 16. @a_soldatenko - delve - gdb using debuggers:
  17. 17.  Advanced debugging techniques of Go code
  18. 18. @a_soldatenko How to set breakpoint inside goroutine?package main import ( "fmt" ) func say(s string, r chan string) { fmt.Println(s) r <- s } func main() { chan1 := make(chan string) chan2 := make(chan string) go say("world", chan1) go say("hello", chan2) res1 := <-chan1 res2 := <-chan2 fmt.Printf("Channel 1: %snChannel 2: %sn", res1, res2) }
  19. 19. @a_soldatenko > main.say() ./main.go:9 (hits goroutine(7):1 total:1) (PC: 0x10c46c9) 4: "fmt" 5: ) 6: 7: func say(s string, r chan string) { 8: fmt.Println(s) => 9: r <- s 10: } 11: 12: func main() { 13: chan1 := make(chan string) 14: chan2 := make(chan string) How to set breakpoint inside goroutine?
  20. 20. @a_soldatenko
  21. 21. @a_soldatenko How to debug channel? package main func main() { ch := make(chan int, 4) ch <- 1 ch <- 2 ch <- 3 ch <- 4 close(ch) }
  22. 22. @a_soldatenko
  23. 23. @a_soldatenko How to debug channel?(dlv) p ch chan int { qcount: 0, dataqsiz: 4, buf: *[4]int [0,0,0,0], elemsize: 8, closed: 0, elemtype: *runtime._type {size: 8, ptrdata: 0, hash: 4149441018, tflag: tflagUncommon|tflagExtraStar| tflagNamed|tflagRegularMemory (15), align: 8, fieldAlign: 8, kind: 2, equal: runtime.memequal64, gcdata: *1, str: 663, ptrToThis: 22432}, sendx: 0, recvx: 0, recvq: waitq<int> { first: *sudog<int> nil, last: *sudog<int> nil,}, sendq: waitq<int> { first: *sudog<int> nil, last: *sudog<int> nil,}, lock: runtime.mutex {key: 0},} (dlv) n > main.main() ./main_ch.go:7 (PC: 0x105e8d9) 2: 3: 4: func main() { 5: ch := make(chan int, 4) 6: ch <- 1 => 7: ch <- 2 8: ch <- 3 9: ch <- 4 10: close(ch) 11: } (dlv) p ch
  24. 24. @a_soldatenko dlv send to channel value similar to set variable.
  25. 25. @a_soldatenko GDB t.me/golang_for_two
  26. 26. @a_soldatenko Gdb and golang go build -ldflags=-compressdwarf=false - gcflags=all="-N -l" -o main main.go
  27. 27. @a_soldatenko Gdb and goroutines
  28. 28. @a_soldatenko Gdb and goroutines
  29. 29. @a_soldatenko Deadlocks happen and are painful to debug.
  30. 30. @a_soldatenko How to detect deadlocks fatal error: all goroutines are asleep - deadlock! goroutine 1 [chan send]: main.main() /Users/andrii/workspace/src/github.com/andriisoldatenko/ gowayfest/main_deadlock.go:5 +0x50 exit status 2
  31. 31. @a_soldatenko Real world examples complicated scenario & tools https://github.com/sasha-s/go-deadlock https://github.com/cockroachdb/cockroach/issues/7972
  32. 32. @a_soldatenko
  33. 33. @a_soldatenko 7 simple rules for debugging concurrency applications - Never assume a particular order of execution - Implement concurrency at the highest level possible - Don’t forget Go only detects when the program as a whole freezes, not when a subset of goroutines get stuck. - STRACE
  34. 34. @a_soldatenko 7 simple rules for debugging concurrency applications - conditional breakpoints your best friend - DEBUG=schedtrace=5000 - go-deadlock
  35. 35. @a_soldatenko References 1 - https://github.com/golang/go/blob/release- branch.go1.14/src/runtime/HACKING.md - https://github.com/golang/go/wiki/LearnConcurrency - https://rakyll.org/go-cloud/ - https://yourbasic.org/golang/detect-deadlock/
  36. 36. @a_soldatenko References 2 - https://blog.minio.io/debugging-go-routine-leaks- a1220142d32c - https://golang.org/src/cmd/link/internal/ld/dwarf.go - https://golang.org/src/runtime/runtime-gdb.py - https://cseweb.ucsd.edu/~yiying/GoStudy- ASPLOS19.pdf - https://golang.org/doc/articles/race_detector.html
  37. 37. Telegram channel https://t.me/golang_for_two
  38. 38. Slides: @a_soldatenko
  39. 39. Thank You @a_soldatenko
  40. 40. Questions ? @a_soldatenko

×