This talk will start with an overview of Go, then dive into some examples of using it to work with Postgres. We'll show basics like running queries, then demonstrate how Go makes difficult things easy, such as inspecting Postgres's TCP wire protocol and providing retry mechanisms and monitoring for restores (including fun stories, tips and tricks). Join us and see why Go - with it's powerful concurrency primitives and unbeatable performance - can be one of the most powerful tools in your toolbox.

1. Gophers Riding Elephants:
Writing PostgreSQL Tools in Go
by AJ Bahnken,
Senior Engineer @ Procore

2. Who am I?
Senior Engineer @ Procore
Work on availability, performance, and (now mostly)
security
Been writing Go code actively for 2 years
Twitter:
Email: aj.bahnken@procore.com
@ajvbahnken

3. Who is this talk for?

4. Overview of Go

6. Go was created by Google

7. It was built for Google as well

8. Reliable
Good for teams
Good for building the kind of things they need to build

9. "Less is exponentially more" by Rob Pike
Go is purposefully lacking certain features.
Link: https://commandcenter.blogspot.com/2012/06/less-is-
exponentially-more.html

10. Statically Typed
Compiled
Garbage Collected

11. package main
import "fmt"
func main() {
fmt.Println("Hello, 世界")
}

12. Why Go with Postgres?

13. 1. Performance

14. It's pretty fast

15. Garbage collector is pretty solid

16. Concurrency is a breeze
package main
import (
"fmt"
"time"
)
func say(s string) {
for i := 0; i < 5; i++ {
time.Sleep(100 * time.Millisecond)
fmt.Println(s)
}
}
func main() {
go say("world")
say("hello") // Allows for the goroutine to run, by blocking.
}

17. $ go run main.go
world
hello
hello
world
world
hello
hello
world
world
hello

18. 2. Reliability

19. Statically Typed (yay!)
bool
string
int int8 int16 int32 int64
uint uint8 uint16 uint32 uint64 uintptr
byte // alias for uint8
rune // alias for int32
// represents a Unicode code point
float32 float64
complex64 complex128

20. Simple
type EventProcessor struct {
eventQueue chan Event
}
func (ep *EventProcessor) Add(event Event) {
ep.eventQueue <­ event
}
func (ep *EventProcessor) Start() {
for {
event := <­ep.eventQueue
go event.Process()
}
}

21. Testing is simple and built in + race detector
$ ls
processing.go processing_test.go utils.go
$ go test
PASS
ok ~/pgnetdetective/processing 0.165s
$ go test ­­race
PASS
ok ~/pgnetdetective/processing 2.133s

22. Error handling instead of exceptions
func MyFunc() (string, error) {
str, err := run()
if err != nil {
return "", err
}
return str, nil
}
func MustMyFunc() string {
str, err := run()
if err != nil {
panic("run() returned an err: "+err.String())
}
return str
}

23. 3. Ease of Use

24. Tooling
(Gofmt, testing, godocs, go build/run, vim-go)

25. Familiarity

26. Library support and ease of installation
$ go get github.com/urfave/cli

27. Distribute a single binary anywhere
$ go build
$ file dbduke
dbduke: Mach­O 64­bit executable x86_64
$ GOOS=linux go build
$ file dbduke
dbduke: ELF 64­bit LSB executable, x86­64, version 1 (SYSV),
statically linked, not stripped
$ GOOS=linux GOARCH=386 go build
$ file dbduke
dbduke: ELF 32­bit LSB executable, Intel 80386, version 1
(SYSV), statically linked, not stripped

28. Performance
Reliability
Ease of Use

29. Interacting with
Postgres in Go

30. database/sql
Docs: https://golang.org/pkg/database/sql/

31. Provides core interface for interacting with SQL databases
Open / Close
Begin / Rollback / Commit
Exec / Query / QueryRow
Ping / Connection Pooling

32. go get github.com/lib/pq

33. package main
import (
"database/sql"
"fmt"
_ "github.com/lib/pq"
)
func main() {
dbUrl := "postgres://postgres@localhost:5432/postgres"
db, err := sql.Open("postgres", dbUrl)
if err != nil {
panic(err.String())
}
var result int
err = db.QueryRow('SELECT 1').Scan(&result)
if err != nil {
panic(err.String())
}
fmt.Printf("1 == %d", result)
}

34. http://go-database-sql.org/

35. Example #1
pgnetdetective
https://github.com/procore/pgnetdetective

40. ?????

43. tcpdump ­n ­w ~/pg.cap ­i any port 5432
~1GB every 10 seconds

44. We needed something faster,
so I decided to rewrite it in Go

45. https://github.com/google/gopacket
Provides packet processing
capabilities for Go

46. // If the destination port is 5432...
if tcp.DstPort == 5432 {
// And the packet payload starts with P or Q...
raw = fmt.Sprintf("%s", tcp.Payload)
if strings.HasPrefix(raw, "P") || strings.HasPrefix(raw, "Q") {
// It is a Parse or Query packet, therefore it contains a Query
combinedQueryMetrics.Add(
metrics.New(
NormalizeQuery(raw),
1,
ip.SrcIP,
tcp.Seq,
),
)
}
} else if tcp.SrcPort == 5432 && tcp.ACK {
responses = append(responses, &ResponsePacket{
DstIP: ip.DstIP,
Ack: tcp.Ack,
Size: uint64(len(tcp.Payload)),
})
}

47. So I got some output like this:
******* Query *******
Query: SELECT attr.attname FROM pg_attribute attr
INNER JOIN pg_constraint cons ON attr.attrelid = cons.conrelid
AND attr.attnum = any(cons.conkey) WHERE cons.contype = p
AND cons.conrelid = "drawing_log_imports"::regclass
TotalNetBytes: 170 MB
TotalResponsePackets: 64041
TotalQueryPackets: 63

48. ummm, catalog queries?

49. Introducing: Resque
https://github.com/resque/resque
http://resque.github.io/

50. ~10,000 jobs per hour
1-8 tables being touched per job
average of 20 columns per table.

51. During spikes, this can get up to 120MB per second.
On to Sidekiq we go...

52. Where Go won with pgnetdetective:
Performance
Community (Ease of Use)

54. Example #2 dbduke
(not open source + still under active development)

55. Context:
1. We restore staging/qa/testing databases frequently
2. It's important that they successfully restore

56. Problem:
1. When restores fail, productivity dies
2. The process of kicking restores off by hand is faulty

57. Further Context for a Solution:
1. Restores sometimes fail from easily recoverable errors

58. A tool for making restores of Postgres
databases manageable and fault tolerant.

59. A tool for making restores of Postgres
databases manageable and fault tolerant.

60. Manageable
Run dbduke as a daemon with jobs

61. $ dbduke jobs
­­­­­­­­­­­­
DBDuke
­­­­­­­­­­­­
* restore ­ 35e1ca93­936b­4c73­8812­b1a69d708791
database: postgres
dumpfile: /data/lite­dump.dmp
started: 17:19:59 Tue Oct 11, 2016 ­0700
flags: ­­no­big­tables ­­maintenance

62. A tool for making restores of Postgres
databases manageable and fault tolerant.

63. Fault Tolerance
Treat restores as a state machine
and recover from failure states

65. Error handling in practice:
1. Error out
2. Log a warning
3. Retry with timeout (with or without backoff)

66. Error out
db, err := sql.Open("postgres", dbUrl)
if err != nil {
log.Fatalf("Could not open postgres db @ `%s`", dbUrl)
}

67. Log warning
query := "DROP SCHEMA IF EXISTS _migration CASCADE"
_, err = db.Exec(query)
if err != nil {
log.Warnf("Query `%s` failed with err: %v", query, err)
}

68. Retry with timeout (without backoff)
func (r *Restorer) BlockTillNotInUse() {
if r.State == state.InUse {
log.Warn("State is currently InUse. Going into retry loop.")
for {
time.Sleep(time.Second * 15)
r.QuitIfTimeout()
currentState, err := state.GetCurrentState()
if err != nil {
log.Errorf(
"Error getting current state. Err: %v",
err,
)
break
}
if currentState != state.InUse {
r.State = currentState
break
}
}
}
}

69. Manageability + Fault Tolerance
Go makes it easy! ™

70. Where Go won with dbduke:
Error Handling (Reliability)
Concurrency (Performance/Ease of Use)

71. In Conclusion
In the context of tool building
Go = Reliability, Performance, and Ease of
Use

75. Procore is hiring! (big surprise)
http://procore.com/careers

76. Thank you! Questions?
aj.bahnken@procore.com / @ajvbahnken

77. Further Resources
Tour of Go
Effective Go (required reading)
Great intoduction to using SQL within Go
Why we import drivers with '_'

78. Sources
Performance Graph