The slides are for GolangUK Conference 2017. https://www.golanguk.com/

1. The Go gopher was designed by Renée French.
The gopher stickers was made by Takuya Ueda.
Licensed under the Creative Commons 3.0 Attributions license.
Static Analysis in Go
@GolangUK Conference
18th Aug. 2017
1

2. Who am I?
Takuya Ueda
@tenntenn
➔ Work for
➔ Communities
&
Go Beginners
in Tokyo
Go Conference
in Tokyo
2

3. Mercari Europe
3
https://www.mercari.com/uk/

4. Who am I?
Takuya Ueda
@tenntenn
➔ Work for
➔ Communities
&
Go Beginners
in Tokyo
Go Conference
in Tokyo
4

5. Agenda
➔ Where’s Gopher?
➔ Static Analysis
➔ Static Analysis in Go
➔ Static Analysis for Products
5

6. Where’s Gopher?
Find Me!!
Powered by https://gopherize.me
6

7. Where’s Gopher?
Find Me!!
Powered by https://gopherize.me
7

8. Where’s “Gopher”?
type Gopher struct { Gopher string `json:"gopher"` }
func main() {
const gopher = "GOPHER"
gogopher := GOPHER()
gogopher.Gopher = gopher
fmt.Println(gogopher)
}
func GOPHER() (gopher *Gopher) {
gopher = &Gopher{ Gopher: "gopher" }
return
}
8

9. We love grep
$ grep Gopher main.go
type Gopher struct { Gopher string `json:"gopher"` }
gogopher.Gopher = gopher
func GOPHER() (gopher *Gopher) {
gopher = &Gopher{ Gopher: "gopher" }
We can search “Gopher” with grep.
9

10. Where’s “Gopher” TYPE?
type Gopher struct { Gopher string `json:"gopher"` }
func main() {
const gopher = "GOPHER"
gogopher := GOPHER()
gogopher.Gopher = gopher
fmt.Println(gogopher)
}
func GOPHER() (gopher *Gopher) {
gopher = &Gopher{ Gopher: "gopher" }
return
}
10

11. How to search “Gopher” type
➔ grep can search only by text
➔ The text must be understood as Go source
code
➔ We need “Static Analysis”
11

12. Static Analysis
12

13. Static Analysis & Dynamic Analysis
➔ Static Analysis
◆ Analyzing a program WITHOUT execution
◆ Analyzing structure of a program from source codes
◆ e.g. linter, code complement, code formatter
➔ Dynamic Analysis
◆ Analyzing a program WITH execution
◆ Investigating variables and result of functions
◆ e.g. race detector
13

14. Reflection
➔ Reflection
◆ Analyzing values and types at runtime.
➔ Reflection in Go
◆ reflect package
◆ It is only way to get struct tags at runtime
◆ encoding package uses reflection to
encode/decode JSONs, XMLs and so on.
14

15. Static Analysis Tools for Go
➔ There are many static analysis tools for Go
gofmt/goimports code formatter
go vet/golint code checker, linter
guru code comprehension tool
gocode code complement tool
errcheck error handlings checker
gorename/gomvpkg refactoring tools
15

16. Go for Static Analysis
➔ Go is easy to static analyze
◆ Static typing
◆ Simple syntax
◆ Type inference
◆ No Implicit type conversion
◆ go package is provided as a standard package
Static Analysis gives
a lot of information
16

17. Sub packages of go package
ast Package ast declares the types used to represent syntax trees for Go packages.
build Package build gathers information about Go packages.
constant Package constant implements Values representing untyped Go constants and their corresponding operations.
doc Package doc extracts source code documentation from a Go AST.
format Package format implements standard formatting of Go source.
importer Package importer provides access to export data importers.
parser Package parser implements a parser for Go source files.
printer Package printer implements printing of AST nodes.
scanner Package scanner implements a scanner for Go source text.
token
Package token defines constants representing the lexical tokens of the Go programming language and basic
operations on tokens (printing, predicates).
types Package types declares the data types and implements the algorithms for type-checking of Go packages.
17

18. Static Analysis in Go
18

19. Flow of Static Analysis
Source Code
Token
Abstract Syntax Tree
(AST)
Type Info
Parse
Tokenize
Type Check
go/scanner
go/token
go/parser
go/ast
go/types
go/constant
19

20. Tokenization - go/scanner,go/token
IDENT ADD INT
Tokens
Source Code: v + 1
➔ Dividing input string into tokens
20

21. Parsing - go/parser,go/ast
➔ Converting tokens to abstract syntax tree
v + 1
IDENT ADD INT
Source Code:
+
v 1
BinaryExpr
Ident BasicLit
Tokens:
Abstract Syntax Tree:
(AST)
21

22. AST of “Hello, World”
package main
import "fmt"
func main() {
fmt.Println("Hello, 世界")
}
Run on Playground
*ast.File
[]ast.Decl
*ast.GenDecl *ast.FuncDecl
22

23. Type Check - go/types,go/constant
➔ Extract type infomation from AST
◆ Identifier Resolution
◆ Type Detection
◆ Constant Evalution
n := 100 + 200
m := n + 300
Constant Evalution
= 300
Type Detection
-> int
Identifier Resolution
23

24. Getting AST from source code
➔ Use parser.Parse* function
◆ ParseExpr,ParseExprFrom
● Parsing expression
● ParseExpr is simple version of ParseExprFrom
◆ ParseFile
● Parsing a file
◆ ParseDir
● Parsing files in a directory
● Calling ParseFile in the function
24

25. Getting AST from an expression
expr, err := parser.ParseExpr(`v + 1`)
if err != nil {
/* handling the error */
}
/* use expr */
➔ Parsing an expression
25

26. Getting AST from a file
const src = `
package main
var v = 100
func main() {
fmt.Println(v+1)
}`
fs := token.NewFileSet()
f, err := parser.ParseFile(fs, "my.go", src, 0)
if err != nil {
/* handling the error */
}
/* use f */
file name which is opened when
src is nil
Source Code
Parsing Mode
26

27. token.FileSet
➔ Recording positions on files
◆ The position (token.Pos) is represented by integer
◆ The value is unique among multiple files
◆ token.FileSet holds offset of each files
◆ The offsets are recorded at parsing
◆ token.FileSet is passed to parsing function as
an output parameter
type Pos int
27

28. Demo 1: Parse and Dump an AST
28
https://youtu.be/lM1Pj6xYxZs

29. Traversing AST - ast.Inspect
➔ Using ast.Inspect
expr, _ := parser.ParseExpr(`v + 1`)
ast.Inspect(expr, func(n ast.Node) bool {
if n != nil { fmt.Printf("%Tn", n) }
return true
})
Traverse the AST
*ast.BinaryExpr
*ast.Ident
*ast.BasicLit
Run on Playground
ast.Walk is more powerful and complex
+
v 1
BinaryExpr
Ident BasicLit
29

30. Traversing AST - Recursively
func traverse(n ast.Node) {
switch n := n.(type) {
case *ast.Ident:
fmt.Println(n.Name)
case *ast.BinaryExpr:
traverse(n.X)
traverse(n.Y)
case *ast.UnaryExpr:
traverse(n.X)
default:
fmt.Println(n)
}
}
print idetifyer’s name
traverse each terms recursively
switching by types
Run on Playground
traverse a term recursively
30

31. Demo 2: Inspect identifiers
31
https://youtu.be/mpUgaaASvHo

32. Type Checking
/* initialize configs for type checking */
cfg := &types.Config{Importer: importer.Default()}
info := &types.Info{
/* TODO: initialize maps which will hold results */
}
pkg,err := cfg.Check("main", fs, []*ast.File{f}, info)
if err != nil {
/* handling the error */
}
/* TODO: use pkg or info */
➔ (*types.Config).Check do a type checking
32

33. types.Info holds results of type checking
type Info struct {
// Types maps expressions to their types, and for constant
// expressions, also their values.
Types map[ast.Expr]TypeAndValue
// Defs maps identifiers to the objects they define.
Defs map[*ast.Ident]Object
// Uses maps identifiers to the objects they denote.
Uses map[*ast.Ident]Object
// Implicits maps nodes to their implicitly declared objects, if any.
Implicits map[ast.Node]Object
// Selections maps selector expressions (excluding qualified identifiers)
// to their corresponding selections.
Selections map[*ast.SelectorExpr]*Selection
// Scopes maps ast.Nodes to the scopes they define.
Scopes map[ast.Node]*Scope
// InitOrder is the list of package-level initializers in the order in which
// they must be executed.
InitOrder []*Initializer
}
33

34. Demo 3: Inspect Gopher type
34
https://youtu.be/AuSDtmiMaXI

35. Static Analysis for Products
Case 1: gpath,go-httpdoc
35

36. go.mercari.io/go-httpdoc
➔ Generate API Docs from tests of handlers
◆ Just write tests of HTTP handlers
◆ gRPC and JSON RPC are supported
36

37. Tests for requests and responses
➔ Requests (Responses) can be test by
specifying fields and expected values,
document for each fields
validator.RequestBody(t, []httpdoc.TestCase{
{"Name", "tenntenn", "User Name"},
{"Attribute.Email", "tenntenn@example.com", "e-mail address"},
}, &createUserRequest{})
37

38. github.com/tenntenn/gpath
➔ Simplefy reflection of struct fields
◆ Reflecting by expression of selectors
◆ Easy to reflect complex struct value
◆ Supports maps and slices (but restricted)
type Bar struct { N []int }
type Foo struct { Bar *Bar }
f := &Foo{ Bar: &Bar{ N: []int{100} }}
v, _ := At(f, `Bar.N[0]`)
fmt.Println(v)
$ go run main.go
100
38

39. Create own static analysis tools
➔ Reduce costs of code reviews by own tools
◆ Custimized linter for your project
◆ Detecting typical bugs with own tools
◆ You should pay your time for more important things
● algorithm, design, performance and so on
39

40. Static Analysis in Production
Case 2: Banner Tool
40

41. Banner Tool
➔ A management tool of in-app banners
In-App Banner
A screenshot of our app (Mercari Atte)
41

42. Banner Tool
➔ A management tool of in-app banners
Banner Tool
● Delivery Conditions
● Response
Getting Banners
w/ OS, API Version
List of Delivered Banners
w/ Image URL, Destination URL, etc...
Apps
42

43. Evaluation of Delivery Conditions
Getting Banners
GET /banner/?os=1
String(os) == "1"
Banner Image, etc...
Delivery Condition:
Banner Tool
"1" == "1"
true
BIND
EVAL
➔ Describing delivery conditions by a Go like expression
◆ Eval expressions with go package
◆ Using a function calling for describing a variable with a type
43

44. Conclutions
➔ Static Analysis is EASY in Go
◆ static typing, simple syntax, go package, etc...
➔ go package can
◆ tokenize, parse and type check
➔ Static analysis in production
◆ own static analysis tools (e.g. gpath, go-httpdoc)
◆ web apps (e.g. Banner Tool)
44

45. Thank you!
twitter: @tenntenn
Qiita: tenntenn
connpass: tenntenn
45