"Session ID: BUD17-TR04 Session Name: Kernel Debug Stories - BUD17-TR04 Speaker: Track: ★ Session Summary ★ This session is a short introductory course on Linux kernel debugging. The course is will examine a number of different debugging challenges and discuss the techniques and tools that can be employed to overcome them. By focusing on stories rather than the minute details of each tool we can cover a lot of topics in a short space of time, providing a springboard for further independent study by trainees. --------------------------------------------------- ★ Resources ★ Event Page: http://connect.linaro.org/resource/bud17/bud17-tr04/ Presentation: https://www.slideshare.net/linaroorg/bud17tr04-kernel-debug-stories Video: https://youtu.be/LA2dxBMatnw --------------------------------------------------- ★ Event Details ★ Linaro Connect Budapest 2017 (BUD17) 6-10 March 2017 Corinthia Hotel, Budapest, Erzsébet krt. 43-49, 1073 Hungary --------------------------------------------------- Keyword: kernel, debug http://www.linaro.org http://connect.linaro.org --------------------------------------------------- Follow us on Social Media https://www.facebook.com/LinaroOrg https://twitter.com/linaroorg https://www.youtube.com/user/linaroorg?sub_confirmation=1 https://www.linkedin.com/company/1026961"

2. ENGINEERS AND DEVICES
WORKING TOGETHER

3. ENGINEERS
AND DEVICES
WORKING
TOGETHER
●
○
○
○
●
○
○
○
○
○
●
○
●

4. ENGINEERS AND DEVICES
WORKING TOGETHER
●
○
○
○ printk()
●
○
○
○
●
○
○
○
○
●
○

5. ENGINEERS AND DEVICES
WORKING TOGETHER
0x0 0x0000ffffffffffff 256TB
0xffff000000000000 0xffff000008000000 128MB
0xffff000008000000 0xffff00000807ffff 512KB
0xffff000008080000 10MB+
Kernel image specific 0xffff7dffbfff0000 ~126TB
0xffff7dfffe7fd000 0xffff7dfffec00000 4MB + 12KB
0xffff7dfffee00000 0xffff7dffffe00000 16MB
0xffff7e0000000000 0xffff800000000000 2048G
0xffff800000000000 0xffffffffffffffff 128TB

6. ENGINEERS AND DEVICES
WORKING TOGETHER

7. ENGINEERS AND DEVICES
WORKING TOGETHER
●
○ ⇒
○ ⇒
○ ⇒
○ ⇒
●
○ printk
○ git grep printk
●
○
○
○

8. ENGINEERS AND DEVICES
WORKING TOGETHER
●
○ FRAME_WARN
○ SCHED_STACK_END_CHECK
●
○ slub_debug=
■ SLUB_DEBUG
○ DEBUG_PAGEALLOC
○ PAGE_POISONING
●
○ DEBUG_SPINLOCK
○ DEBUG_MUTEXES
○ DEBUG_ATOMIC_SLEEP
○ DEBUG_LOCK_ALLOC
○ LOCKUP_DETECTOR
●

9. ENGINEERS AND DEVICES
WORKING TOGETHER
●
○
○
○
●
○
○
●
○
○ PROVE_LOCKING
○ KASAN

10. ENGINEERS AND DEVICES
WORKING TOGETHER
PROVE_LOCKING
lock(subsys) lock(driver)
lock(driver) lock(subsys)
DEBUG_SPINLOCK
DEBUG_MUTEXES
lock(subsys)
lock(driver)
unlock(driver)
unlock(subsys)
lock(driver)
lock(subsys)
PROVE_LOCKING
PROVE_LOCKING

11. ENGINEERS
AND DEVICES
WORKING
TOGETHER
●
○
○
○
●
○
○
○
○
○
●
○
●

12. ENGINEERS AND DEVICES
WORKING TOGETHER
●
●
●
●
●
●

13. ENGINEERS AND DEVICES
WORKING TOGETHER
make cscope git grep 'struct foo {'
printk()
[ 0.001636] xyz: Found 10 widgets ⇒ git grep "Found .* widgets"
I can’t reproduce but
my customer can

14. ENGINEERS AND DEVICES
WORKING TOGETHER
●
○
○
○
●
○
○ ⇒
○ quiet
LOG_BUF_SHIFT log_bug_len=
● pr_debug()
○ <8> <7>
○
○ DYNAMIC_DEBUG
I can’t reproduce but
my customer can

15. ENGINEERS AND DEVICES
WORKING TOGETHER
I can’t reproduce but
my customer can
SP
LR (r30)
FP (r29)
r19..r28
r18
IP0/IP1 (r16-r17)
r9..r15
r8
r0..r7

16. ENGINEERS AND DEVICES
WORKING TOGETHER
git grep REGMAP_ALLOW_WRITE_DEBUGFS
I can’t reproduce but
my customer can
config DEBUG_FS
bool "Debug Filesystem"
select SRCU
help
debugfs is a virtual file
system that kernel developers
use to put debugging files
into. Enable this option to be
able to read and write to
these files.
If unsure, say N.

17. ENGINEERS AND DEVICES
WORKING TOGETHER
I can’t reproduce but
my customer can
<release_thread>:
stp x29, x30, [sp,#-16]!
mov x29, sp
mov x0, x30
bl ffff000008092ea0 <_mcount>
ldp x29, x30, [sp],#16
ret
<release_thread>:
stp x29, x30, [sp,#-16]!
mov x29, sp
mov x0, x30
nop
ldp x29, x30, [sp],#16
ret
<release_thread>:
stp x29, x30, [sp,#-16]!
mov x29, sp
mov x0, x30
bl ftrace_caller
ldp x29, x30, [sp],#16
ret

18. ENGINEERS AND DEVICES
WORKING TOGETHER
●
●
○ ftrace=function ftrace_filter=mydrv_* tp_printk
○ ftrace=function_graph ftrace_graph_notrace=rcu*,*lock,*spin*
●
cd /sys/kernel/debug/tracing
echo function > current_tracer
echo 1 > tracing_on
●
○
○
○ ftrace_dump_on_oops
○
● trace_printk()
I can’t reproduce but
my customer can

19. ENGINEERS AND DEVICES
WORKING TOGETHER
/proc/vmcore
console=ttyS0,1115200 …
… crashkernel=128M
-p
./kexec -p vmlinux --dtb=xxx.dtb
--append="root=/dev/mmcblk0p9 rw 1
maxcpus=1 reset_devices"
I can’t reproduce but
my customer can

20. ENGINEERS AND DEVICES
WORKING TOGETHER
●
●
●
●
●
●
●
printk()

21. ENGINEERS AND DEVICES
WORKING TOGETHER
●
●
●
●
○
○
○
echo 'sched:*' > /sys/kernel/debug/tracing/set_event
My widget missed
its deadline

22. ENGINEERS AND DEVICES
WORKING TOGETHER
●
●
●
●
●
●
My widget missed
its deadline

23. ENGINEERS AND DEVICES
WORKING TOGETHER
●
●
●
●
.

24. ENGINEERS AND DEVICES
WORKING TOGETHER
●
●
●
●
●
●

25. ENGINEERS AND DEVICES
WORKING TOGETHER
●
○
○ initcall_debug
○ initcall_debug clk_disable_unused() clk_ignore_unused
○ DEBUG_SPINLOCK DEBUG_MUTEX LOCKUP_DETECTOR
○
●
○
○
○
○
○
My board just
stopped dead

26. ENGINEERS AND DEVICES
WORKING TOGETHER
●
○
●
○
○
○
●
○
○
○
●
○
○
My board just
stopped dead

27. ENGINEERS AND DEVICES
WORKING TOGETHER
●
●
●
○
○
○
●
●
●
My board just
stopped dead

28. ENGINEERS AND DEVICES
WORKING TOGETHER
My board just
stopped dead
[ 277.592103] PCSR of cluster0 cpu1 is 0xffff0000080c4f78
[ 277.592103] PCSR of cluster0 cpu1 is 0xffff0000080c4f78
[ 277.600833] PCSR of cluster0 cpu2 is 0xffff00000809a934
[ 277.609564] PCSR of cluster0 cpu3 is 0xffff00000809a934

29. ENGINEERS AND DEVICES
WORKING TOGETHER
●
●
●
●
○
○
○

30. ENGINEERS AND DEVICES
WORKING TOGETHER
git bisect start <bad_sha1> <good_sha1>
git bisect run compile_kernel_and_run_test.sh
compile_kernel_and_test.sh
compile_kernel_and_test.sh
I’m sure this
used to work

31. ENGINEERS AND DEVICES
WORKING TOGETHER
ktest.pl
compile_kernel_and_test.sh
Documentation/ tools/testing/ktest/sample.conf
tools/testing/ktest/examples
I’m sure this
used to work

32. ENGINEERS AND DEVICES
WORKING TOGETHER
●
●
●
●
●
●

33. ENGINEERS AND DEVICES
WORKING TOGETHER
My board just
randomly failed

34. ENGINEERS
AND DEVICES
WORKING
TOGETHER
●
○
○
○
●
○
○
○
○
○
●
○
●

35. ENGINEERS AND DEVICES
WORKING TOGETHER

36. ENGINEERS
AND DEVICES
WORKING
TOGETHER
Graphic by Jost, CC0-PD

37. ENGINEERS
AND DEVICES
WORKING
TOGETHER
●
○
○
○
●
○
○
○
○
○
●
○
●

38. ENGINEERS AND DEVICES
WORKING TOGETHERom: http://linaro.co/bud17-kdb