This presentation is about a methodology which allows patching of a running Linux kernel, its technical details, limitations as well as kpatch tools. The talk was delivered by Ruslan Bilovol (Associate Manager, Consultant, GlobalLogic) at GlobalLogic Embedded Career Day #2 on February 10, 2018. More about GlobalLogic Embedded Career Day #2: https://www.globallogic.com/ua/events/globallogic-kyiv-embedded-career-day-2-materials

1. Linux Kernel Live Patching
Ruslan Bilovol

2. Agenda
2
• Why?
• History
• From Kprobes to LivePatch
• LivePatch
• Limitations
• Current status in mainline Linux kernel
• Kpatch
• LivePatch writer recommendations
• Demo
• Questions

3. Why?
3
• High cost of downtime caused by reboot
• Apply hotfixes without reboot
• Maximize uptime
• Scheduled downtime (delay reboot)

4. Linux Kernel Live Patching: History
• 2008: ksplice
– university research, acquired by Oracle, source is closed
• 2014: kPatch (Red Hat)
– stopping the kernel for live patching
– manual/automatic patches creation
• 2014: kGraft (SUSE)
– lazy kernel live patching
– only manual patches creation
• 2015: Livepatch (Linux kernel 4.0+)
– combined consistency model of kGraft + kPatch
4

5. Commercial use
5
• Oracle Ksplice is available for Oracle Linux and RHEL servers with an
Oracle Linux Premier Support license
– $2299/node/year
• RHEL Livepatching requires a Red Hat Enterprise Linux Server
Premium Subscription
– $1299/node/year
• SUSE Livepatching is available as an add-on to SUSE Linux
Enterprise Server 12 Priority Support subscription
– $1,499/node/year
• Canonical Livepatching service

6. From Kprobes to LivePatch

7. Kprobes
7
• Enables you to dynamically break into any kernel routine
• Can trap at almost any kernel code address
– you can probe even interrupt handlers
• Probed instruction replaced with a breakpoint instruction
• On trap executes "pre_handler", replaced instruction and
"post_handler"

8. Kprobes
8

9. Ftrace
• "Kernel Function Tracer"
• Enables the kernel to trace every kernel function
• Uses 'gcc -pg' option to insert “mcount” function call placed at the start of
every kernel function
• Big impact on performance (13%)
9

10. Dynamic Ftrace
• "Dynamic Kernel Function Tracer"
• mcount points to a simple return
– Starting with gcc verson 4.6, the -mfentry has been added for x86,
which calls " fentry " instead of "mcount"
• Build-time tool parses ELF to find these location
• On boot dynamic ftrace updates all the locations into NOPs
10

11. Dynamic Ftrace
• Then NOP sequence can be dynamically patched into a tracer call
• Modules are processed as they are loaded and before they are
executed
• No measurable performance degradation
11

12. LivePatch?
12
• So we can replace functions in running kernel?
• ...
• Live Kernel Patching!

13. LivePatch
13
• Need to be more clever
– SMP synchronization
– task may sleep in a to-be-patched function
– function semantic changes
– data structure changes
– what if patch contains multiple functions that depend on each other

14. The LivePatch

15. LivePatch
15
• Initially implemented for x86-64 (kernel v4.0)
– extended to PPC64 and s390
• Is a .ko module
– just like any other kernel module
• Uses dynamic FTRACE to replace original function
• Creates /sys/kernel/livepatch dir
– to monitor live patches status

16. LivePatch
16

17. Hybrid Consistency Model
17
• Avoid calling a new function from old and vice versa
• A thread calls either all old functions or all new ones
• Per-thread migration
– stack checking of sleeping tasks
– new thread inherits its parent patched state
– interrupt handler inherits interrupted thread state
• Per-patched-set migration
• Stack checking
– need reliable stack information
– frame pointers are not fast enough
• Interrupt handler inherits the patched state of interrupted task

18. Hybrid Consistency Model
18

19. ORC unwinder
19
• replaces GCC frame pointers approach
– much simpler and faster
– performance improvement (5-10% for some workloads)
– 3.2% kernel .text savings
• objtool for stack’s metadata validity on .o files and ORC data
generation
• disadvantage: needs more memory to store the ORC unwind tables
• enabled by default since v4.14

20. Shadow variables
20
• associate additional "shadow" data with existing data structures
– allocated separately
• based on global in-kernel hashtable
• associates pointers to parent objects and a numeric identifier of the
shadow data
– there may be multiple shadow variables attached to parent object

21. Modules live patching
21
• patch can be inserted before module loading
• LivePatch core gets notification on module load/unload
– patches module before it get executing
– unpatches module on module unload
• module can be inserted-removed multiple times

22. Patch/unpatch callbacks
22
• execute callback functions when a kernel object is (un)patched
• considered a "power feature"
– safe updates to global data
– "patches" to init and probe functions
– patching otherwise unpatchable code (i.e. assembly)
– ability refuse to load livepatch or a module
• originally livepatch didn’t have such ability

23. LivePatch example
23
static int livepatch_cmdline_proc_show(struct seq_file *m, void *v) { seq_printf(m, "%sn", "this has been live
patched");
return 0;
}
static struct klp_func funcs[] = { { .old_name = "cmdline_proc_show", .new_func = livepatch_cmdline_proc_show, }, { } };
static struct klp_object objs[] = { { /* name being NULL means vmlinux */ .funcs = funcs, }, { } };
static struct klp_patch patch = { .mod = THIS_MODULE, .objs = objs, }; static int livepatch_init(void) {
int ret;
ret = klp_register_patch(&patch); if (ret) return ret;
ret = klp_enable_patch(&patch);
if (ret) { WARN_ON(klp_unregister_patch(&patch)); return ret; } return 0;
}
static void livepatch_exit(void) { WARN_ON(klp_unregister_patch(&patch));
}
module_init(livepatch_init); module_exit(livepatch_exit);

24. LivePatch Limitations
24
• useful for small and critical fixes
– e.g. can’t do kernel upgrade to a new version
• Kprobes in the original function are ignored
• can’t do semantics changes
• can’t do data structure changes
• inability to patch hand-written ASM
– e.g. no fentry
– can be hacked by patch/unpatch callbacks feature
• vDSO can’t be patched (-pg is removed)

25. How it looks like?
25

26. Current status in mainline Linux kernel
Initial LivePatch support (x86-64)
26
v4.0
s390 architecture support v4.1
PPC64 architecture support v4.7
New consistency model (allows removal of livepatch modules) v4.12
ORC unwinder v4.14
Shadow variables v4.15
Patch/unpatch callbacks v4.15
Removed “immediate” feature v4.16
Atomic replace feature in progress (v4.17?)
AArch64 architecture support (depends on GCC 8.x) in progress (v5.???)

27. kpatch Set of Tools

28. Why need automatic patches generation?
28
• Human can produce errors
• Inlining
– inlined buggy function to other functions
• GCC can do non-trivial optimization by default
• Various IPA (inter-procedural optimization)
– transform arguments passed by reference to arguments passed by value
– create variants of a function with fewer arguments
– optimize functions if values passed to them are constants
• several clones of a function are possible

29. Kpatch
29
• Red Hat project of live kernel patching
– includes few userspace tools
• Used in production (RH relies on it)
• Now compatible with upstream Livepatch
• Consists of 3 parts
– kpatch/Livepatch kernel module stub
– run-time patches management on target
• systemd service to load livepatches on boot
• ‘kpatch’ tool to manage patches
– kpatch-build set of tools for automatic patches generation

30. kpatch-build
30
• build original kernel
– with -ffunction-sections -fdata-sections flags
– use custom script as cross-compiler to collect .o files
• build patched kernel
– watch for changed objects
• analyze changed objects
– compare the objects at a section level
– apply various rules to determine any object local sections that are
dependencies of the changed section
– use a lot of additional magic
– create a diff of changed objects
• attach changed functions to LivePatch kernel stub
– finally generate .ko file

31. kpatch limitations
31
• can’t do .data section changes (don’t remove static variables)
• no cross-compilation support (patch is in upstream)
• no out-of-src build
• patching of original sources
• supports only RHEL/Fedora/OracleLinux and Ubuntu/Debian
– Yocto support is on the way
• every patch must also be analyzed in-depth by a human
• can’t generate patches with shadow variables, load/unload hooks
• no out-of-tree modules support

32. LivePatch writer recommendations
32
• make changes as small as possible
– e.g. this is “hot fix”, not a kernel upgrade to a new version
• be careful with data structure changes
• be careful with synchronization changes
– you can create deadlock situation
• watch for inlined functions
• FUNC , line preprocessor macros
– resulting code may be changed unexpectedly
• code removal
– Livepatch replaces function, not removes it
• static variables
– don’t remove them, keep unused (kpatch-build tracks .data changes)
• https://github.com/dynup/kpatch/blob/master/doc/patch-author-guide.md

33. Demo

34. Questions?

35. Sources
35
kernel.org
https://assets.ubuntu.com/v1/ac3aa269-DS_Canonical_Livepatch_Service_sc
reen-AW_08.17.pdf
https://www.linuxplumbersconf.org/2014/ocw/system/presentations/1773/origi
nal/ftrace-kernel-hooks-2014.pdf
https://osseu17.sched.com/event/BxJF/kernel-live-patching-current-state-and-
future-development-miroslav-benes-suse
https://github.com/dynup/kpatch