Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
kGraft 
Live patching of the Linux kernel 
Jiˇrí Kosina, Petr Mládek, Vojt ˇech Pavlík, Jiri Slaby 
SUSE Labs 
September 2...
Why Live Patching? 
1000 machines & severe security problem 
Needs fixing now! 
Rebooting the machines 
Is not a quick way...
Where is Live Patching Useful? 
Common tiers of change management 
1 Incident response – we are exploited 
2 Emergency cha...
Presentation Outline 
1 KGRAFT 
2 KGRAFT Internals 
3 Live Demo 
4 Conclusion 
Jiri Slaby (SUSE Labs) kGraft September 25t...
Section 1 
KGRAFT 
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 5 / 23
KGRAFT 
Research project 
Live patching technology 
Developed by SUSE Labs 
Specifically for the Linux kernel 
Based on mo...
Advantages of KGRAFT 
Does not require stopping the kernel 
Ever! 
Not even for short time periods 
Unlike competing techn...
How does KGRAFT work? 
A kGraft patch is a .ko kernel module 
The .ko is inserted into the kernel using insmod 
All linkin...
Limitations 
kGraft is designed for fixing critical bugs 
Primarily for simple changes 
Changes in kernel data structure l...
Section 2 
KGRAFT Internals 
Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 10 / 23
KGRAFT and fentry 
KGRAFT needs some space at the start of a function 
To insert a jump to a patched function 
The space c...
Using 5-byte NOPs Space 
KGRAFT uses the ftrace infrastructure to perform patching 
INT3 handler is installed with a JMP t...
Patching a Function 
Patching during runtime, no stop_kernel(); 
Callers are never patched 
Rather, callee’s NOPs are repl...
Patching a Function in Pictures 
kernel_func 
buggy_func(); 
buggy_func 
JMP fixed 
fixed_func 
Jiri Slaby (SUSE Labs) kGr...
Issue: Non-consistency 
What happens when 
Replaced function changes semantics and subsequent calls rely 
on each other? 
...
Cure: RCU-like Replacement 
We need to provide a consistent “world-view” to each thread 
User processes 
Kernel processes ...
RCU-like Replacement 
kernel_func 
heavy work 
buggy_func(); 
reality_check 
which universe 
are you 
coming from? 
buggy_...
Lazy Replacement 
All processes must wake up or execute a syscall 
Sometimes this requires a signal to be sent (like for g...
Lazy Replacement 
kernel_func 
heavy work 
buggy_func(); 
buggy_func 
fixed_func 
Userspace 
Kernelspace 
Jiri Slaby (SUSE...
Get It 
Upstreaming 
KGRAFT was submitted and reviewed upstream 
There are other groups working on competing technologies ...
Maintenance 
KGRAFT patch is an RPM package 
Once installed, always protected 
Jiri Slaby (SUSE Labs) kGraft September 25t...
Live Demo 
Kernel with a security vulnerability 
Exploit program 
kGraft patch 
Jiri Slaby (SUSE Labs) kGraft September 25...
Conclusion 
SUSE provides 
Demanded live Linux kernel patching 
Dubbed KGRAFT 
Jiri Slaby (SUSE Labs) kGraft September 25t...
Upcoming SlideShare
Loading in …5
×

Kernel Recipes 2014 - kGraft: Live Patching of the Linux Kernel

1,533 views

Published on

The talk introduces the need of live kernel patching. Further, it explains what is kGraft, how it works, what are its limitations, and our plans with the implementation in the future. The presentation includes also a live demo if stars constellation allows.

Jiri Slaby, SUSE

Published in: Software
  • Be the first to comment

Kernel Recipes 2014 - kGraft: Live Patching of the Linux Kernel

  1. 1. kGraft Live patching of the Linux kernel Jiˇrí Kosina, Petr Mládek, Vojt ˇech Pavlík, Jiri Slaby SUSE Labs September 25th 2014 Paris, France Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 1 / 23
  2. 2. Why Live Patching? 1000 machines & severe security problem Needs fixing now! Rebooting the machines Is not a quick way to fix an issue Has a risk of not coming up Live patching Allows quick response Leaves an actual update to a scheduled downtime window Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 2 / 23
  3. 3. Where is Live Patching Useful? Common tiers of change management 1 Incident response – we are exploited 2 Emergency change – we could be exploited (we are vulnerable) 3 Scheduled change – time is not critical, we are safe Live patching fits in with 1 and 2 Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 3 / 23
  4. 4. Presentation Outline 1 KGRAFT 2 KGRAFT Internals 3 Live Demo 4 Conclusion Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 4 / 23
  5. 5. Section 1 KGRAFT Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 5 / 23
  6. 6. KGRAFT Research project Live patching technology Developed by SUSE Labs Specifically for the Linux kernel Based on modern Linux technologies INT3/IPI-NMI self-modifying code Lazy update mechanism fentry-based NOP space allocation Standard kernel module loading/linking mechanisms Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 6 / 23
  7. 7. Advantages of KGRAFT Does not require stopping the kernel Ever! Not even for short time periods Unlike competing technologies Allows code review on KGRAFT patch sources Patches can be built from C source directly No need for object code manipulation Only an alternative: object code based automated patch generation kGraft is lean Small amount of code Leveraging other Linux technologies No complex instruction en/decoders or such Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 7 / 23
  8. 8. How does KGRAFT work? A kGraft patch is a .ko kernel module The .ko is inserted into the kernel using insmod All linking (incl. the fix) is done by kernel KGRAFT replaces whole functions in the kernel Even while those functions may be executed An updated KGRAFT module can replace an existing patch Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 8 / 23
  9. 9. Limitations kGraft is designed for fixing critical bugs Primarily for simple changes Changes in kernel data structure layout require special care Depending on the size of the change, reboot may be needed Same as with other live patching techniques KGRAFT depends on a stable build environment Having history of built kernels Best suited for Linux distributions Their customers Anyone who builds their own kernels Not good for 3rd party support Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 9 / 23
  10. 10. Section 2 KGRAFT Internals Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 10 / 23
  11. 11. KGRAFT and fentry KGRAFT needs some space at the start of a function To insert a jump to a patched function The space can be provided by GCC profiling -pg -mfentry KGRAFT uses this fentry call instructions Patched out at boot Replaced with 5-byte NOPs kernel_func CALL fentry kernel_func 5-byte NOP Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 11 / 23
  12. 12. Using 5-byte NOPs Space KGRAFT uses the ftrace infrastructure to perform patching INT3 handler is installed with a JMP to the destination address 1 First byte of NOP is replaced by INT3 2 Remaining bytes are replaced by address 3 First byte is replaced by JMP 4 NMI IPIs are used to flush instruction decoders on other CPUs kernel_func kernel_func 5-byte NOP INT3 | xxxx INT3 handler JMP addr kernel_func INT3 | addr kernel_func JMP addr Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 12 / 23
  13. 13. Patching a Function Patching during runtime, no stop_kernel(); Callers are never patched Rather, callee’s NOPs are replaced by a JMP to the new function JMP remains forever But this takes care of function pointers, including in structures Like indirect calls (handler->function()) Does not require saving any old data in case we want to un-patch Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 13 / 23
  14. 14. Patching a Function in Pictures kernel_func buggy_func(); buggy_func JMP fixed fixed_func Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 14 / 23
  15. 15. Issue: Non-consistency What happens when Replaced function changes semantics and subsequent calls rely on each other? It is called recursively? kernel_func buggy_func(); buggy_func(); BOOM! buggy_func KGRAFT patch comes in fixed_func Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 15 / 23
  16. 16. Cure: RCU-like Replacement We need to provide a consistent “world-view” to each thread User processes Kernel processes Interrupts Solution: “reality check” trampoline Per-thread flag set on each kernel entry/exit Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 16 / 23
  17. 17. RCU-like Replacement kernel_func heavy work buggy_func(); reality_check which universe are you coming from? buggy_func fixed_func Userspace Kernelspace Welcome to the new universe! Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 17 / 23
  18. 18. Lazy Replacement All processes must wake up or execute a syscall Sometimes this requires a signal to be sent (like for getty’s) Once all processes have the "new universe" flag set Patching is complete Trampolines can be removed Files to check /proc/<pid>/kgr_in_progress /sys/kernel/kgraft/kgr_in_progress Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 18 / 23
  19. 19. Lazy Replacement kernel_func heavy work buggy_func(); buggy_func fixed_func Userspace Kernelspace Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 19 / 23
  20. 20. Get It Upstreaming KGRAFT was submitted and reviewed upstream There are other groups working on competing technologies KPATCH, KSPLICE, criu-based aproach, . . . SUSE will work together with them Expectations: common standard kernel live patching Publishing Part of SLE12 kernel tree GIT repository upstream http://git.kernel.org/pub/scm/linux/kernel/git/jirislaby/ kgraft.git Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 20 / 23
  21. 21. Maintenance KGRAFT patch is an RPM package Once installed, always protected Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 21 / 23
  22. 22. Live Demo Kernel with a security vulnerability Exploit program kGraft patch Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 22 / 23
  23. 23. Conclusion SUSE provides Demanded live Linux kernel patching Dubbed KGRAFT Jiri Slaby (SUSE Labs) kGraft September 25th 2014, Paris 23 / 23

×