Live patching -- the process of updating software while it is running -- has become a popular topic of late, with several companies offering services providing live patching for Linux culminating with an implementation being contributed to Linux. While Xen supports live migration of VMs which mitigates the need for live patching, there are still some reasons why live patching is useful. For example, to minimize downtime, when running applications with low latency requirements, and when using PCI passthrough. This talk reviews the state of live patching for Xen. A live patching implementation has recently been contributed to Xen so we will look at the design and implementation in detail, noting how and why it differs from live patching for Linux. We will look at some of the ways that live patches can be created and the pitfalls to watch out for when doing so. There will also be a status report of ongoing and future work for live patching for Xen. Finally, the talk will show a demo of live patching in action by building a live patch for a known security issue and applying it to a vulnerable system. Xen LivePatch is a new tech preview feature in Xen 4.7 for live patching the hypervisor. It is the result of collaboration between several members in the community guided by the existing live patching efforts for Linux. We will talk about this and how Xen's requirements differ from Linux. The core idea of Xen LivePatch is to insert a "trampoline" in each buggy function with a jump to a replacement function. This is simple in principle, but there are a number of complexities which make it interesting to look at. Live patches can be built by hand, by compiling the replacement functions in a separate source file. This is a manual process and can also be unreliable due to the nature of compiler optimizations. So we repurposed kpatch's live patch creation tool for Xen. This computes a binary difference of the output binaries before and after a source patch is applied. This simplifies the process of creating live patches and overcomes issues with compiler optimizations but is still not without difficulties. We look at the process of creating live patches and why it is hard. Also see https://fosdem.org/2017/schedule/event/iaas_livepatxen/

1. LIVE PATCHING THE XEN PROJECT HYPERVISOR
Ross Lagerwall (Citrix)
FOSDEM 2017

2. Introduction
Xen supports live migration — so why live patch?
VM is using a pass-through device.
Live migration downtime is too long.
To avoid host downtime.
Not enough spare resources to move all VMs off the host.
There are several good reasons to live patch the hypervisor!

3. Patching basics
Inline replacement of hypervisor code would be possible but
not necessarily practical.
We replace old functions with new functions instead.
Xen Live Patch has a function-level granularity.

4. Payloads
A payload is a set of replacement functions plus metadata.
It is packaged in a relocatable object le, like a kernel module.
Since the module is relocatable, all the symbols need to be
resolved.
Xen has a simple linker to do this while loading the payload.
Plus a whole lot more: perform relocations, apply alternative
instructions, and parse the replacement table, hook
functions, and bug and exception frames.

5. Stacking payloads
Payloads are speci c to a particular build of the hypervisor.
They depend on the exact hypervisor internal ABI — not
stable.
The hypervisor and each live patch contains a build-id.
Each payload is designed to be applied on top of a speci c
build-id to prevent applying a patch on the wrong base.
This model allows loading a stack of patches.

6. Applying payloads
A loaded payload needs to be applied to take effect.
To avoid modifying code while it is being executed, the system
is quiesced.
We check in the return-to-guest path if any live patches need
to be applied.
At this point, there is a xed, small subset of functions that
could be on the stack.

8. Hypervisor interface
The sysctl hypercall was extended with four new sub-
operations for:
Uploading a payload
Listing payloads
Querying a payload state
Sending a command to apply, revert or unload a payload
The new sub-operations are controllable through XSM.
The xen-livepatch tool is used to access this functionality.

9. Compared with Linux live patching
Latency for VMs is not usually that critical so we don't use
kGraft's model.
Xen's model is largely similar to kPatch but there is no need
to perform stack checking.
Xen does not have ftrace and is not compiled with -pg to
get mcount() calls so the start of the function is
overwritten.
Can patch any function which is at least 5 bytes long.
Overhead is a single unconditional jump.

10. Building live patches
How are these created?
Build them by hand? — NO!
Enter livepatch-build-tools!
livepatch-build-tools is based on kpatch-build
http://xenbits.xen.org/gitweb/?p=livepatch-build-tools.git

11. Building live patches: Inputs
$ livepatch­build ­s xen ­c orig.config
­­depends 55776af8c7377e7191d733797543b87a59631c50
­p xsa182.patch ­o outdir
Takes as input:
The exact source tree from the running Xen.
The .con g from the original build of Xen.
A build-id onto which the livepatch will be applied.
A source patch.

12. Building live patches: Process
livepatch-build does:
1. Build Xen
2. Apply the source patch
3. Build Xen with "-ffunction-sections -fdata-sections"
4. Revert the source patch
5. Build Xen again with "-ffunction-sections -fdata-sections"
6. Create a livepatch from the changed object les.

13. For each pair of changed objects, ‘original’ and ‘patched’, run
:
Building live patches: Diff
create-diff-tool
Load objects and check that the headers match.
Adjust the ELFs to make them easier to process.
Correlate sections, symbols, and static locals.
Compare and mark as SAME, CHANGED or NEW.
For each CHANGED function or NEW global, include it and
its references recursively.

14. Building live patches: Diff
Handle special sections (bug frames, altinstructions,
exception tables).
For each CHANGED function, create an entry in a special
livepatch section (.livepatch.funcs).
Write out the new object le.
Link each object le together into a relocatable module.

15. Handling live patches with data
New data and read-only data is handled correctly.
Changing initialized data or existing data structures is hard
so such changes are prevented.
Hook functions allow code to be executed at various stages
during the patch apply (or revert) process.
Allows data to be transformed during patch apply, even if
the data is dynamically allocated
Allows once-off initializations.
Use shadow variables to attach new members to existing
data structures.

16. Compared with Linux live patching
Linux live patches are contained in modules.
Xen does not support loadable modules which avoids many
issues.
kGraft does not use a tool to automatically create live
patches.
kpatch is moving towards not using kpatch-build.

17. Future Work
Support for handling NMIs and MCEs while
applying/reverting payloads.
Support for signing payloads similarly to how kernel
modules can be signed.
This would make it more dif cult to load a malicious
module.
Add OSStest support.
Remove the experimental tag.