Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Seven problems of Linux containers

147 views

Published on

Kir Kolyshkin

Published in: Software
  • Login to see the comments

  • Be the first to like this

Seven problems of Linux containers

  1. 1. parallels.com || openvz.org || criu.org Seven Problems of Linux Containers Kir Kolyshkin <kir@openvz.org> 28 April 2013 LinuxFest Northwest
  2. 2. parallels.com || openvz.org || criu.org Seventy Seven Problems of Linux Containers Kir Kolyshkin <kir@openvz.org> 28 April 2013 LinuxFest Northwest (of which I am going to cover six)
  3. 3. parallels.com || openvz.org || criu.org Problem 1: Effective virtualization ● Virtualization is partitioning ● Historical way: $M mainframes ● Modern way: virtual machines ● Problem: performance overhead ● Partial solution: hardware support (Intel VT, AMD V)
  4. 4. parallels.com || openvz.org || criu.org Solution: isolation ● Run many isolated userspace instances on top of sone single (Linux) kernel ● All processes see each other – files, process information, network, shared memory, users, etc. ● Make them unsee it!
  5. 5. parallels.com || openvz.org || criu.org
  6. 6. parallels.com || openvz.org || criu.org One historical way to unsee chroot()
  7. 7. parallels.com || openvz.org || criu.org Namespaces ● Implemented in the Linux kernel – PID – net – IPC – UTS – mnt – user ● clone() with CLONE_NEW* flags
  8. 8. parallels.com || openvz.org || criu.org Problem 2: Shared resources ● All containers share the same set of resources (CPU, RAM, disk, various kernel things ...) ● Need fair distribution of goods so everyone gets their share ● Need DoS prevention ● Need prioritization – “All animals are equal, but some animals are more equal than others” -- George Orwell
  9. 9. parallels.com || openvz.org || criu.org
  10. 10. parallels.com || openvz.org || criu.org Solution: OpenVZ resource controls ● OpenVZ: – user beancounters ● controls 20 parameters – hierarchical CPU scheduler – disk quota per containers – I/O priorities per-container ● Dynamic control, can “resize” runtime
  11. 11. parallels.com || openvz.org || criu.org Solution: cgroups ● Cgroups is a mechanism to control resources per hierarchical groups of processes ● Cgroups is nothing without controllers: – blkio, cpu, cpuacct, cpuset, devices, freezer, memory, net_cls, net_prio ● Cgroups are orthogonal to namespaces ● Still a work in progress (kernel memory)
  12. 12. parallels.com || openvz.org || criu.org Problem 3: easy resources ● User Beancounters are complicated: – http://wiki.openvz.org/UBC_consistency_check – user has to set all these parameters – some of which are interdependent ● We created a collection of valid configs, ● ... wrote a whole book about UBC ● ... and a set of tools to help
  13. 13. parallels.com || openvz.org || criu.org
  14. 14. parallels.com || openvz.org || criu.org Solution: VSwap ● Only two primary parameters: RAM and swap – others still exist, but no longer required to set ● Swap is virtual, no actual I/O is performed ● Slow down to emulate real swap ● Only when actual global RAM shortage occurs, virtual swap goes into the real swap ● Currently only available in OpenVZ kernel
  15. 15. parallels.com || openvz.org || criu.org Problem 4: fast live migration ● We can migrate an OpenVZ container from one physical server to another without a shutdown ● We want to do it fast even for huge containers – huge disk: use shared storage – huge RAM: ???
  16. 16. parallels.com || openvz.org || criu.org Normal migration process ● (Assuming shared storage) ● 1 Freeze the container ● 2 Dump its complete state to a dump file ● 3 Copy dump file to destination server ● 4 Undump ● 5 Unfreeze ● Problem: huge dump file
  17. 17. parallels.com || openvz.org || criu.org Solution 1: network swap ● 1 Dump the minimal memory, lock the rest ● 2 Restore the minimal memory, mark the rest as swapped out ● 3 Set up network swap from the source ● 4 Unfreeze. Missing RAM will be “swapped in” ● 5 Migrate the rest of RAM and kill it on source
  18. 18. parallels.com || openvz.org || criu.org
  19. 19. parallels.com || openvz.org || criu.org Solution 1: network swap ● 1 Dump the minimal memory, lock the rest ● 2 Copy, undump what we have, mark the rest as swapped out ● 3 Set up network swap served from the source ● 4 Unfreeze. Missing RAM will be “swapped in” ● 5 Migrate the rest of RAM and kill it on source ● PROBLEM? Reliability, no way to rollback
  20. 20. parallels.com || openvz.org || criu.org Solution 2: Iterative RAM migration ● 1 Ask kernel to track modified pages ● 2 Copy all memory to destination system ● 3 Ask kernel for list of modified pages ● 4 Copy those pages ● 5 GOTO 3 until satisfied ● 6 Freeze and do migration as usual
  21. 21. parallels.com || openvz.org || criu.org Problem 5: upstreaming ● OpenVZ was developed separately ● Then we wanted to merge it upstream (i.e. to vanilla Linux kernel) ● Problem?
  22. 22. parallels.com || openvz.org || criu.org
  23. 23. parallels.com || openvz.org || criu.org Problem 5: upstreaming ● OpenVZ was developed separately ● Then we wanted to merge it upstream (i.e. to vanilla Linux kernel) ● Problem: ● upstream devs are not accepting our work
  24. 24. parallels.com || openvz.org || criu.org Solution 1: rewrite from scratch ● User Beancounters -> CGroups ● Did 2 rewrites for PID namespace until it finally got accepted ● Network namespace redone ● It works! ● about 1500 patches got landed to vanilla ● II Parallels made it to top10 contributors
  25. 25. parallels.com || openvz.org || criu.org Solution 2: CRIU ● We tried hard to merge checkpoint/restore ● Other people tried hard too, no luck ● Can't make it to the kernel, let's go userspace ● With minimal kernel intervention when required ● Kernel exports most of information already, so let's just add missing bits and pieces
  26. 26. parallels.com || openvz.org || criu.org CRIU ● Checkpoint / Restore (mostly) In Userspace Tools currently at version 0.4 ● Will do 1.0 release this year ● Kernel 3.8 has about 120 patches from us – 95% of needed features are there ● Memory snapshot recently made it to -mm tree
  27. 27. parallels.com || openvz.org || criu.org
  28. 28. parallels.com || openvz.org || criu.org Problem 6: common file system ● Container is just a directory on host, all CTs reside on the same FS ● File system journal is a bottleneck ● Lots of small-size files I/O on CT backup ● No sub-tree disk quota support in upstream ● No per-container snapshots ● Live migration: rsync -- changed inodes ● File system type and properties are fixed
  29. 29. parallels.com || openvz.org || criu.org Solution 1: LVM ● Only works only on top of block device ● Hard to manage (e.g. how to migrate huge volume?) ● No dynamic allocation ● Complicated management
  30. 30. parallels.com || openvz.org || criu.org Solution 2: loop device ● VFS operations leads to double page-caching – (already fixed in the recent kernels) ● No dynamic allocation, max space is used ● Limited feature set
  31. 31. parallels.com || openvz.org || criu.org Solution 3: ploop ● Basic idea: same as loop, just better ● Modular design: – various image formats (qcow2 in TODO) – various I/O backends ● More features: – live resize – instant live snapshots – write tracker to help in live migration
  32. 32. parallels.com || openvz.org || criu.org Any problems questions? ● kir@openvz.org ● Twitter: @kolyshkin

×