Posted on May 2, 2013 by s3hh

Recently I took some time to work on implementing container clones through the lxc API. lxc-clone previously existed as a shell script which could create snapshot clones of lvm and btrfs containers. There were several shortcomings to this:

1. clone was not exportable through the API (to be used in python, lua, go and c programs). Now it is, so a Go program can create a container clone in one function call.
2. expanding the set of supported clone types became unsavory
3. overlayfs was only supported as ‘ephemeral containers’, which could be made persistent through the use of pre-mount hooks. They were not first class citizens. Now they are.

The result is now in upstream git as well as in the packages at the ubuntu-lxc/daily ppa. Supported backing store types currently include dir (directory), lvm, btrfs, overlayfs, and zfs. Hopefully loop and qemu-nbd will be added soon. They each are somewhat different due to the nature of the backing store itself, so I’ll go over each. However in my opinion the coolest thing you can do with this is:

# create a stock directory backed container
sudo lxc-create -t ubuntu -n dir1
# create an overlayfs snapshot of it
sudo lxc-clone -s -B overlayfs dir1 s1

The -s argument asks for a snapshot (rather than copy) clone, and -B specifies the backing store type for the new container. When container s1 starts, it will mount a private writeable overlay (/var/lib/lxc/dir1/delta0) over a readonly mount of the original /var/lib/lxc/dir1/rootfs.

Now make some changes to start customizing s1. Checkpoint that state by cloning it:

sudo lxc-clone -s s1 s2

This will reference the same rootfs (/var/lib/lxc/dir1/rootfs) and rsync the overlayfs delta from s1 to s2. Now you can keep working on s1, keeping s2 as a checkpoint. Make more changes, and create your next snapshot

sudo lxc-clone -s s1 s3
…
sudo lxc-clone -s s1 s4
…

If at some point you realize you need to go back to an older snapshot, say s3, then you can

sudo lxc-clone -s s1 s1_bad # just to make sure
sudo lxc-destroy -n s1
sudo lxc-clone -s s3 s1

and pick up where you left off. Finally, if you’re happy and want to tar up what you have to ship it or copy to another machine, clone it back to a directory backed container:

sudo lxc-clone -B dir s1 dir_ship
sudo tar zcf /var/lib/lxc/dir_ship.tgz /var/lib/lxc/dir_ship

So far I’ve shown dir (directory) backing store and overlayfs. Specific to directory backed containers is that they cannot be snapshotted, except by converting them to overlayfs backed containers. Specific to overlayfs containers is that the original directory backed container must not be deleted, since the snapshot depends on it. (I’ll address this soon, marking the snapshotted container so that lxc-destroy will leave it alone, but that is not yet done)

To use btrfs containers, the entire lxc configuration path must be btrfs. However since the configuration path is flexible, that’s not as bad as it used to be. For instance, I mounted a btrfs at $HOME/lxcbase, then did

sudo lxc-create -t ubuntu -P $HOME/lxcbase -n b1

(The ‘-P’ argument chooses a custom ‘lxcpath’, or lxc configuration path, than the default /var/lib/lxc. You can also specify a global default other than /var/lib/lxc in /etc/lxc/lxc.conf.) lxc-create detects the btrfs and automatically makes the container a new subvolume, which can then be snapshotted

sudo lxc-clone -s b1 b2

For zfs, a zfsroot can be specified in /etc/lxc/lxc.conf. I created a zfs pool called ‘lxc’ (which is actually the default for the lxc tools, so I did not list it in /etc/lxc/lxc.conf), then did

sudo lxc-create -B zfs -t ubuntu -n z1
or
sudo lxc-clone -B zfs dir1 z1

This created ‘lxc/z1′ as a new zfs fs and mounted it under /var/lib/lxc/z1/rootfs. Next I could

sudo lxc-clone -s z1 z2

Now lxc-destroy needs some smarts still built-in to make zfs backed containers easier to destroy. That is because when lxc-clone creates z2 from z1, it must first create a snapshot ‘lxc/z1@z2′, then clone that to ‘lxc/z2′. So before you can destroy z1, you currently must

sudo lxc-destroy -n z2
sudo zfs destroy lxc/z1@x2

Finally, you can also use LVM. LVM snapshot container clones have been supported longer than any others (with btrfs being second). I like the fact that you can use any filesystem inside the LV. However, the two major shortcomings are that you cannot snapshot a snapshot, and that you must (depending at least on the filesystem type) choose a filesystem size in advance.

To clone LVM conatiners, you either need a vg called ‘lxc’, or you can specify a default vg in /etc/lxc/lxc.conf. You can create the initial lvm container with

sudo lxc-create -t ubuntu -n lvm1 –fssize 2G –fstype xfs
or
sudo lxc-clone -B lvm dir1 lvm1

Then snapshot it using

sudo lxc-clone -s lvm1 lvm2

Note that unlike overlayfs, snapshots in zfs, btrfs, and lvm are safe from having the base container destroyed. In btrfs, that is because the btrfs snapshot is metadata based, so destroying the base container simply does not delete any of the data in use by the snapshot container. LVM and zfs both will note that there are active snapshots of the base rootfs and prevent the base container from being destroyed.

This entry was posted in Uncategorized and tagged LXC, server. Bookmark the permalink.