Following is a description of the files and directories which are installed and used by LXC.

When USE_LXC_BRIDGE is set to true in /etc/default/lxc (as it is by default), a bridge called lxcbr0 is created at startup. This bridge is given the private address 10.0.3.1, and containers using this bridge will have a 10.0.3.0/24 address. A dnsmasq instance is run listening on that bridge, so if another dnsmasq has bound all interfaces before the lxc-net upstart job runs, lxc-net will fail to start and lxcbr0 will not exist.

If you have another bridge - libvirt's default virbr0, or a br0 bridge for your default NIC - you can use that bridge in place of lxcbr0 for your containers.

LXC stores container information and (with the default backing store) root filesystems under /var/lib/lxc. Container creation templates also tend to store cached distribution information under /var/cache/lxc.

If you wish to use another filesystem than /var, you can mount a filesystem which has more space into those locations. If you have a disk dedicated for this, you can simply mount it at /var/lib/lxc. If you'd like to use another location, like /srv, you can bind mount it or use a symbolic link. For instance, if /srv is a large mounted filesystem, create and symlink two directories:

sudo mkdir /srv/lxclib /srv/lxccache
sudo rm -rf /var/lib/lxc /var/cache/lxc
sudo ln -s /srv/lxclib /var/lib/lxc
sudo ln -s /srv/lxccache /var/cache/lxc

or, using bind mounts:

sudo mkdir /srv/lxclib /srv/lxccache
sudo sed -i '$a \
/srv/lxclib /var/lib/lxc    none defaults,bind 0 0 \
/srv/lxccache /var/cache/lxc none defaults,bind 0 0' /etc/fstab
sudo mount -a

It is possible to use LVM partitions as the backing stores for containers. Advantages of this include flexibility in storage management and fast container cloning. The tools default to using a VG (volume group) named lxc, but another VG can be used through command line options. When a LV is used as a container backing store, the container's configuration file is still /var/lib/lxc/CN/config, but the root fs entry in that file (lxc.rootfs) will point to the lV block device name, i.e. /dev/lxc/CN.

Containers with directory tree and LVM backing stores can co-exist.

If your host has a btrfs /var, the LXC administration tools will detect this and automatically exploit it by cloning containers using btrfs snapshots.

LXC ships with an Apparmor profile intended to protect the host from accidental misuses of privilege inside the container. For instance, the container will not be able to write to /proc/sysrq-trigger or to most /sys files.

The usr.bin.lxc-start profile is entered by running lxc-start. This profile mainly prevents lxc-start from mounting new filesystems outside of the container's root filesystem. Before executing the container's init, LXC requests a switch to the container's profile. By default, this profile is the lxc-container-default policy which is defined in /etc/apparmor.d/lxc/lxc-default. This profile prevents the container from accessing many dangerous paths, and from mounting most filesystems.

If you find that lxc-start is failing due to a legitimate access which is being denied by its Apparmor policy, you can disable the lxc-start profile by doing:

sudo apparmor_parser -R /etc/apparmor.d/usr.bin.lxc-start
sudo ln -s /etc/apparmor.d/usr.bin.lxc-start /etc/apparmor.d/disabled/

This will make lxc-start run unconfined, but continue to confine the container itself. If you also wish to disable confinement of the container, then in addition to disabling the usr.bin.lxc-start profile, you must add:

lxc.aa_profile = unconfined

to the container's configuration file. If you wish to run a container in a custom profile, you can create a new profile under /etc/apparmor.d/lxc/. Its name must start with lxc- in order for lxc-start to be allowed to transition to that profile. After creating the policy, load it using:

sudo apparmor_parser -r /etc/apparmor.d/lxc-containers

The profile will automatically be loaded after a reboot, because it is sourced by the file /etc/apparmor.d/lxc-containers. Finally, to make container CN use this new lxc-CN-profile, add the following line to its configuration file:

lxc.aa_profile = lxc-CN-profile

lxc-execute does not enter an Apparmor profile, but the container it spawns will be confined.

Control groups (cgroups) are a kernel feature providing hierarchical task grouping and per-cgroup resource accounting and limits. They are used in containers to limit block and character device access and to freeze (suspend) containers. They can be further used to limit memory use and block i/o, guarantee minimum cpu shares, and to lock containers to specific cpus. By default, LXC depends on the cgroup-lite package to be installed, which provides the proper cgroup initialization at boot. The cgroup-lite package mounts each cgroup subsystem separately under /sys/fs/cgroup/SS, where SS is the subsystem name. For instance the freezer subsystem is mounted under /sys/fs/cgroup/freezer. LXC cgroup are kept under /sys/fs/cgroup/SS/INIT/lxc, where INIT is the init task's cgroup. This is / by default, so in the end the freezer cgroup for container CN would be /sys/fs/cgroup/freezer/lxc/CN.

The container administration tools must be run with root user privilege. A utility called lxc-setup was written with the intention of providing the tools with the needed file capabilities to allow non-root users to run the tools with sufficient privilege. However, as root in a container cannot yet be reliably contained, this is not worthwhile. It is therefore recommended to not use lxc-setup, and to provide the LXC administrators the needed sudo privilege.

The user namespace, which is expected to be available in the next Long Term Support (LTS) release, will allow containment of the container root user, as well as reduce the amount of privilege required for creating and administering containers.

As listed above, the lxc package includes two upstart jobs. The first, lxc-net, is always started when the other, lxc, is about to begin, and stops when it stops. If the USE_LXC_BRIDGE variable is set to false in /etc/defaults/lxc, then it will immediately exit. If it is true, and an error occurs bringing up the LXC bridge, then the lxc job will not start. lxc-net will bring down the LXC bridge when stopped, unless a container is running which is using that bridge.

The lxc job starts on runlevel 2-5. If the LXC_AUTO variable is set to true, then it will look under /etc/lxc for containers which should be started automatically. When the lxc job is stopped, either manually or by entering runlevel 0, 1, or 6, it will stop those containers.

To register a container to start automatically, create a symbolic link /etc/default/lxc/name.conf pointing to the container's config file. For instance, the configuration file for a container CN is /var/lib/lxc/CN/config. To make that container auto-start, use the command:

sudo ln -s /var/lib/lxc/CN/config /etc/lxc/auto/CN.conf

The easiest way to create containers is using lxc-create. This script uses distribution-specific templates under /usr/lib/lxc/templates/ to set up container-friendly chroots under /var/lib/lxc/CN/rootfs, and initialize the configuration in /var/lib/lxc/CN/fstab and /var/lib/lxc/CN/config, where CN is the container name

The simplest container creation command would look like:

sudo lxc-create -t ubuntu -n CN

This tells lxc-create to use the ubuntu template (-t ubuntu) and to call the container CN (-n CN). Since no configuration file was specified (which would have been done with `-f file'), it will use the default configuration file under /etc/lxc/lxc.conf. This gives the container a single veth network interface attached to the lxcbr0 bridge.

The container creation templates can also accept arguments. These can be listed after --. For instance

sudo lxc-create -t ubuntu -n oneiric1 -- -r oneiric

passes the arguments '-r oneiric1' to the ubuntu template.

For rapid provisioning, you may wish to customize a canonical container according to your needs and then make multiple copies of it. This can be done with the lxc-clone program. Given an existing container called C1, a new container called C2 can be created using

sudo lxc-clone -o C1 -n C2

If /var/lib/lxc is a btrfs filesystem, then lxc-clone will create C2's filesystem as a snapshot of C1's. If the container's root filesystem is lvm backed, then you can specify the -s option to create the new rootfs as a lvm snapshot of the original as follows:

sudo lxc-clone -s -o C1 -n C2

Both lvm and btrfs snapshots will provide fast cloning with very small initial disk usage.

To start a container, use lxc-start -n CN. By default lxc-start will execute /sbin/init in the container. You can provide a different program to execute, plus arguments, as further arguments to lxc-start:

sudo lxc-start -n container /sbin/init loglevel=debug

If you do not specify the -d (daemon) option, then you will see a console (on the container's /dev/console, see Consoles for more information) on the terminal. If you specify the -d option, you will not see that console, and lxc-start will immediately exit success - even if a later part of container startup has failed. You can use lxc-wait or lxc-monitor (see Monitoring container status ) to check on the success or failure of the container startup.

To obtain LXC debugging information, use -o filename -l debuglevel, for instance:

sudo lxc-start -o lxc.debug -l DEBUG -n container

Finally, you can specify configuration parameters inline using -s. However, it is generally recommended to place them in the container's configuration file instead. Likewise, an entirely alternate config file can be specified with the -f option, but this is not generally recommended.

While lxc-start runs the container's /sbin/init, lxc-execute uses a minimal init program called lxc-init, which attempts to mount /proc, /dev/mqueue, and /dev/shm, executes the programs specified on the command line, and waits for those to finish executing. lxc-start is intended to be used for system containers, while lxc-execute is intended for application containers (see this article for more).

You can stop a container several ways. You can use shutdown, poweroff and reboot while logged into the container. To cleanly shut down a container externally (i.e. from the host), you can issue the sudo lxc-shutdown -n CN command. This takes an optional timeout value. If not specified, the command issues a SIGPWR signal to the container and immediately returns. If the option is used, as in sudo lxc-shutdown -n CN -t 10, then the command will wait the specified number of seconds for the container to cleanly shut down. Then, if the container is still running, it will kill it (and any running applications). You can also immediately kill the container (without any chance for applications to cleanly shut down) using sudo lxc-stop -n CN. Finally, lxc-kill can be used more generally to send any signal number to the container's init.

While the container is shutting down, you can expect to see some (harmless) error messages, as follows:

$ sudo poweroff
[sudo] password for ubuntu: =

$ =

Broadcast message from ubuntu@cn1
        (/dev/lxc/console) at 18:17 ...

The system is going down for power off NOW!
 * Asking all remaining processes to terminate...
   ...done.
 * All processes ended within 1 seconds....
   ...done.
 * Deconfiguring network interfaces...
   ...done.
 * Deactivating swap...
   ...fail!
umount: /run/lock: not mounted
umount: /dev/shm: not mounted
mount: / is busy
 * Will now halt

A container can be frozen with sudo lxc-freeze -n CN. This will block all its processes until the container is later unfrozen using sudo lxc-unfreeze -n CN.

Two commands are available to monitor container state changes. lxc-monitor monitors one or more containers for any state changes. It takes a container name as usual with the -n option, but in this case the container name can be a posix regular expression to allow monitoring desirable sets of containers. lxc-monitor continues running as it prints container changes. lxc-wait waits for a specific state change and then exits. For instance,

sudo lxc-monitor -n cont[0-5]*

would print all state changes to any containers matching the listed regular expression, whereas

sudo lxc-wait -n cont1 -s 'STOPPED|FROZEN'

will wait until container cont1 enters state STOPPED or state FROZEN and then exit.

Containers have a configurable number of consoles. One always exists on the container's /dev/console. This is shown on the terminal from which you ran lxc-start, unless the -d option is specified. The output on /dev/console can be redirected to a file using the -c console-file option to lxc-start. The number of extra consoles is specified by the lxc.tty variable, and is usually set to 4. Those consoles are shown on /dev/ttyN (for 1 <= N <= 4). To log into console 3 from the host, use

sudo lxc-console -n container -t 3

or if the -t N option is not specified, an unused console will be automatically chosen. To exit the console, use the escape sequence Ctrl-a q. Note that the escape sequence does not work in the console resulting from lxc-start without the -d option.

Each container console is actually a Unix98 pty in the host's (not the guest's) pty mount, bind-mounted over the guest's /dev/ttyN and /dev/console. Therefore, if the guest unmounts those or otherwise tries to access the actual character device 4:N, it will not be serving getty to the LXC consoles. (With the default settings, the container will not be able to access that character device and getty will therefore fail.) This can easily happen when a boot script blindly mounts a new /dev.

Several commands are available to gather information on existing containers. lxc-ls will report all existing containers in its first line of output, and all running containers in the second line. lxc-list provides the same information in a more verbose format, listing running containers first and stopped containers next. lxc-ps will provide lists of processes in containers. To provide ps arguments to lxc-ps, prepend them with --. For instance, for listing of all processes in container plain,

sudo lxc-ps -n plain -- -ef

lxc-info provides the state of a container and the pid of its init process. lxc-cgroup can be used to query or set the values of a container's control group limits and information. This can be more convenient than interacting with the cgroup filesystem. For instance, to query the list of devices which a running container is allowed to access, you could use

sudo lxc-cgroup -n CN devices.list

or to add mknod, read, and write access to /dev/sda,

sudo lxc-cgroup -n CN devices.allow "b 8:* rwm"

and, to limit it to 300M of RAM,

lxc-cgroup -n CN memory.limit_in_bytes 300000000

lxc-netstat executes netstat in the running container, giving you a glimpse of its network state.

lxc-backup will create backups of the root filesystems of all existing containers (except lvm-based ones), using rsync to back the contents up under /var/lib/lxc/CN/rootfs.backup.1. These backups can be restored using lxc-restore. However, lxc-backup and lxc-restore are fragile with respect to customizations and therefore their use is not recommended.

Use lxc-destroy to destroy an existing container.

sudo lxc-destroy -n CN

If the container is running, lxc-destroy will exit with a message informing you that you can force stopping and destroying the container with

sudo lxc-destroy -n CN -f

One of the Linux kernel features used by LXC to create containers is private namespaces. Namespaces allow a set of tasks to have private mappings of names to resources for things like pathnames and process IDs. (See Resources for a link to more information). Unlike control groups and other mount features which are also used to create containers, namespaces cannot be manipulated using a filesystem interface. Therefore, LXC ships with the lxc-unshare program, which is mainly for testing. It provides the ability to create new tasks in private namespaces. For instance,

sudo lxc-unshare -s 'MOUNT|PID' /bin/bash

creates a bash shell with private pid and mount namespaces. In this shell, you can do

root@ubuntu:~# mount -t proc proc /proc
root@ubuntu:~# ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  6 10:20 pts/9    00:00:00 /bin/bash
root       110     1  0 10:20 pts/9    00:00:00 ps -ef

so that ps shows only the tasks in your new namespace.

Ephemeral containers are one-time containers. Given an existing container CN, you can run a command in an ephemeral container created based on CN, with the host's jdoe user bound into the container, using:

lxc-start-ephemeral -b jdoe -o CN -- /home/jdoe/run_my_job

When the job is finished, the container will be discarded.

Following is a table of all container commands:

The container setup is controlled by the LXC configuration options. Options can be specified at several points:

Container networking in LXC is very flexible. It is triggered by the lxc.network.type configuration file entries. If no such entries exist, then the container will share the host's networking stack. Services and connections started in the container will be using the host's IP address. If at least one lxc.network.type entry is present, then the container will have a private (layer 2) network stack. It will have its own network interfaces and firewall rules. There are several options for lxc.network.type:

Two other options are to use vlan or macvlan, however their use is more complicated and is not described here. A few other networking options exist:

Cgroup options can be specified using lxc.cgroup entries. lxc.cgroup.subsystem.item = value instructs LXC to set cgroup subsystem's item to value. It is perhaps simpler to realize that this will simply write value to the file item for the container's control group for subsystem subsystem. For instance, to set the memory limit to 320M, you could add

lxc.cgroup.memory.limit_in_bytes = 320000000

which will cause 320000000 to be written to the file /sys/fs/cgroup/memory/lxc/CN/limit_in_bytes.

An important part of container setup is the mounting of various filesystems into place. The following is an example configuration file excerpt demonstrating the commonly used configuration options:

lxc.rootfs = /var/lib/lxc/CN/rootfs
lxc.mount.entry=proc /var/lib/lxc/CN/rootfs/proc proc nodev,noexec,nosuid 0 0
lxc.mount = /var/lib/lxc/CN/fstab

The first line says that the container's root filesystem is already mounted at /var/lib/lxc/CN/rootfs. If the filesystem is a block device (such as an LVM logical volume), then the path to the block device must be given instead.

Each lxc.mount.entry line should contain an item to mount in valid fstab format. The target directory should be prefixed by /var/lib/lxc/CN/rootfs, even if lxc.rootfs points to a block device.

Finally, lxc.mount points to a file, in fstab format, containing further items to mount. Note that all of these entries will be mounted by the host before the container init is started. In this way it is possible to bind mount various directories from the host into the container.

Creating Containers showed how to create LXC containers. If you've created a valid LXC container in this way, you can manage it with libvirt. Fetch a sample xml file from

wget http://people.canonical.com/~serge/o1.xml

Edit this file to replace the container name and root filesystem locations. Then you can define the container with:

virsh -c lxc:/// define o1.xml

If you prefer to create a pristine new container just for LXC, you can download an ubuntu cloud image, extract it, and point a libvirt LXC xml file to it. For instance, find the url for a root tarball for the latest daily Ubuntu 12.04 LTS cloud image using

url1=`ubuntu-cloudimg-query precise daily $arch --format "%{url}\n"`
url=`echo $url1 | sed -e 's/.tar.gz/-root\0/'`
wget $url
filename=`basename $url`

Extract the downloaded tarball, for instance

mkdir $HOME/c1
cd $HOME/c1
sudo tar zxf $filename

Download the xml template

wget http://people.canonical.com/~serge/o1.xml

In the xml template, replace the name o1 with c1 and the source directory /var/lib/lxc/o1/rootfs with $HOME/c1. Then define the container using

virsh define o1.xml

As we've seen, you can create a libvirt-lxc container using

virsh -c lxc:/// define container.xml

To start a container called container, use

virsh -c lxc:/// start container

To stop a running container, use

virsh -c lxc:/// destroy container

Note that whereas the lxc-destroy command deletes the container, the virsh destroy command stops a running container. To delete the container definition, use

virsh -c lxc:/// undefine container

To get a console to a running container, use

virsh -c lxc:/// console container

Exit the console by simultaneously pressing control and ].

It is a core container feature that containers share a kernel with the host. Therefore, if the kernel contains any exploitable system calls, the container can exploit these as well. Once the container controls the kernel it can fully control any resource known to the host.