I’ve been playing around with LXC over the past few weeks and one of the things I tried out was limiting the memory that the container is allowed to use. I didn’t plan on explaining all the ins-and-outs of LXC here, but a short description is that LXC provides a virtualizedish environment that is more than a chroot gives you, but less than a full-blown virtual machine. If you want more details, please check out stgraber’s blog post about LXC in 12.04.

Kernel Configuration

The first thing you need to do in order to limit memory usage for LXC is make sure your kernel is properly configured, you need the following flag enabled:

CONFIG_CGROUP_MEM_RES_CTLR=y

If you plan on also limiting swap space usage, you’ll also need:

CONFIG_CGROUP_MEM_RES_CTLR_SWAP=y

These flags are enabled for me in my 12.10 kernel (3.5.0-22) and so presumably you’ll have them in 12.04.

Setting the Cap

First, I’m going to create my container. Following the instructions from stgraber’s blog post, and calling the container “memlimit”:

sudo lxc-create -t ubuntu -n memlimit

Once the container is built, we need to modify the config files. Look at /var/lib/lxc/memlimit/config. We need to add a few lines to that file. I’m going to limit memory to 512M and total usage or memory + swap to 1G. Note the second setting is for overall memory + swap, not just swap usage.

lxc.cgroup.memory.limit_in_bytes = 512M
lxc.cgroup.memory.memsw.limit_in_bytes = 1G

Now let’s start the container and get some debug info out of it to make sure these were set:

sudo lxc-start -n memlimit -l debug -o debug.out

The debug.out file will show up wherever you ran lxc-start from, so let’s see if it picked up our limits:

lxc-start 1359136997.617 DEBUG lxc_conf - cgroup 'memory.limit_in_bytes' set to '512M'
...
lxc-start 1359136997.617 DEBUG lxc_conf - cgroup 'memory.memsw.limit_in_bytes' set to '1G'

Looks good to me!

Note, I tried setting this once to 1.5G and it seems that the fields are only happy with whole numbers, it complained about 1.5G. That error message appeared in the same debug log I used above.

A list of more of the values you can set in here is available here.

Measuring Memory Usage

The view of /proc/meminfo inside the container and outside the container are the same. This means that you cannot rely on tools like top to show how much memory the container is using. In other words, when run inside the container top will correctly only show processes inside the container, it will also show overall memory usage for the entire system. To get valid information, instead we need to examine some files in /sys:

Current memory usage:
/sys/fs/cgroup/memory/lxc/memlimit/memory.usage_in_bytes

Current memory + swap usage:
/sys/fs/cgroup/memory/lxc/memlimit/memory.memsw.usage_in_bytes

Maximum memory usage:
/sys/fs/cgroup/memory/lxc/memlimit/memory.max_usage_in_bytes

Maximum memory + swap usage:
/sys/fs/cgroup/memory/lxc/memlimit/memory.memsw.max_usage_in_bytes

You can use expr to show it as kb or mb which is easier to read for me:

expr `cat memory.max_usage_in_bytes` / 1024
8188

What Happens When the Limit is Reached?

When the cap is reached, the container simply behaves as if the system ran out of memory. Calls to malloc will start failing (returning -1), leading to strange and bad things happening. Dialog boxes may not open, you may not be able to save files, and more than likely where people didn’t bother to check the returned value from malloc (aka, everyone), you’ll get segfaults. You can alleviate the pressure like normal systems do, by enabling swap inside the container, but once that limit is reached, you’ll have the same problem. In this case the host system’s kernel should start firing up the OOM killer and killing stuff inside the container.

Here is my extremely simple test program to drive up memory usage, install gcc in your container and you can try it too:

#include 
#include 

int main(void) {
    int i;
    for (i=0; i<65536; i++) {
        char *q = malloc(65536);
        printf ("Malloced: %ld\n", 65536*i);
    }
    sleep(9999999);
}

You can simply compiled with: gcc -o foo foo.c

I used the following simple shell construct to watch the memory usage. This needs to be run outside the container and I ran it from the /sys directory mentioned above:

while true; do echo -n "Mem Usage (mb): " \&\& expr `cat memory.usage_in_bytes` / 1024 / 1024; echo -n "Mem+swap Usage (mb): " \&\& expr `cat memory.memsw.usage_in_bytes` / 1024 / 1024; sleep 1; done

With the above shell script runnint, I fired up a bunch of copies of foo one bye one. Here’s the memory usage from that script:

Running a few copies:

Mem+swap Usage (mb): 825
Mem Usage (mb): 511
Mem+swap Usage (mb): 859
Mem Usage (mb): 511

A new copy of foo is starting:

Mem+swap Usage (mb): 899
Mem Usage (mb): 511
Mem+swap Usage (mb): 932
Mem Usage (mb): 511
Mem+swap Usage (mb): 1010
Mem Usage (mb): 511

The OOM killer just said “Nope!”

Mem+swap Usage (mb): 814
Mem Usage (mb): 511
Mem+swap Usage (mb): 825
Mem Usage (mb): 511

At the point where the OOM killer fired up, I see this in my container:
[1] Killed ./foo

So the limits are set, and they’re working.

Conclusion

If you are using LXC or considering using LXC, you can use a memory limit to protect the host from a container run amok. You could also use it to test your code in an artificially restricted environment. In either case, try the tools above and let me know how it works for you.