overlay2 for Docker within an unprivileged LXC container

For my Jenk­ins in­stal­la­tion I use a Docker agent in­side an LXC con­tainer. I want this con­tainer to be un­priv­i­leged, so that the host is some­what pro­tected from mis­con­fig­u­ra­tion (not de­lib­er­ate at­tacks). The de­fault setup works fine, but after a bit of ex­per­i­ment­ing, I no­ticed that I was soon run­ning out of disk-space. The rea­son for that turned out that Docker had fallen back to using the vfs stor­age back­end in­stead of overlay2, which ba­si­cally cre­ates a copy for every layer and every run­ning con­tainer.

# docker info | grep Storage
 Storage Driver: vfs

Fur­ther in­ves­ti­ga­tion showed, that this was due to the fact that the con­tainer was un­priv­i­leged. Short ex­per­i­ments with mak­ing the con­tainer priv­i­leged also yielded is­sues with cgroup man­age­ment of the outer docker con­tainer on the host. So what was the rea­son for the is­sues? It seems that the ID map­ping / shift­ing of the user IDs pre­vented the over­lay2 dri­ver from work­ing.

There­fore I de­cided to try to mount a host di­rec­tory as a “de­vice” into the con­tainer’s /var/lib/docker. But using the shift=true op­tion, this again fails, since this way the un­der­ly­ing filesys­tem is shiftfs and not plain ext4 (see sup­ported filesys­tems for var­i­ous stor­age dri­vers). So a so­lu­tion with­out “shift” is re­quired.

Shift­ing UIDs is done by a fixed off­set for a con­tainer, in my case it’s 1,000,000. You need to fig­ure this out for your sys­tem, but likely it’s the same. So by cre­at­ing the ex­ter­nal stor­age di­rec­tory with this as owner and then mount­ing it in­side the con­tainer with­out shift­ing, things start to get work­ing.

export CONTAINER_NAME=mycontainer
export DOCKER_STORAGE_DIRECTORY=/mnt/pool/mycontainer/var-lib-docker

mkdir -p "$DOCKER_STORAGE_DIRECTORY"
chown 1000000:1000000 "$DOCKER_STORAGE_DIRECTORY"

lxc config device add "$CONTAINER_NAME" var-lib-docker disk source="$DOCKER_STORAGE_DIRECTORY" path=/var/lib/docker

# important, security.nesting is required for nested containers to work!
lxc config set "$CONTAINER_NAME" security.nesting=true

After this docker info | grep Storage fi­nally showed what I wanted:

# docker info | grep Storage
 Storage Driver: overlay2

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.