The curious case of User Management in Docker on Openshift

Naturally some of our applications are equipped being packaged as docker containers. In a recent deployment, together with a client, we encountered something strange that happend while deploying a container on Openshift.

The dockerfile in question looks a little bit like the following:

FROM python:3.8-slim

WORKDIR /app

RUN pip install poetry
COPY pyproject.toml poetry.lock /app/

ENV VIRTUAL_ENV=.venv
RUN python -m venv $VIRTUAL_ENV

RUN poetry config virtualenvs.create false && \
    poetry install


ARG UID=1000
RUN groupadd -g $UID appuser
RUN adduser --disabled-password --uid $UID --gid $UID --gecos "?" appuser
RUN usermod -a -G appuser appuser

COPY --chown appuser:appuser backend /app/backend

USER appuser
CMD ["/bin/bash","-c", "sh start.sh"]


EXPOSE 8080

In the beginning we pull the python base image, add poetry as our package manager and install all packages in a virtual environment folder called .venv. Then we copy our backend into the container and lastly define a start command. In addition, we create a user called appuser, which has permissions in the backend folder to start the entry script.

Now building and running the container locally works just fine. Also on our internal testing deployment system everything runs smoothly. But something strange happend when we deployed the application onto our clients Openshift instance. We immediately got Permission Denied errors:

2021-09-24 12:00:48 backend-stage-5-28zsz __main__[1] INFO Gino log level is <bound method Logger.getEffectiveLevel of <Logger gino.engine._SAEngine (INFO)>>
Traceback (most recent call last):
  File "/app/.venv/lib/python3.8/site-packages/cacheman/registers.py", line 125, in pickle_saver
    pickle_pre_saver(cache_dir, cache_name, contents, tmp_exts)
  File "/app/.venv/lib/python3.8/site-packages/cacheman/registers.py", line 138, in pickle_pre_saver
    with open('.'.join([cache_path] + extensions), 'wb') as pkl_file:
PermissionError: [Errno 13] Permission denied: './data/cache/f52cfaa024ff85a57973749bc9fbfb40.pkl.tmp.CQONHYOPPKJNURFDRE'
Traceback (most recent call last):
  File "/app/.venv/lib/python3.8/site-packages/cacheman/registers.py", line 125, in pickle_saver
    pickle_pre_saver(cache_dir, cache_name, contents, tmp_exts)
  File "/app/.venv/lib/python3.8/site-packages/cacheman/registers.py", line 138, in pickle_pre_saver
    with open('.'.join([cache_path] + extensions), 'wb') as pkl_file:
PermissionError: [Errno 13] Permission denied: './data/cache/f52cfaa024ff85a57973749bc9fbfb40.pkl.tmp.CQONHYOPPKJNURFDRE'

What is happening here? In the application we have a library that generates .pkl files in order to save information that is used as a cache throught the application. But somehow the application is not allowed to write onto the cache directory. But why did it work locally and in our testing environment ?

A first problem here is that we only give the appuser permissions to the backend directory and not the .venv environment directory. Since the code lies in the virtual environment we must give the user permission to also write in there. But it still spits out Permission Errors.

We sat down and searched the OpenShift documention and found the following:

4. Why doesn't my Docker image run on OpenShift? Security! Origin runs with the following security policy by default:

Containers run as a non-root unique user that is separate from other system users They cannot access host resources, run privileged, or become root. They are given CPU and memory limits defined by the system administrator Any persistent storage they access will be under a unique SELinux label, which prevents others from seeing their content. These settings are per project, so containers in different projects cannot see each other by default Regular users can run Docker, source, and custom builds By default, Docker builds can (and often do) run as root. You can control who can create Docker builds through the builds/docker and builds/custom policy resource. Regular users and project admins cannot change their security quotas. Many Docker containers expect to run as root (and therefore edit all the contents of the filesystem). The Image Author's guide gives recommendations on making your image more secure by default: Don't run as root Make directories you want to write to group-writable and owned by group id 0 Set the net-bind capability on your executables if they need to bind to ports <1024 Otherwise, you can see the security documentation for descriptions on how to relax these > restrictions.

This gives us a hint what is going on here. By default, OpenShift Enterprise runs containers using an arbitrarily assigned user ID. This provides additional security against processes escaping the container due to a container engine vulnerability and thereby achieving escalated permissions on the host node.

So a fix is to add the user to the root group:

RUN chgrp -R 0 /some/directory \
  && chmod -R g+rwX /some/directory

This should be fine since the root group does not have any special permissions, but can still read and write files onto some/directory. Another option is to pass in the current user into the container as an environment variable.

docker build --build-arg UID=$(id -u) .

Having a non-privileged user helps you to run processes that should not have root permission. That being said User Management in docker is always precarious. The docker daemon always runs as the root user and binds to the Unix socket therefore different behaviours between different systems arise.