This post is not about the differences between Docker and Kubernetes; it is about the differences needed for containers. And why you should not expect any container from dockerhub.io to run under Kubernetes.
The main difference discussed here is that
- Docker and Docker Swarm run containers from a ‘root’ environment by default
- Kubernetes containers run under the user that started Kubernetes or as defined by the RunasUser and RunasGroup parameters in the deployment yaml file
For Docker this allows complex applications with components running under multiple users to be run from a container, as the container sartup script is run as root the startup script can ‘su’ down to each of the neeed users to start those components. As long as any UID needed for bind mounts is documented that works well.
The same startup logic cannot be used for Kubernetes containers. The container startup script must be owned and executable by the RunasUser/RunasGroup pair and as a general rule all application components would be started by that one user. This helps in keeping containers designed for one purpose as simple and as small as possible (microservice) but makes running an entire application with more than one component in a container difficult.
Many container images designed for docker require to be run in a root environment. Should you try to run an image that has the startup script secured to root or user X with a RunasUser of Y under kubernetes the pod will just go into CrashLoopBackOff status because of permission denied on the script.
It is certanly possible to design containers to run under both environments. What makes it difficult is that there is no environment variable set for a container by docker of kubernetes to let the container know where it is running; not a major issue as you can simply provide your own.
From personal experience with my own small containers designed for docker that I wanted to move only used ‘su’ to move from root to the user I wanted the app to run under, the conversion to support both is simply to pass a custom environment variable to the container if running under kubernetes and if the variable exists assume the RunasUser/RunasGroup were set correctly and just start the app; if not present assume docker and ‘su’ down to the expected user to start the app. Actually as I was writing this post I just thought an easier way would be to simply check if the script is running as root and determine the environment that way, which I am going to use instead :-).
It is certainly possible to start apps under multiple users under Kubernetes; you would simply add ‘sudoers’ entries for the RunasUser to allow it to run the startup commands under the correct user. But if the container is designed to run under either engine would have sudoer entries in the container not required by docker and containers are audited these days :-).
For me personally I only want to build one container per app and have it run under either.
If you are relying on external container images just remember images on dockerhub.io are for Docker, don’t expect them to even start on Kubernetes. Images built explicitly for Kubernetes are not intended to be run as root so should not be run under Docker.
If you are developing your own containers and use both engines, design the container to be able to run under both. All that is needed is the awareness of the fact Docker will run startup scripts as root and Kubernetes prefers not to.
This post is primarily because containers I was moving from docker to minikube were going into CrashLoopBackOff and after sorting that out thought it may be of interest to others as it doesn’t seem to be highlighted anywhere.
