Docker Isolation, and non-Isolation

Docker is not KVM, there are major security trade-offs with a container, The key ones are shown below.

Processes are not isolated

The processes that are run by containers run for all intents and purposes as processes on the Docker host machine, this is an issue as an admin on the host machine may inadvertently cause failure of applications within a container. For example in a memory shortage situation on the host an admin may kill a process that is a memory hog on the host without realising it is spawned from a container application.

The display below is a process display on the host, every single one of the results (apart from the grep itself) are processes launced from within the container.

[root@vosprey2 log]# ps -ef | grep herc
root      1203  1185  0 15:45 ?        00:00:00 /bin/bash /home/mark/hercules/tk4-minus/
root      1240  1203  0 15:45 ?        00:00:00 /usr/bin/su -c bash /home/mark/hercules/tk4-minus/ > /var/tmp/hercstart.log 2>&1 mark
mark      1241  1240  0 15:45 ?        00:00:00 bash -c bash /home/mark/hercules/tk4-minus/ > /var/tmp/hercstart.log 2>&1
mark      1242  1241  0 15:45 ?        00:00:00 bash /home/mark/hercules/tk4-minus/
mark      1244  1203  0 15:45 ?        00:00:00 SCREEN -t hercules -S hercules -p hercules -d -m hercules -f mark/marks.conf
mark      1246  1244  9 15:45 pts/0    00:00:49 hercules -f mark/marks.conf
mark      3321  1246  0 15:46 pts/0    00:00:00 hercules -f mark/marks.conf
mark      3322  3321  0 15:46 pts/0    00:00:00 /bin/bash /home/mark/hercules/tk4-minus/mark/scripts/
mark      3333  3322  0 15:46 pts/0    00:00:00 /bin/bash /home/mark/hercules/tk4-minus/mark/scripts/
root     11963 10154  0 15:53 pts/0    00:00:00 grep --color=auto herc
[root@vosprey2 log]# 

Network connections are hidden

In a complete reverse of the above issue, network connections to applications within a Docker container are not visible on the host machine. That can make the job of diagnosing network connectivity issues difficult as the admin would normally look on the host for established tcpip sessions; but established sessions to container applications are not visible on the host.

Refer to the output below. The container application listens on port 3270 but while netstat shows no established sessions in this example a computer remote to the docker host does have an established connection via the host into the container on that port. A docker exec into the application lets us see that established connection from and it is dangerous to not have that connection displayed on the docker host. The last command in the output below was run on the computer that established the session, it correctly shows it is connected to the docker host (189) to access the port mapped to the container even though displays on the docker host do not show the connection.

By dangerous I mean in a large environment if there are network issues to be resolved an admin cannot be expected to attach to or exec into every container to see what established connections exist on a host (assuming the container is even built to contain the netstat command), not having established connections to a host displayable on the host I would consider a serious issue. And a ‘ip netns’ shows no network namespaces are in use so I really don’t know how the connection is being hidden.

[root@vosprey2 log]# netstat -an | grep 3270
tcp6       0      0 :::3270                 :::*                    LISTEN     

[root@vosprey2 log]# docker exec -ti mvs38j1 /bin/bash
bash-5.0# netstat -an | grep 3270
tcp        0      0  *               LISTEN     
tcp        0      0         ESTABLISHED
tcp        0      0          ESTABLISHED
tcp        0      0     ESTABLISHED
tcp        0      0          ESTABLISHED
tcp        0      0         ESTABLISHED
tcp        0      0          ESTABLISHED
tcp        0      0         ESTABLISHED
unix  2      [ ACC ]     STREAM     LISTENING     16297755 /run/screen/S-mark/71.c3270A
unix  2      [ ACC ]     STREAM     LISTENING     16297819 /run/screen/S-mark/76.c3270B
bash-5.0# exit

[root@vosprey2 log]# netstat -an | grep 3270 
tcp6       0      0 :::3270                 :::*                    LISTEN     
[root@vosprey2 log]#
[root@vosprey2 log]# ip netns
[root@vosprey2 log]#

[mark@phoenix mvs38j]$ netstat -an | grep 3270
tcp        0      0      ESTABLISHED
[mark@phoenix mvs38j]$ 

The dangers of running apps in containers as non-root

Everybody will tell you user applications should never be run as the root user, and applications in containers should also follow that rule.

There is a major issue with running applications as a non-root user in containers however.

Remember the processes lauched within a host container run as actual processes on the docker host, and as containers are supposed to be portable there is no way to guarantee that UIDs assigned to users within the container will match UIDs on the docker host.

Refer the the output shown below. Within the container the application runs under the userid ‘ircserver’ with a uid of 1000, all good right.

[root@vosprey2 log]# docker exec -ti ircserver1 /bin/bash
bash-5.0# ps -ef 
root         1     0  0 03:28 ?        00:00:00 /bin/bash /home/ircserver/marks_irc_server
ircserv+    23     1  0 03:28 ?        00:00:00 /home/ircserver/IRC/inspircd-3.3.0/run/bin/inspircd --config=/home/ircserver/IRC/inspircd-3.3.0/run/conf/inspircd.conf
root        81     1  0 03:58 ?        00:00:00 sleep 600
root        82     0  7 04:01 ?        00:00:00 /bin/bash
root        87    82  0 04:01 ?        00:00:00 ps -ef
bash-5.0# grep ircserver /etc/passwd
ircserver:x:1000:1000:Used to run the inspircd IRC server:/home/ircserver:/bin/bash

Wrong !. Displaying the process on the docker host shows a different story. On the docker host it also runs under UID 1000 correctly, but on the docker host the ircserver user does not exist and another user is assigned uid 1000.

[root@vosprey2 log]# ps -ef | grep IRC | grep -v grep
mark     17875 17834  0 15:28 ?        00:00:00 /home/ircserver/IRC/inspircd-3.3.0/run/bin/inspircd --config=/home/ircserver/IRC/inspircd-3.3.0/run/conf/inspircd.conf
[root@vosprey2 log]# 
[root@vosprey2 log]# grep 1000 /etc/passwd
mark:x:1000:1000:Mark Dickinson:/home/mark:/bin/bash
[root@vosprey2 log]# 

The obvious major issue here is that unless all container applications are run as root they cannot be considered portable, as there is no way to ensure UIDs either match or do not exist on every docker host that may ever run the image.

Imagine again from a admin perspective trying to troubleshoot a memory hog process, and a ‘ps’ shows the issue causing process is being run by user ‘fred’ but user fred swears he never started the process; he may not have started the process, it could have been spawned from a container using a matching UID for fred. Admitedly if they were all root processes admins would still have to track down the container causing the impact; but would not have been sidetracked into wasting time chasing after fred.

Also lets not forget that ‘fred’ can at any time kill any of those processes he has been granted ownership of, causing issues with the application within the container.

Operating system dependent

Containers are not truely portable, they should be considered operating system dependent and a container build on Fedora 30 should only run on a Fedora 30 host; and the container and host should be on similar patch levels. If an OS upgrade is done all containers should be rebuilt.

Obviously this depends on the complexity of your container applications. One I have been working on requires –device mapping and overlaying the host /lib/modules directory over the container… because when the container is run ‘uname’ reports the host kernel level not the kernel version the container was build from so the container does not have the correct modules. But it is fair to say a container OS must be a very close match to the host OS in order to function.


If you are security conscious you would not consolidate applications running on KVM hosts into Docker containers.

If your environment is secure and totally locked down then docker containers can be used to consolidate applications running on KVM instances. You will get no application memory savings from moving an application (if an app needs 1Gb to run on a KVM instance it will still need 1Gb to run in a Docker container) but you will get around 750Mb OS alone overhead back from each KVM instance shutdown if you are migrating from KVM to Docker.

And of course containers start faster than booting a KVM, which I personally do not consider a selling point. If designed properly images enable copies of applications to be started in containers on multiple hosts with minimal effort, of course KVM live migration has been a thing for a long time now so thats not really a major selling point either. Being able to encapulate an entire online application in an image is useful.

Docker networking between Docker containers in a multihost docker engine environment is much hyped, and much is touted about docker swarms, so I’m sure docker internal networking works well although I have not needed to play with any of that stuff.

External networking from the docker containers is another issue. One network issue I have already highlighted above, from a troubleshooting viewpoint on the host.

There are many other networking issues I will probably cover in a later post, when I figure out how to resolve them, lets just say if your docker container needs pass-through access to external hosts outside the internal docker network, prepare for a long frustrating time.

About mark

At work, been working on Tandems for around 30yrs (programming + sysadmin), plus AIX and Solaris sysadmin also thrown in during the last 20yrs; also about 5yrs on MVS (mainly operations and automation but also smp/e work). At home I have been using linux for decades. Programming background is commercially in TAL/COBOL/SCOBOL/C(Tandem); 370 assembler(MVS); C, perl and shell scripting in *nix; and Microsoft Macro Assembler(windows).
This entry was posted in Unix. Bookmark the permalink.