Docker is not KVM, there are major security trade-offs with a container, The key ones are shown below.

Processes are not isolated

The processes that are run by containers run for all intents and purposes as processes on the Docker host machine, this is an issue as an admin on the host machine may inadvertently cause failure of applications within a container. For example in a memory shortage situation on the host an admin may kill a process that is a memory hog on the host without realising it is spawned from a container application.

The display below is a process display on the host, every single one of the results (apart from the grep itself) are processes launced from within the container.

[root@vosprey2 log]# ps -ef | grep herc
root      1203  1185  0 15:45 ?        00:00:00 /bin/bash /home/mark/hercules/tk4-minus/start_system_wrapper.sh
root      1240  1203  0 15:45 ?        00:00:00 /usr/bin/su -c bash /home/mark/hercules/tk4-minus/start_system.sh > /var/tmp/hercstart.log 2>&1 mark
mark      1241  1240  0 15:45 ?        00:00:00 bash -c bash /home/mark/hercules/tk4-minus/start_system.sh > /var/tmp/hercstart.log 2>&1
mark      1242  1241  0 15:45 ?        00:00:00 bash /home/mark/hercules/tk4-minus/start_system.sh
mark      1244  1203  0 15:45 ?        00:00:00 SCREEN -t hercules -S hercules -p hercules -d -m hercules -f mark/marks.conf
mark      1246  1244  9 15:45 pts/0    00:00:49 hercules -f mark/marks.conf
mark      3321  1246  0 15:46 pts/0    00:00:00 hercules -f mark/marks.conf
mark      3322  3321  0 15:46 pts/0    00:00:00 /bin/bash /home/mark/hercules/tk4-minus/mark/scripts/printer_interface.sh
mark      3333  3322  0 15:46 pts/0    00:00:00 /bin/bash /home/mark/hercules/tk4-minus/mark/scripts/printer_interface.sh
root     11963 10154  0 15:53 pts/0    00:00:00 grep --color=auto herc
[root@vosprey2 log]# 

Network connections are hidden

In a complete reverse of the above issue, network connections to applications within a Docker container are not visible on the host machine. That can make the job of diagnosing network connectivity issues difficult as the admin would normally look on the host for established tcpip sessions; but established sessions to container applications are not visible on the host.

Refer to the output below. The container application listens on port 3270 but while netstat shows no established sessions in this example a computer remote to the docker host does have an established connection via the host into the container on that port. A docker exec into the application lets us see that established connection from 192.168.1.187 and it is dangerous to not have that connection displayed on the docker host. The last command in the output below was run on the computer that established the session, it correctly shows it is connected to the docker host (189) to access the port mapped to the container even though displays on the docker host do not show the connection.

By dangerous I mean in a large environment if there are network issues to be resolved an admin cannot be expected to attach to or exec into every container to see what established connections exist on a host (assuming the container is even built to contain the netstat command), not having established connections to a host displayable on the host I would consider a serious issue. And a ‘ip netns’ shows no network namespaces are in use so I really don’t know how the connection is being hidden.

[root@vosprey2 log]# netstat -an | grep 3270
tcp6       0      0 :::3270                 :::*                    LISTEN     

[root@vosprey2 log]# docker exec -ti mvs38j1 /bin/bash
bash-5.0# netstat -an | grep 3270
tcp        0      0 0.0.0.0:3270            0.0.0.0:*               LISTEN     
tcp        0      0 127.0.0.1:3270          127.0.0.1:39892         ESTABLISHED
tcp        0      0 127.0.0.1:39896         127.0.0.1:3270          ESTABLISHED
tcp        0      0 172.17.0.2:3270         192.168.1.187:48842     ESTABLISHED
tcp        0      0 127.0.0.1:39892         127.0.0.1:3270          ESTABLISHED
tcp        0      0 127.0.0.1:3270          127.0.0.1:39896         ESTABLISHED
tcp        0      0 127.0.0.1:39894         127.0.0.1:3270          ESTABLISHED
tcp        0      0 127.0.0.1:3270          127.0.0.1:39894         ESTABLISHED
unix  2      [ ACC ]     STREAM     LISTENING     16297755 /run/screen/S-mark/71.c3270A
unix  2      [ ACC ]     STREAM     LISTENING     16297819 /run/screen/S-mark/76.c3270B
bash-5.0# exit
exit

[root@vosprey2 log]# netstat -an | grep 3270 
tcp6       0      0 :::3270                 :::*                    LISTEN     
[root@vosprey2 log]#
[root@vosprey2 log]# ip netns
[root@vosprey2 log]#

[mark@phoenix mvs38j]$ netstat -an | grep 3270
tcp        0      0 192.168.1.187:48842     192.168.1.189:3270      ESTABLISHED
[mark@phoenix mvs38j]$ 

The dangers of running apps in containers as non-root

Everybody will tell you user applications should never be run as the root user, and applications in containers should also follow that rule.

There is a major issue with running applications as a non-root user in containers however.

Remember the processes lauched within a host container run as actual processes on the docker host, and as containers are supposed to be portable there is no way to guarantee that UIDs assigned to users within the container will match UIDs on the docker host.

Refer the the output shown below. Within the container the application runs under the userid ‘ircserver’ with a uid of 1000, all good right.

[root@vosprey2 log]# docker exec -ti ircserver1 /bin/bash
bash-5.0# ps -ef 
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 03:28 ?        00:00:00 /bin/bash /home/ircserver/marks_irc_server
ircserv+    23     1  0 03:28 ?        00:00:00 /home/ircserver/IRC/inspircd-3.3.0/run/bin/inspircd --config=/home/ircserver/IRC/inspircd-3.3.0/run/conf/inspircd.conf
root        81     1  0 03:58 ?        00:00:00 sleep 600
root        82     0  7 04:01 ?        00:00:00 /bin/bash
root        87    82  0 04:01 ?        00:00:00 ps -ef
bash-5.0# grep ircserver /etc/passwd
ircserver:x:1000:1000:Used to run the inspircd IRC server:/home/ircserver:/bin/bash
bash-5.0# 

Wrong !. Displaying the process on the docker host shows a different story. On the docker host it also runs under UID 1000 correctly, but on the docker host the ircserver user does not exist and another user is assigned uid 1000.

[root@vosprey2 log]# ps -ef | grep IRC | grep -v grep
mark     17875 17834  0 15:28 ?        00:00:00 /home/ircserver/IRC/inspircd-3.3.0/run/bin/inspircd --config=/home/ircserver/IRC/inspircd-3.3.0/run/conf/inspircd.conf
[root@vosprey2 log]# 
[root@vosprey2 log]# grep 1000 /etc/passwd
mark:x:1000:1000:Mark Dickinson:/home/mark:/bin/bash
[root@vosprey2 log]# 

The obvious major issue here is that unless all container applications are run as root they cannot be considered portable, as there is no way to ensure UIDs either match or do not exist on every docker host that may ever run the image.

Imagine again from a admin perspective trying to troubleshoot a memory hog process, and a ‘ps’ shows the issue causing process is being run by user ‘fred’ but user fred swears he never started the process; he may not have started the process, it could have been spawned from a container using a matching UID for fred. Admitedly if they were all root processes admins would still have to track down the container causing the impact; but would not have been sidetracked into wasting time chasing after fred.

Also lets not forget that ‘fred’ can at any time kill any of those processes he has been granted ownership of, causing issues with the application within the container.

Operating system dependent

Containers are not truely portable, they should be considered operating system dependent and a container build on Fedora 30 should only run on a Fedora 30 host; and the container and host should be on similar patch levels. If an OS upgrade is done all containers should be rebuilt.

Obviously this depends on the complexity of your container applications. One I have been working on requires –device mapping and overlaying the host /lib/modules directory over the container… because when the container is run ‘uname’ reports the host kernel level not the kernel version the container was build from so the container does not have the correct modules. But it is fair to say a container OS must be a very close match to the host OS in order to function.

Summary

If you are security conscious you would not consolidate applications running on KVM hosts into Docker containers.

If your environment is secure and totally locked down then docker containers can be used to consolidate applications running on KVM instances. You will get no application memory savings from moving an application (if an app needs 1Gb to run on a KVM instance it will still need 1Gb to run in a Docker container) but you will get around 750Mb OS alone overhead back from each KVM instance shutdown if you are migrating from KVM to Docker.

And of course containers start faster than booting a KVM, which I personally do not consider a selling point. If designed properly images enable copies of applications to be started in containers on multiple hosts with minimal effort, of course KVM live migration has been a thing for a long time now so thats not really a major selling point either. Being able to encapulate an entire online application in an image is useful.

Docker networking between Docker containers in a multihost docker engine environment is much hyped, and much is touted about docker swarms, so I’m sure docker internal networking works well although I have not needed to play with any of that stuff.

External networking from the docker containers is another issue. One network issue I have already highlighted above, from a troubleshooting viewpoint on the host.

There are many other networking issues I will probably cover in a later post, when I figure out how to resolve them, lets just say if your docker container needs pass-through access to external hosts outside the internal docker network, prepare for a long frustrating time.

Logcheck on fedora, by default when installed, runs every hour. For my use this generates so much email traffic (as it runs on all my servers) that it becomes garbage, and I ended up just deleting all the emails rather than reading them.

In reference to the garbage, by garbage I mean that running hourly, against the same log files, resulted in the emails being sent containing pretty much duplicate information from the prior run; pointless. Yes I could change the time parameters but I don’t want the emails every hour anyway.

So I decided to change it from hourly to daily, while that does not mean I will read them all it does mean I have a chance to actually check other emails from other subsystems that tended to be deleted while I was bulk deleting the logcheck ones.

The scheduling for logcheck does not reside in /etc/cron.hourly to be simply moved to /etc/cron.daily to be run whenever /etc/anacrontab decides to run it during the defined day, it is actually in a file /etc/cron.d/logcheck to run at a specific time.

Point of interest: files in /etc/cron.d are similar to, but not the same as normal crontab entries, the main difference being there is a requirement to specify the userid a job is to be run as. The interesting point however is that files in /etc/cron.d also have the interesting property of being able to be scheduled to run at ‘@reboot’ rather than a specific crontab format time such as ‘2 * * * *’, which I may find a use for at some point.

So I simply changed the default scheduling time in /etc/cron.d/logcheck from ‘2 * * * *’ to ‘2 2 * * *’ to run at 2:02am daily rather than at two minutes past every hour.

I also have found a need to create custom suppression filters, which will be an ongoing work in progress, those rules also need to be copied to all my servers.

So as this affected all my servers I created a puppet class for logcheck and added it to the ‘base’ profile, and added it to my generic ‘allservers’ role that contains common configurations and packages needed on all servers. This will ensure all servers are correctly setup as I was starting to lose track of which ones I had manually updated the cron.d entry on, and now I am creating custom filter rules that will be updated fairly regularly I just want to update everything from one place, which puppet lets me do.

While some may think the simple change to scheduling time was probably not worth a post, I think the ability to use ‘@reboot’ as a scheduling time option in files under /etc/cron.d has potential, which many of you may not have known about had you not read this pointless post. Irritatingly I knew that function existed but had forgotten about it, so pherhaps this post is more a reminder to me :-)

One of my machines has started rebooting for no reason that I can find, possibly power surges as my UPS seems have now have a dead battery.

One difficulty in finding the issue is that the messages file if full of audit messages, which are a waste of space as those same messages also get logged to /var/log/audit as well.

How to turn them off was a question posted on fedoraforum.org that I posted the answer to as I found a similar match on stackexchange, but having rebuilt a few machines and having to search out the answer I posted again I have decided to place the inforamtion in my blog as well, simply so it is easier for me to find.

Adding as the first two lines under the #### RULES #### section in /etc/rsyslog.conf and doing a systemctl restart rsyslog will stop the messages being logged to the messages file, they are still logged to audit.log if you want to look at them.


# no audit
:programname, isequal, "audit" ~


That is not ideal as it logs a depreciated warning and is simply suppression that does not address the source of the problem which is the audit messages being somewhere configured to write to both syslog and the audit log. It does not address the issue as you say, of why the messages are being generated in the first place.

But getting rid of the bulk of the messages being logged to /var/log/messages may help me track down why my machine is rebooting itself… which is a pain as it has a luks encrypted disk(s) so while it tries to restart itself, it cannot.

There was a post on the fedora forums stating that the certbot apache plugin does not work on fedora, so I had a look. The post was correct, the apache plugin for certbot wants to use the “apachectl -v” command which simply does not exist on fedora, so certbot with the –apache option will always error and fail on fedora.

That rpm package for the apache plugin should probably be removed from the Fedora repositories, but that is not my call. The cerbot package itself I found works OK and provides all the hands off functionality needed.

As I was interested in replacing my self signed certificates with a “real” one from LetsEncrypt I had a more detailed look into certbot and it’s options.

I had avoided looking into it in the past as my sites name is resolved by dyndns, is a .dyndns.org address, and as I don’t own the dyndns.org name wrongly assumed I could not obtain a certificate that included it. But is is possible using the –webroot option.

The goal of course is to use LetsEncrypt to provide valid certificates for my website to use, and to have them managed in an automated hands-off way.

One important thing to note in using certbot is that despite the entire idea of SSL certs is to have all traffic across port 443 it is necessary to also have ports open for the unencrypted port 80 traffic and your web server handling requests on that port as that is needed by certbot. If you get errors about your server being unreachable it will be because you have port 80 blocked (or your dns name is unresolveable on the internet which is a completely seperate issue).

The first time you run certbot it will prompt for details such as your name and contact information. Unfortunately I did not take a copy of the output for that as it was the first of many failed attempts as my port 80 was blocked at that time. Once the information is entered it is not asked for again.
I assume it is embedded in the private files created under /etc/letsencrypt/live/mdickinson.dyndns.org or other files under the directories created under /etc/letsencrypt; it is enough to know enter the correct details in the first place as they would appear to be difficult to change.

Anyway, once port 80 is opened, using the –webroot option works perfectly. You need to run the command as root as it needs to create lots of directories under /etc/letsencrypt. The –webroot path needs to be the directory your websites html directory is served from, certbot will create a temporary directory under there that the LetsEncrypt servers will attempt to read to verify that the internet dns name used as your domain name does in fact resolve to your webserver, if they can from the big wide internet resolve your server address and read the temporary files created under your webserver path it verifies you have admin control over the machine the webserver is hosted on and creates the certificates.

A log of the working request, unfortunately missing the prompts for site details as mentioned above, is below.

[root@vosprey2 httpd]# certbot certonly --webroot -w /var/www/html -d your.hostname.org
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for your.hostname.org
Using the webroot path /var/www/html for all unmatched domains.
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/your.hostname.org/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/your.hostname.org/privkey.pem
   Your cert will expire on 2019-08-13. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

The following changes were made to /etc/httpd/conf.d/ssl.conf to replace my self signed csr certificate files with the LetsEncrypt ones. They were the only changes needed.

[root@vosprey2 conf.d]# grep -i letsencrypt ssl.conf
SSLCertificateFile /etc/letsencrypt/live/your.hostname.org/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/your.hostname.org/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/your.hostname.org/fullchain.pem

A “systemctl restart httpd” to pick up the new keys and clicking on the lock on a https browsing of the website shows the letsencrypt provided certificate is working perfectly.

Note that I changed the cerificate paths to the location that LetsEncrypt places the certificates, this in theory means that a simple “certbot renew” as shown in the output logs will replace the certificates with no need to make further changes to my configuration.

Of course a scheduled job will need to be setup to run the command and perform a restart of httpd, plus parse the output to find the next expiry date and reschedule the job; all on the todo list. For now I have a LetsEncrypt certificate instead of a self signed one.

Using certbot to obtain LetsEncrypt signed certificates for Fedora 30 is quite easy; if you haven’t already give it a try.

While the resolution was found in the Fedora forums as RedHat is built from successfull pilots of new code in Fedora in may exist in RedHat systems as well.

I had major issues upgrading my KVM F29 instances to F30. Every attempt attempt resulted in the “grub>” prompt. The beauty of KVM instances of course is I could just restore the qcow2 disk image and retry a lot of different options until I found the post with the solution to the issue.

This does affect physical machines as well, the post I found with the resloution was posted by someone who had hit the issue with a physical system and had to resort to live-cd to rebuild their grub setup. I was lucky in that the only physical machine I had updated was recently rebuilt from F28 and uograded to F29 before I did the F30 upgrade, so was up-to-date enough to upgrade OK, if I had attempted the upgrade on my main machine it would have been bricked, well at the “grub>” prompt requiring hours of work with a live-cd to recover.

I am dubious about taking the risk of updating the F29 OS on my dual boot laptop as there are lots of posts reporting that grub2 can only find windoze partitions after an upgrade to F30, but I guess I will have to attempt it at some point. That is probably a seperate issue I may post on if that goes wrong.

Anyway the issue identified by the post I found is that on BIOS machines (and KVM as it emulates a BIOS environment) I have been running a while that have been upgraded through F24,F25,F26,F27,F28,F29 and now F30 the updates will update the grub2 software packages installed but do not actually update the physical grub2 boot environment; the MBR (or boot sector if you boot off a partition instead of the MBR) itself is not updated with the newer versions of the software; the boot sector becomes stale.

With the update to F30 the differences between the stale entries in the MBR and the installed software on the system(s) result in the “grub>” prompt being displayed instead of the boot menu. Somewhere along the upgrade chain the way grub2 is layed out on the filesystem has changed and the older MBR version of grub2 cannot handle the new layout.

The way to sucessfully update the KVM instances is to replace the normal upgrade of

sudo dnf upgrade --refresh
sudo dnf install dnf-plugin-system-upgrade
sudo dnf system-upgrade download --releasever=30
sudo dnf system-upgrade reboot

With the below, there is just the one extra command

sudo dnf upgrade --refresh
sudo dnf install dnf-plugin-system-upgrade
sudo dnf system-upgrade download --releasever=30
sudo grub2-install /dev/vda   # < ---- the extra line, change to your boot disk MBR or partition
sudo dnf system-upgrade reboot

Obviously on physical hardware you need to know if you are booting from the MBR or a partition on a disk, and perform the grub2-install onto the correct boot device; and also it is only needed if you also have not done so for a while, personally I will do it for every new upgrade.

Fedora 30 grub issues after the upgrade are getting quite a few posts raised on the furums, this must be one of the messiest upgrades in quite a while. Not looking forward to upgrading the dual boot laptop at all.

I used to rely on BasKet notes on both Fedora and Redhat

After rebuilding a Desktop I discovered it was no longer available in the Fedora repositories. This may not be a new thing as I discovered I had also downloaded/saved the RPM file for it so must have manually installed in to use in the past, I would assume to avoid losing years worth of how-to notes I had saved. That application appears to no longer be in active development which is a shame.

This time I decided to move forward, and after a fair bit of searching I found that ‘CherryTree’ provided similar functionality and as an added bonus the documentation indicated it could import basket files. It uses SQLite for storage but unlike many other products that use that it provides a database compression/reclaim from within the application menu (addmitedly I have not tried to compress the database yet as it will be a good few months of entries before I have enough fragmentation to see if it does anything).

Anyway, the ‘CherryTree’ package is still supported in the Fedora repositories so it was a simple case of ‘dnf -y install cherrytree’ to make it available.

I uncompressed a baskets backup and used CherryTree to import the baskets data… it worked. So I can move on using a currently supported application with loss of years of notes. Its functionality for day-to-day use is also very similar, so a good match as a replacement application.

Having had to rebuild another machine, After it was up and running I had a server with the OS installed on the boot disk and two spare internal disks that used to be in a raid array. While the machine supported hardware raid I chose to setup software raid as it gives me more control, and personally I would rather ‘tweak’ with the command line than at the hardware bios level.

Anyway, rather than re-create the wheel, there is a very good article at https://www.tecmint.com/create-raid1-in-linux/ that explains how to mirror up a spare two disks.

I put a LUKs filesystem on mirrored disks and made it a /home directory.