These are the issues I had personally, as every user has a different setup your distance may vary :-). They are covered in detail below.
- Desktop upgrade does not work, no big deal (workaround works)
- Network manager misbehaves (unsolved)
- Bacula database tables must be created from scratch, the table upgrade script does not result in a usable database (recreating all tables works)
- mariadb-libs package did not upgrade correctly. After a “dnf reinstall mariadb-libs” the package appears to be installed correctly, but the library files are still causing errors (unsolved)
- nrpe plugins, as always some stop working (unsolved, have to write replacements)
- further things yet to discover :-)
Desktop users have to do it from the command line as well
One of my VM hosts has a full Gnome install, and it popped up the Fedora27 now available upgrade window. So I used it, after the reboot it said it was upgrading but it restarted still on F26… and popped up the upgrade window that I used again, after the reboot it said it was upgrading and after the reboot it was still on F26.
So I stopped wasting time and used the command line method with the dnf system-upgrade commands, that sucessfully upgraded to F27.
NetworkManager will always start
I had network issues on one host, so I disabled NetworkManager, but with NetworkManager disabled the blasted thing starts always anyway.
[root@vmhost3 ~]# systemctl status NetworkManager
? NetworkManager.service - Network Manager
Loaded: loaded (/usr/lib/systemd/system/NetworkManager.service; disabled; vendor preset: enabled)
Active: active (running) since Sun 2017-11-19 10:13:36 NZDT; 2min 31s ago
Docs: man:NetworkManager(8)
Main PID: 1323 (NetworkManager)
Tasks: 3 (limit: 4915)
CGroup: /system.slice/NetworkManager.service
??1323 /usr/sbin/NetworkManager --no-daemon
Nov 19 10:14:23 vmhost3 NetworkManager[1323]: [1511039663.2271] device (virbr0): bridge port virbr0-nic wa
Nov 19 10:14:23 vmhost3 NetworkManager[1323]: [1511039663.2271] device (virbr0-nic): Activation: connectio
Nov 19 10:14:23 vmhost3 NetworkManager[1323]: [1511039663.2272] device (virbr0): state change: secondaries
Nov 19 10:14:23 vmhost3 NetworkManager[1323]: [1511039663.2273] manager: NetworkManager state is now CONNE
Nov 19 10:14:26 vmhost3 NetworkManager[1323]: [1511039666.6732] device (virbr0): Activation: successful, d
Nov 19 10:14:26 vmhost3 NetworkManager[1323]: [1511039666.6772] device (virbr0-nic): state change: ip-conf
Nov 19 10:14:26 vmhost3 NetworkManager[1323]: [1511039666.6780] device (virbr0-nic): state change: seconda
Nov 19 10:14:26 vmhost3 NetworkManager[1323]: [1511039666.6783] device (virbr0): bridge port virbr0-nic wa
Nov 19 10:14:26 vmhost3 NetworkManager[1323]: [1511039666.6784] device (virbr0-nic): released from master
Nov 19 10:14:54 vmhost3 NetworkManager[1323]: [1511039694.5366] bluez: use BlueZ version 5
[root@vmhost3 ~]#
That does not appear to be causing my issue however as my network-scripts are configured not to use NetworkManager and I have “network” enabled; but it does not appear to be working correctly on that one host. And doing a systemctl restart network (which I would hope leaves NetworkManager out of it) does not fix the issue.
[root@vmhost3 network-scripts]# grep -i DNS ifcfg* ifcfg-br0:DNS2="192.168.1.179" ifcfg-br0:DNS1="192.168.1.181" ifcfg-br0:DNS3="192.168.1.1" ifcfg-enp2s0:IPV6_PEERDNS="no" ifcfg-enp2s0:PEERDNS="yes" ifcfg-enp2s0:IPV6_PEERDNS="yes" [root@vmhost3 network-scripts]# grep -i GATE ifcfg* ifcfg-br0:GATEWAY0="192.168.1.1" ifcfg-lo:# If you're having problems with gated making 127.0.0.0/8 a martian, [root@vmhost3 network-scripts]# cat /etc/resolv.conf # Generated by NetworkManager nameserver 192.168.1.181 nameserver 192.168.1.179 [root@vmhost3 network-scripts]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 br0 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 br0 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0 [root@vmhost3 network-scripts]# [root@vmhost3 network-scripts]# uname -a Linux vmhost3 4.13.12-300.fc27.x86_64 #1 SMP Wed Nov 8 16:38:01 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Worse, it is not consistently possible to manually correct the blasted thing. Using “route add default gw 192.168.1.1” worked after the fourth reboot; prior to that it made no difference to the routing table.
The /etc/resolv.conf has to be manually edited to add the third DSN nameserver entry.
On a second host that has been upgraded networking is setup almost correctly, it adds two entries for the default route instead of only one expected but it seems to work OK.
[root@vmhost1 network-scripts]# grep -i DNS ifcfg* ifcfg-br0:DNS1="192.168.1.181" ifcfg-br0:DNS2="192.168.1.179" ifcfg-br0:DNS3="192.168.1.1" ifcfg-em1:IPV6_PEERDNS="no" ifcfg-em1:PEERDNS="yes" [root@vmhost1 network-scripts]# grep -i GATE ifcfg* ifcfg-br0:GATEWAY0="192.168.1.1" ifcfg-lo:# If you're having problems with gated making 127.0.0.0/8 a martian, [root@vmhost1 network-scripts]# cat /etc/resolv.conf # Generated by NetworkManager nameserver 192.168.1.181 nameserver 192.168.1.179 nameserver 192.168.1.1 [root@vmhost1 network-scripts]# route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 br0 0.0.0.0 192.168.1.1 0.0.0.0 UG 425 0 0 br0 192.168.1.0 0.0.0.0 255.255.255.0 U 425 0 0 br0 [root@vmhost1 network-scripts]# uname -a Linux vmhost1 4.13.12-300.fc27.x86_64 #1 SMP Wed Nov 8 16:38:01 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux [root@vmhost1 network-scripts]#
The only difference between the configuration files on the two hosts is the interface names and the UUID/HWADDR of the entries so both servers should have the same result, but they don’t.
Bacula Director Database Update
No problems running the normal mysql_upgrade script but there are issues with bacula if you are running mariadb in safe mode.
The upgrade script for mariadb needs work. In the file /usr/libexec/bacula/update_mysql_tables you must search for the line “UPDATE Version SET VersionId=16;” and wrap two lines about it thus…
SET SQL_SAFE_UPDATES=0; UPDATE Version SET VersionId=16; SET SQL_SAFE_UPDATES=1;
This avoids the error “ERROR 1175 (HY000): You are using safe update mode and you tried to update a table without a WHERE that uses a KEY column” which prevents the version number being updated.
However it remained broken. It looks like the upgrade is a no-go, and I will have to try recreating all the tables from scratch, Backups running after the mysql upgrade scripts completed sucessfully stil failed with database errors…
21-Nov 17:00 bacula-dir JobId 0: Fatal error: sql_create.c:84 Create DB Job record INSERT INTO Job (Job,Name,Type,Level,JobStatus,SchedTime,JobTDate,ClientId,Comment) VALUES ('BackupVMhost3.2017-11-21_17.00.00_12','BackupVMhost3','B','I','C','2017-11-21 17:00:00',1511236800,11,'') failed. ERR=Field 'StartTime' doesn't have a default value
21-Nov 17:00 bacula-dir JobId 0: Fatal error: sql_create.c:84 Create DB Job record INSERT INTO Job (Job,Name,Type,Level,JobStatus,SchedTime,JobTDate,ClientId,Comment) VALUES ('BackupPuppet.2017-11-21_17.00.01_13','BackupPuppet','B','I','C','2017-11-21 17:00:01',1511236801,13,'') failed. ERR=Field 'StartTime' doesn't have a default value
21-Nov 23:55 bacula-dir JobId 0: Fatal error: sql_create.c:84 Create DB Job record INSERT INTO Job (Job,Name,Type,Level,JobStatus,SchedTime,JobTDate,ClientId,Comment) VALUES ('BackupCatalog.2017-11-21_23.55.00_14','BackupCatalog','B','F','C','2017-11-21 23:55:00',1511261700,7,'') failed. ERR=Field 'StartTime' doesn't have a default value
That did actually supprise me as the upgrade scripts for bacula normally work; but not this time.
The solution unfortunately is to stop bacula-dir and bacula-sd, delete all the tables in the bacula database and recreate it from scratch using /usr/libexec/bacula/make_mysql_tables; plus you must also delete everything from /var/spool/bacula
Then you can restart bacula-sd and bacula-dir. And unfortunately as you no longer have any backups you must then run backups for each of your servers which will be full backups on the first run for each. You could wait until the backups are scheduled to run but as they will be full backups on the first run it is probably better to manually do them at a time when you know network congestion is at a minimum.
But at least backups are working again.
The HTTP service
As always, if you need to share files between webapps and scripts, vi /usr/lib/systemd/system/httpd.service and change PrivateTmp=true to PrivateTmp=false.
Mariadb library issues
While mariadb is running OK and the mysql_upgrade script had no problems, there appears to be something wrong with the mariadb-libs packaging, certainly not all the files rpm -ql showed for the package existed after the upgrade to F27.
This issue was found in investigating issues with the NRPE mysql plugin so refer to that section for details on the issues with mariadb-libs. At this point in time I have not found a resolution.
Note that a “dnf reinstall mariadb-libs” did create all the expected files, but even though the ld.conf.so.d file for mysql is correct even after rerunning ldconfig the mysql library files are not found by programs wanting to use them; at least not found by th nrpe plugin which is what I am trying to get working first.
NRPE plugins
As always the systemd startup scripts for NRPE have to be edited to turn off the requirement to use SSL, thats no real issue as I have gotten used to that.
And as always more of the supplied plugins have stopped working. The HTTP and MYSQL ones have stopped working this time.
The HTTP plugin
The HTTP plugin is triggering error 400’s from the server for no reason that I can determine. As seen in the log clip the server responds OK to non-nrpe requests.
192.168.1.170 - - [21/Nov/2017:09:47:20 +1300] "GET / HTTP/1.0" 400 226 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" 192.168.1.170 - - [21/Nov/2017:09:52:20 +1300] "GET / HTTP/1.0" 400 226 "-" "check_http/v2.2.1 (nagios-plugins 2.2.1)" 192.168.1.179 - - [21/Nov/2017:09:54:34 +1300] "GET / HTTP/1.1" 200 2285 "-" "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0"
Using telnet to “GET / HTTP/1.0” and “GET / HTTP/1.1” manually also get an error 400 (although using just “GET /” works OK); so pherhaps a behaviour change in HTTP itself… although Firefox has no issue retrieving the page with the full string.
The MYSQL plugin, possibly an issue with mariadb packages and not the plugin ?
[root@vosprey2 httpd]# /usr/lib64/nagios/plugins/check_mysql /usr/lib64/nagios/plugins/check_mysql: error while loading shared libraries: libmysqlclient.so.18: cannot open shared object file: No such file or directory
RPM says that file is installed as part of mariadb-libs… which is installed but that file is not.
[root@vosprey2 httpd]# dnf provides libmysqlclient.so.18 Last metadata expiration check: 0:17:01 ago on Tue 21 Nov 2017 10:22:24 NZDT. mariadb-libs-3:10.2.9-3.fc27.i686 : The shared libraries required for MariaDB/MySQL clients Repo : fedora Matched from: Provide : libmysqlclient.so.18 [root@vosprey2 httpd]# rpm -qa | grep -i mariadb-libs mariadb-libs-10.2.9-3.fc27.x86_64 [root@vosprey2 httpd]# rpm -ql mariadb-libs /etc/ld.so.conf.d/mariadb-x86_64.conf /etc/my.cnf.d/client.cnf /usr/lib/.build-id /usr/lib/.build-id/a5 /usr/lib/.build-id/a5/c17835b924d9830cc6b91764d3c3e80f650d05 /usr/lib64/mysql/libmariadb.so.3 /usr/lib64/mysql/libmysqlclient.so.18 [root@vosprey2 httpd]# cd /usr/lib64/mysql [root@vosprey2 mysql]# ls -la total 424 drwxr-xr-x. 3 root root 4096 Nov 22 10:07 . dr-xr-xr-x. 85 root root 69632 Nov 19 13:33 .. -rw-r--r--. 1 root root 6773 Oct 6 12:18 INFO_BIN -rw-r--r--. 1 root root 172 Sep 25 19:33 INFO_SRC lrwxrwxrwx. 1 root root 15 Oct 6 12:18 libmariadb.so -> libmariadb.so.3 -rwxr-xr-x. 1 root root 338792 Oct 6 12:22 libmariadb.so.3 lrwxrwxrwx. 1 root root 13 Oct 6 12:18 libmysqlclient_r.so -> libmariadb.so lrwxrwxrwx. 1 root root 13 Oct 6 12:18 libmysqlclient.so -> libmariadb.so drwxr-xr-x. 2 root root 4096 Nov 19 13:14 plugin
A re-install of the mariadb-libs package does create the required symbolic link. It does not fix the issue which is unfortunate :-(.
[root@vosprey2 mysql]# dnf reinstall mariadb-libs Last metadata expiration check: 2:42:08 ago on Wed 22 Nov 2017 07:25:33 NZDT. Dependencies resolved. ========================================================================================================= Package Arch Version Repository Size ========================================================================================================= Reinstalling: mariadb-libs x86_64 3:10.2.9-3.fc27 fedora 150 k Transaction Summary ========================================================================================================= Total download size: 150 k Is this ok [y/N]: y Downloading Packages: mariadb-libs-10.2.9-3.fc27.x86_64.rpm 293 kB/s | 150 kB 00:00 --------------------------------------------------------------------------------------------------------- Total 101 kB/s | 150 kB 00:01 Running transaction check Transaction check succeeded. Running transaction test Transaction test succeeded. Running transaction Preparing : 1/1 Reinstalling : mariadb-libs-3:10.2.9-3.fc27.x86_64 1/2 Running scriptlet: mariadb-libs-3:10.2.9-3.fc27.x86_64 1/2 Erasing : mariadb-libs-3:10.2.9-3.fc27.x86_64 2/2 Running scriptlet: mariadb-libs-3:10.2.9-3.fc27.x86_64 2/2 Verifying : mariadb-libs-3:10.2.9-3.fc27.x86_64 1/2 Verifying : mariadb-libs-3:10.2.9-3.fc27.x86_64 2/2 Reinstalled: mariadb-libs.x86_64 3:10.2.9-3.fc27 Complete! [root@vosprey2 mysql]# ls INFO_BIN libmariadb.so libmysqlclient_r.so libmysqlclient.so.18 INFO_SRC libmariadb.so.3 libmysqlclient.so plugin [root@vosprey2 mysql]# ls -la total 424 drwxr-xr-x. 3 root root 4096 Nov 22 10:07 . dr-xr-xr-x. 85 root root 69632 Nov 19 13:33 .. -rw-r--r--. 1 root root 6773 Oct 6 12:18 INFO_BIN -rw-r--r--. 1 root root 172 Sep 25 19:33 INFO_SRC lrwxrwxrwx. 1 root root 15 Oct 6 12:18 libmariadb.so -> libmariadb.so.3 -rwxr-xr-x. 1 root root 338792 Oct 6 12:22 libmariadb.so.3 lrwxrwxrwx. 1 root root 13 Oct 6 12:18 libmysqlclient_r.so -> libmariadb.so lrwxrwxrwx. 1 root root 13 Oct 6 12:18 libmysqlclient.so -> libmariadb.so lrwxrwxrwx. 1 root root 15 Oct 6 12:18 libmysqlclient.so.18 -> libmariadb.so.3 drwxr-xr-x. 2 root root 4096 Nov 19 13:14 plugin [root@vosprey2 mysql]# /usr/lib64/nagios/plugins/check_mysql /usr/lib64/nagios/plugins/check_mysql: error while loading shared libraries: libmysqlclient.so.18: cannot open shared object file: No such file or directory [root@vosprey2 mysql]# cd /etc/ld.so.conf.d [root@vosprey2 ld.so.conf.d]# ls bind99-x86_64.conf kernel-4.13.12-200.fc26.x86_64.conf mariadb-x86_64.conf kernel-4.12.11-300.fc26.x86_64.conf kernel-4.13.12-300.fc27.x86_64.conf [root@vosprey2 ld.so.conf.d]# cat mariadb* /usr/lib64/mysql [root@vosprey2 ld.so.conf.d]# ldconfig [root@vosprey2 ld.so.conf.d]# /usr/lib64/nagios/plugins/check_mysql /usr/lib64/nagios/plugins/check_mysql: error while loading shared libraries: libmysqlclient.so.18: cannot open shared object file: No such file or directory
I will have to yet again write custom plugins to replace the ones that have stopped working.
