\nThese are the issues I had personally, as every user has a different setup your distance may vary :-). They are covered in detail below.\n<\/p>\n
\nOne of my VM hosts has a full Gnome install, and it popped up the Fedora27 now available upgrade window. So I used it, after the reboot it said it was upgrading but it restarted still on F26… and popped up the upgrade window that I used again, after the reboot it said it was upgrading and after the reboot it was still on F26.\n<\/p>\n
\nSo I stopped wasting time and used the command line method with the dnf system-upgrade commands, that sucessfully upgraded to F27.\n<\/p>\n
\nI had network issues on one host<\/em><\/b>, so I disabled NetworkManager, but with NetworkManager disabled the blasted thing starts always anyway.\n<\/p>\n \nThat does not appear to be causing my issue however as my network-scripts are configured not to use NetworkManager and I have “network” enabled; but it does not appear to be working correctly on that one host<\/em><\/b>. And doing a systemctl restart network (which I would hope leaves NetworkManager out of it) does not fix the issue.\n<\/p>\n \nWorse, it is not consistently possible to manually correct the blasted thing. Using “route add default gw 192.168.1.1” worked after the fourth reboot; prior to that it made no difference to the routing table.\n<\/p>\n \nThe \/etc\/resolv.conf has to be manually edited to add the third DSN nameserver entry.\n<\/p>\n \nOn a second host that has been upgraded networking is setup almost correctly, it adds two entries for the default route instead of only one expected but it seems to work OK.\n<\/p>\n \nThe only difference between the configuration files on the two hosts is the interface names and the UUID\/HWADDR of the entries so both servers should have the same result, but they don’t.\n<\/p>\n \nNo problems running the normal mysql_upgrade script but there are issues with bacula if you are running mariadb in safe mode.\n<\/p>\n \nThe upgrade script for mariadb needs work. In the file \/usr\/libexec\/bacula\/update_mysql_tables you must search for the line “UPDATE Version SET VersionId=16;” and wrap two lines about it thus…\n<\/p>\n \nThis avoids the error “ERROR 1175 (HY000): You are using safe update mode and you tried to update a table without a WHERE that uses a KEY column” which prevents the version number being updated.\n<\/p>\n \nHowever it remained broken<\/b>. It looks like the upgrade is a no-go, and I will have to try recreating all the tables from scratch, Backups running after the mysql upgrade scripts completed sucessfully stil failed with database errors…\n<\/p>\n \nThat did actually supprise me as the upgrade scripts for bacula normally work; but not this time.\n<\/p>\n \nThe solution<\/em><\/b> unfortunately is to stop bacula-dir and bacula-sd, delete all the tables in the bacula database and recreate it from scratch using \/usr\/libexec\/bacula\/make_mysql_tables; plus you must also delete everything from \/var\/spool\/bacula \nThen you can restart bacula-sd and bacula-dir. And unfortunately as you no longer have any backups you must then run backups for each of your servers which will be full backups on the first run for each. You could wait until the backups are scheduled to run but as they will be full backups on the first run it is probably better to manually do them at a time when you know network congestion is at a minimum.\n<\/p>\n \nBut at least backups are working again.\n<\/p>\n \nAs always, if you need to share files between webapps and scripts, vi \/usr\/lib\/systemd\/system\/httpd.service and change PrivateTmp=true to PrivateTmp=false.\n<\/p>\n \nWhile mariadb is running OK and the mysql_upgrade script had no problems, there appears to be something wrong with the mariadb-libs packaging, certainly not all the files rpm -ql showed for the package existed after the upgrade to F27.\n<\/p>\n \nThis issue was found in investigating issues with the NRPE mysql plugin so refer to that section for details on the issues with mariadb-libs. At this point in time I have not found a resolution.\n<\/p>\n \nNote that a “dnf reinstall mariadb-libs” did create all the expected files, but even though the ld.conf.so.d file for mysql is correct even after rerunning ldconfig the mysql library files are not found by programs wanting to use them; at least not found by th nrpe plugin which is what I am trying to get working first.\n<\/p>\n \nAs always the systemd startup scripts for NRPE have to be edited to turn off the requirement to use SSL, thats no real issue as I have gotten used to that.\n<\/p>\n \nAnd as always more of the supplied plugins have stopped working. The HTTP and MYSQL ones have stopped working this time.\n<\/p>\n \nThe HTTP plugin is triggering error 400’s from the server for no reason that I can determine. As seen in the log clip the server responds OK to non-nrpe requests.\n<\/p>\n \nUsing telnet to “GET \/ HTTP\/1.0” and “GET \/ HTTP\/1.1” manually also get an error 400 (although using just “GET \/” works OK); so pherhaps a behaviour change in HTTP itself… although Firefox has no issue retrieving the page with the full string.\n<\/p>\n \nRPM says that file is installed as part of mariadb-libs… which is installed but that file is not.\n<\/p>\n \nA re-install of the mariadb-libs package does create the required symbolic link. It does not fix the issue<\/b> which is unfortunate :-(.\n<\/p>\n \nI will have to yet again write custom plugins to replace the ones that have stopped working.<\/p>\n","protected":false},"excerpt":{"rendered":" These are the issues I had personally, as every user has a different setup your distance may vary :-). They are covered in detail below. Desktop upgrade does not work, no big deal (workaround works) Network manager misbehaves (unsolved) Bacula … Continue reading \r\n[root@vmhost3 ~]# systemctl status NetworkManager\r\n? NetworkManager.service - Network Manager\r\n Loaded: loaded (\/usr\/lib\/systemd\/system\/NetworkManager.service; disabled; vendor preset: enabled)\r\n Active: active (running) since Sun 2017-11-19 10:13:36 NZDT; 2min 31s ago\r\n Docs: man:NetworkManager(8)\r\n Main PID: 1323 (NetworkManager)\r\n Tasks: 3 (limit: 4915)\r\n CGroup: \/system.slice\/NetworkManager.service\r\n ??1323 \/usr\/sbin\/NetworkManager --no-daemon\r\n\r\nNov 19 10:14:23 vmhost3 NetworkManager[1323]:
\r\n[root@vmhost3 network-scripts]# grep -i DNS ifcfg*\r\nifcfg-br0:DNS2=\"192.168.1.179\"\r\nifcfg-br0:DNS1=\"192.168.1.181\"\r\nifcfg-br0:DNS3=\"192.168.1.1\"\r\nifcfg-enp2s0:IPV6_PEERDNS=\"no\"\r\nifcfg-enp2s0:PEERDNS=\"yes\"\r\nifcfg-enp2s0:IPV6_PEERDNS=\"yes\"\r\n[root@vmhost3 network-scripts]# grep -i GATE ifcfg*\r\nifcfg-br0:GATEWAY0=\"192.168.1.1\"\r\nifcfg-lo:# If you're having problems with gated making 127.0.0.0\/8 a martian,\r\n[root@vmhost3 network-scripts]# cat \/etc\/resolv.conf\r\n# Generated by NetworkManager\r\nnameserver 192.168.1.181\r\nnameserver 192.168.1.179\r\n[root@vmhost3 network-scripts]# route -n\r\nKernel IP routing table\r\nDestination Gateway Genmask Flags Metric Ref Use Iface\r\n169.254.0.0 0.0.0.0 255.255.0.0 U 1003 0 0 br0\r\n192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 br0\r\n192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0\r\n[root@vmhost3 network-scripts]#\r\n[root@vmhost3 network-scripts]# uname -a\r\nLinux vmhost3 4.13.12-300.fc27.x86_64 #1 SMP Wed Nov 8 16:38:01 UTC 2017 x86_64 x86_64 x86_64 GNU\/Linux\r\n<\/pre>\n
\r\n[root@vmhost1 network-scripts]# grep -i DNS ifcfg*\r\nifcfg-br0:DNS1=\"192.168.1.181\"\r\nifcfg-br0:DNS2=\"192.168.1.179\"\r\nifcfg-br0:DNS3=\"192.168.1.1\"\r\nifcfg-em1:IPV6_PEERDNS=\"no\"\r\nifcfg-em1:PEERDNS=\"yes\"\r\n[root@vmhost1 network-scripts]# grep -i GATE ifcfg*\r\nifcfg-br0:GATEWAY0=\"192.168.1.1\"\r\nifcfg-lo:# If you're having problems with gated making 127.0.0.0\/8 a martian,\r\n[root@vmhost1 network-scripts]# cat \/etc\/resolv.conf\r\n# Generated by NetworkManager\r\nnameserver 192.168.1.181\r\nnameserver 192.168.1.179\r\nnameserver 192.168.1.1\r\n[root@vmhost1 network-scripts]# route -n\r\nKernel IP routing table\r\nDestination Gateway Genmask Flags Metric Ref Use Iface\r\n0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 br0\r\n0.0.0.0 192.168.1.1 0.0.0.0 UG 425 0 0 br0\r\n192.168.1.0 0.0.0.0 255.255.255.0 U 425 0 0 br0\r\n[root@vmhost1 network-scripts]# uname -a\r\nLinux vmhost1 4.13.12-300.fc27.x86_64 #1 SMP Wed Nov 8 16:38:01 UTC 2017 x86_64 x86_64 x86_64 GNU\/Linux\r\n[root@vmhost1 network-scripts]#\r\n<\/pre>\n
Bacula Director Database Update<\/h2>\n
\r\nSET SQL_SAFE_UPDATES=0;\r\nUPDATE Version SET VersionId=16;\r\nSET SQL_SAFE_UPDATES=1;\r\n<\/pre>\n
\r\n21-Nov 17:00 bacula-dir JobId 0: Fatal error: sql_create.c:84 Create DB Job record INSERT INTO Job (Job,Name,Type,Level,JobStatus,SchedTime,JobTDate,ClientId,Comment) VALUES ('BackupVMhost3.2017-11-21_17.00.00_12','BackupVMhost3','B','I','C','2017-11-21 17:00:00',1511236800,11,'') failed. ERR=Field 'StartTime' doesn't have a default value\r\n21-Nov 17:00 bacula-dir JobId 0: Fatal error: sql_create.c:84 Create DB Job record INSERT INTO Job (Job,Name,Type,Level,JobStatus,SchedTime,JobTDate,ClientId,Comment) VALUES ('BackupPuppet.2017-11-21_17.00.01_13','BackupPuppet','B','I','C','2017-11-21 17:00:01',1511236801,13,'') failed. ERR=Field 'StartTime' doesn't have a default value\r\n21-Nov 23:55 bacula-dir JobId 0: Fatal error: sql_create.c:84 Create DB Job record INSERT INTO Job (Job,Name,Type,Level,JobStatus,SchedTime,JobTDate,ClientId,Comment) VALUES ('BackupCatalog.2017-11-21_23.55.00_14','BackupCatalog','B','F','C','2017-11-21 23:55:00',1511261700,7,'') failed. ERR=Field 'StartTime' doesn't have a default value\r\n<\/pre>\n
\n<\/e><\/p>\nThe HTTP service<\/h2>\n
Mariadb library issues<\/h2>\n
NRPE plugins<\/h2>\n
The HTTP plugin<\/h3>\n
\r\n192.168.1.170 - - [21\/Nov\/2017:09:47:20 +1300] \"GET \/ HTTP\/1.0\" 400 226 \"-\" \"check_http\/v2.2.1 (nagios-plugins 2.2.1)\"\r\n192.168.1.170 - - [21\/Nov\/2017:09:52:20 +1300] \"GET \/ HTTP\/1.0\" 400 226 \"-\" \"check_http\/v2.2.1 (nagios-plugins 2.2.1)\"\r\n192.168.1.179 - - [21\/Nov\/2017:09:54:34 +1300] \"GET \/ HTTP\/1.1\" 200 2285 \"-\" \"Mozilla\/5.0 (X11; Fedora; Linux x86_64; rv:57.0) Gecko\/20100101 Firefox\/57.0\"\r\n<\/pre>\n
The MYSQL plugin, possibly an issue with mariadb packages and not the plugin ?<\/h3>\n
\r\n[root@vosprey2 httpd]# \/usr\/lib64\/nagios\/plugins\/check_mysql\r\n\/usr\/lib64\/nagios\/plugins\/check_mysql: error while loading shared libraries: libmysqlclient.so.18: cannot open shared object file: No such file or directory\r\n<\/pre>\n
\r\n[root@vosprey2 httpd]# dnf provides libmysqlclient.so.18\r\nLast metadata expiration check: 0:17:01 ago on Tue 21 Nov 2017 10:22:24 NZDT.\r\nmariadb-libs-3:10.2.9-3.fc27.i686 : The shared libraries required for MariaDB\/MySQL clients\r\nRepo : fedora\r\nMatched from:\r\nProvide : libmysqlclient.so.18\r\n\r\n[root@vosprey2 httpd]# rpm -qa | grep -i mariadb-libs\r\nmariadb-libs-10.2.9-3.fc27.x86_64\r\n\r\n[root@vosprey2 httpd]# rpm -ql mariadb-libs\r\n\/etc\/ld.so.conf.d\/mariadb-x86_64.conf\r\n\/etc\/my.cnf.d\/client.cnf\r\n\/usr\/lib\/.build-id\r\n\/usr\/lib\/.build-id\/a5\r\n\/usr\/lib\/.build-id\/a5\/c17835b924d9830cc6b91764d3c3e80f650d05\r\n\/usr\/lib64\/mysql\/libmariadb.so.3\r\n\/usr\/lib64\/mysql\/libmysqlclient.so.18\r\n\r\n[root@vosprey2 httpd]# cd \/usr\/lib64\/mysql\r\n[root@vosprey2 mysql]# ls -la\r\ntotal 424\r\ndrwxr-xr-x. 3 root root 4096 Nov 22 10:07 .\r\ndr-xr-xr-x. 85 root root 69632 Nov 19 13:33 ..\r\n-rw-r--r--. 1 root root 6773 Oct 6 12:18 INFO_BIN\r\n-rw-r--r--. 1 root root 172 Sep 25 19:33 INFO_SRC\r\nlrwxrwxrwx. 1 root root 15 Oct 6 12:18 libmariadb.so -> libmariadb.so.3\r\n-rwxr-xr-x. 1 root root 338792 Oct 6 12:22 libmariadb.so.3\r\nlrwxrwxrwx. 1 root root 13 Oct 6 12:18 libmysqlclient_r.so -> libmariadb.so\r\nlrwxrwxrwx. 1 root root 13 Oct 6 12:18 libmysqlclient.so -> libmariadb.so\r\ndrwxr-xr-x. 2 root root 4096 Nov 19 13:14 plugin\r\n<\/pre>\n
\r\n\r\n[root@vosprey2 mysql]# dnf reinstall mariadb-libs\r\nLast metadata expiration check: 2:42:08 ago on Wed 22 Nov 2017 07:25:33 NZDT.\r\nDependencies resolved.\r\n=========================================================================================================\r\n Package Arch Version Repository Size\r\n=========================================================================================================\r\nReinstalling:\r\n mariadb-libs x86_64 3:10.2.9-3.fc27 fedora 150 k\r\n\r\nTransaction Summary\r\n=========================================================================================================\r\n\r\nTotal download size: 150 k\r\nIs this ok [y\/N]: y\r\nDownloading Packages:\r\nmariadb-libs-10.2.9-3.fc27.x86_64.rpm 293 kB\/s | 150 kB 00:00 \r\n---------------------------------------------------------------------------------------------------------\r\nTotal 101 kB\/s | 150 kB 00:01 \r\nRunning transaction check\r\nTransaction check succeeded.\r\nRunning transaction test\r\nTransaction test succeeded.\r\nRunning transaction\r\n Preparing : 1\/1 \r\n Reinstalling : mariadb-libs-3:10.2.9-3.fc27.x86_64 1\/2 \r\n Running scriptlet: mariadb-libs-3:10.2.9-3.fc27.x86_64 1\/2 \r\n Erasing : mariadb-libs-3:10.2.9-3.fc27.x86_64 2\/2 \r\n Running scriptlet: mariadb-libs-3:10.2.9-3.fc27.x86_64 2\/2 \r\n Verifying : mariadb-libs-3:10.2.9-3.fc27.x86_64 1\/2 \r\n Verifying : mariadb-libs-3:10.2.9-3.fc27.x86_64 2\/2 \r\n\r\nReinstalled:\r\n mariadb-libs.x86_64 3:10.2.9-3.fc27 \r\n\r\nComplete!\r\n[root@vosprey2 mysql]# ls\r\nINFO_BIN libmariadb.so libmysqlclient_r.so libmysqlclient.so.18\r\nINFO_SRC libmariadb.so.3 libmysqlclient.so plugin\r\n[root@vosprey2 mysql]# ls -la\r\ntotal 424\r\ndrwxr-xr-x. 3 root root 4096 Nov 22 10:07 .\r\ndr-xr-xr-x. 85 root root 69632 Nov 19 13:33 ..\r\n-rw-r--r--. 1 root root 6773 Oct 6 12:18 INFO_BIN\r\n-rw-r--r--. 1 root root 172 Sep 25 19:33 INFO_SRC\r\nlrwxrwxrwx. 1 root root 15 Oct 6 12:18 libmariadb.so -> libmariadb.so.3\r\n-rwxr-xr-x. 1 root root 338792 Oct 6 12:22 libmariadb.so.3\r\nlrwxrwxrwx. 1 root root 13 Oct 6 12:18 libmysqlclient_r.so -> libmariadb.so\r\nlrwxrwxrwx. 1 root root 13 Oct 6 12:18 libmysqlclient.so -> libmariadb.so\r\nlrwxrwxrwx. 1 root root 15 Oct 6 12:18 libmysqlclient.so.18 -> libmariadb.so.3\r\ndrwxr-xr-x. 2 root root 4096 Nov 19 13:14 plugin\r\n[root@vosprey2 mysql]# \/usr\/lib64\/nagios\/plugins\/check_mysql\r\n\/usr\/lib64\/nagios\/plugins\/check_mysql: error while loading shared libraries: libmysqlclient.so.18: cannot open shared object file: No such file or directory\r\n\r\n[root@vosprey2 mysql]# cd \/etc\/ld.so.conf.d\r\n[root@vosprey2 ld.so.conf.d]# ls\r\nbind99-x86_64.conf kernel-4.13.12-200.fc26.x86_64.conf mariadb-x86_64.conf\r\nkernel-4.12.11-300.fc26.x86_64.conf kernel-4.13.12-300.fc27.x86_64.conf\r\n[root@vosprey2 ld.so.conf.d]# cat mariadb*\r\n\/usr\/lib64\/mysql\r\n[root@vosprey2 ld.so.conf.d]# ldconfig\r\n[root@vosprey2 ld.so.conf.d]# \/usr\/lib64\/nagios\/plugins\/check_mysql\r\n\/usr\/lib64\/nagios\/plugins\/check_mysql: error while loading shared libraries: libmysqlclient.so.18: cannot open shared object file: No such file or directory\r\n<\/pre>\n