My Server Health Check Scripts, and config repository

These scripts are a bit of a hybrid. The origional purpose was a server health checking utility which it still performs, but as the data collection is cpu intensive and the processing of that data so cpu intensive I moved all processing of collected results to a seperate server.

The config repository part came about as the data to be processed (from a full server scan anyway) contains detailed listings of all file permissions of every file on the server, plus the contents of selected files from /etc, so on the assumption I would add later checks against files under /etc at some point the scripts also tar up all the files in /etc to be shipped to the processing server (as this includes files such as passwd and shadow some may consider this a security risk); It also collects a full hardware listing (if hardware display packages are installed) plus list of all installed packages for each server. All useful if you need to rebuild a server.

However the only implemented function is still the primary designed function of performing server health checks, it just provides a lot more server information from each server to be pushed up to the processing server that I may one day process.

All reporting against the collected information should be done on a dedicated machine with a web server running on it.
The reports produced are html pages starting with an overview page of all servers status and total exception counts and drill down to details of the exceptions found for each server.

• it checks for bad file permissions and ownership on all files to the recursion depth permitted by the operating system (although a max directory depth can be passed as a parameter to the collection script if you want to limit the checks)
• it checks all open TCP and UDP ports against ports expected to be open (tcp/tcp6/udp/udp6/raw/raw6) including listing the process that actually has the port open. Also reports on any custom file entries expecting a port open that may be obsolete (if the port was not actually in use at the time the data was collected)
• checks server firewall rules to report on any open ports in the firewall rules and alert on those that have no matching listening port on the server (locates obsolete rules)
• checks various security files in /etc for sensible default settings
• checks all suid files on the system to ensure no new ones appear unnoticed
• special checks implemented to ensure webserver files are read-only (optional)
• per server customisation files to define expected server environment
• and does a lot more most auditing tools do not

The latest dowload tarball (version 0.20) (released 30 September 2020) is available if you are interested, there is a doc folder explaining usage; but basically run the data collection script on all servers and copy all the *txt files produced to a reporting server, and run the processing script against the directory containing the text files, then point your web browser at the results/index.html file to see all the reports. A copy of the documentation for the toolkit is available online here, this documentation is for the older version 0.19 and is unlikely to be as up to date as the one packaged in the tar.gz file.

Note: whenever upgrading between versions you should always run the new collection script on all servers beefore attempting the processing script as data collection parameters may change between versions.

No matter how tightly you think you have locked down your server(s) I would be supprised if there were not at least hundreds if not thousands of exceptions found when using these scripts for the first time.

After processing there is a menu for all servers with a summary of insecurities found, permitting drill-down to each individual servers summary results be check section.
From version 0.08 onward this page also shows (dates in yellow) servers that have new collected data files available since the last processing was done for the server; which can be also be displayed with the '--checkchanged=list' option and processed with the '--checkchanged=process' option
From version 0.11 onward servers that have not had fresh data collected in the last 14 days are alerted on, as you should have automated collection/processing of these checks. From version 0.12 onward the 14 days limit has been made customisable per server and shown on the main index page.
Below is an example of the 0.14 index page.

From each individual server summary page you can see a summary of insecurities found for each section of the checks, and drill-down to a detailed report for each section.

Please note this is being actively maintained, check back often for changes.

It should also be noted that as a general rule each new version will take longer to run than the previous version as additional checks are added; although obviously this depends on what your servers are used for (a web server with 200,000+ files to be checked obviously takes longer than a tftpboot server with 25,000 files to be checked so you probably won't notice much difference on servers with a large number of files).

You should always use the latest version for bug-fixes and enhancements. Upgrading to a new processing version will always require a full re-processing of all servers (although existing collector files can be used for the re-processing although some information for a server will be unavailable in that case as backward compatibility for versions prior to 0.13 has ended and there is also the possibility of false alerts due to data expected from 0.13 not being collected by older collector versions).