IMPORTANT NOTES
===============
Currently only Debian (native) and Rhel family (Alma, Rocky, CentOS, Fedora etc)
are supported, partial support for SunOS (openindiana) are also implemented.
Do not expect these scripts to work properly on other servers such as Ubuntu;
because every distro puts some key files in different places and I only test
on OS releases I run.

Known Bugs with the current release that need fixing
====================================================
* when processing a capture with a large number of files _SOMETIMES_ an error is thrown
  "bin/process_server_details.sh: line 4422: ----------------------------------: command not found"
  which is not the contents of line 4422. A rerun with no changes does not have the same issue.
  Ocurrence is completely random so hard to track down. Does not affect any results produced
* similar to above issue, the below has been seen (with archive flag not selected)
   11-02-2026 11:35 - ...DONE, completed processing server xxxxxxx
   test/process_server_details.sh: line 6406: mp}.tar.gz: command not found
   11-02-2026 11:35 - ...DONE, report archive created as /reports_.tar.gz
   test/process_server_details.sh: line 6417: syntax error near unexpected token `fi'
   test/process_server_details.sh: line 6417: `fi'
  which also cannot be repeated if just rerun without any changes (just up-arrow and enter for rerun)
- the above two may be memory corruption starting on my test server ???, it has 32Gb of memory
  so unlikely to be a limitation on a small 6000 bash script there.

* if process one server is passed a servername no data file exists for it does not clear the
  lockfile; need to add that one line between the error msg and the script exit (minor, no hurry)
* if you ^C or kill a running processing script the lockfile remains, you must use the
  processing script --clearlock option to remove it.  Maybe one day implement a trap handler.
* Not a bug but take note:
  Nowhere in the directory path of the processing script can there be an underscore</b> ( _ )
  character used, the underscore character is used by the script for parsing and having it
  in the directory path will cause problems. Not really an issue, just do not do it
* Not a bug but take note:
  In customisation files do not put comments on the end of a data line unless it is
  part of the data line syntax and it has a field for them as they will be treated as
  part of the data. Comments are only lines with # in column 1
* >>> Updates to support data collected from OpenIndiana/SunOS are still in progress
  and incomplete. 

Current Limitations with the current release (some may consider these bugs)
===========================================================================
* performance - on an 8cpu system with 32Gb of memory it takes 2hrs+14mins to process
  634902 file checks (a single server full scan data collection), so using the default
  full scan data collection should be done infrequently
  (refer to the '--scanlevel=N' directive to limit scan data collection)
* reprocessing logic skip, when a single server is selected for processing and the
  processing script version has changed all servers data will need to be re-processed
  also (user is prompted) to ensure everything is processed with the latest version.
  That is skipped if processing only changed files as that is intended for daily batch so
  prompts are not possible and avoids a batch job suddenly taking hours instead of minutes.
* cron job checks - only cron jobs have security permissions checked, not anacron files
  or any queued 'at' jobs
* cron job checks - stacked commands are able to be tested if seperated by ';' '&&' '|',
  the '||' syntax is not supported yet.
* cron job checks - checks on what commands are run in a cron job
  where a cron job uses a system command (echo, cd, php) rather than
  a discrete script an alert will normally be raised as system commands are normally owned
  by root and not the owner of the cron job. A list of commands considered to be non-disruptive
  can be used to suppress alerts for things like 'echo', 'php', 'bash' etc. to allow those
  to be used without alerting; although commands such as 'ls', 'cd', 'find' etc. will
  always alert as there is no way of determing what directory a stacked crontab command
  line has obtained if it is using combinations of these. These are 'hard coded' in the
  collection script (as the collection script can check the files on the server it is
  being run on [the processing script may be on a different server so cannot]).
  If you wish to alter the defaults search in the collection script for the variables
  CRON_CMD_IGNORE_LIST, CRON_CMD_SHELL_LIST, CRON_CMD_FATAL_LIST strings and update those.
  Note: for files of this type not common across all servers a custom file entry can be added
  for use by the processing script  to allow them to be OK if they are owned by root
  (as all system utilities must be owned by root)
* Does not yet distinguish between a firewall inbound rule and an outbound rule. It WILL in a
  future release as I need that but currently if you have an outbound rule for a port
  number and that port is also listening on the local server it treats it as expected
  when it should not do; likewise if no listening port is on the server for it it
  will raise an error expecting one. Temporarily the toolkit allows a 
  TCP|UDP_OUTBOUND_SUPPRESS parameter to suppress the later case but thats just a
  temporary workaround
  No attempt is made to follow firewall chains or determine zones,
  any rule to open a port is considered an open port. This is needed to cleanly fix the above which
  is why fixing the above will take some time
* server firewall rule checks - currently only handles iptables or netfilter (iptables and nft commands)
  It does not yet (and may never) support IPF (ipfilter) firewall tables  
  needed for the SunOS (openindiana) checks I have started implementing. For now too complicated,
  especially if running zones need to be taken into account, no free time.
* suid file check has a SunOS/OpenIndiana specific issue, suid files are created at boot time
  with names '/proc/NNN/a.out' (NNN being the pid that changes every reboot). This seems to
  be sendmail that runs a process as root and another as smmsp (the later using a suid a.out of root)
  on openindiana. So a TODO in a later release is to add extra processing for suid files under
  /proc/NNN/a.out to check what program is running with pid NNN to see if it is a process allowed
  to do that...  as a 'ps' list is in the capture file that is possible when I get time;
  but for now they will always alert as unexpected file with suid set.

Irritations caused by environments
==================================
* servers running NetworkManager will have firewall ports opened that users may not expect,
  This is not an issue with this toolkit but
  with a lack of control over what NetworkManager decides to open; prior to version 0.12 any
  firewall accept rule for a port not in use alerted as an obsolete firewall rule, from
  0.12 onward a custom file rule can be used to identify and downgrade alerts to warnings for ports opened
  by NetworkManager rather than ports you explicitly configured.
  Examples: 'firewall-cmd --list-services' shows 'cockpit dhcpv6-client dns http ntp ssh' and
  'firewall-cmd --list-ports' does not show the following ports,
  but the following ports have firewall rules opening the ports, udp 67(/usr/lib/firewalld/services/dhcp.xml),
  udp 68(/usr/lib/firewalld/services/RH-Satellite-6.xml),udp 69(/usr/lib/firewalld/services/tftp.xml);
  _and worse_ some processes that use dynamic ports will occasionally use those ports so unexpected apps are exposed
  Also Fedora is a pain, at some point (in F33?) cupsd ('ipp' and 'ipp-client' firewalld rules) repo supplied
  firewalld services changed from opening ports 631 on tcp and tcpv6 to opening on
  631 tcp and tcpv6 plus udp, but cups does not use the udp port so this toolkit
  will correctly raise an alert for a open firewall port that is not in use but you
  are at the mercy of random changes like that in ditributions (I have overrides for almost every
  firewalld service I use to prevent that now but it will affect all other users)
* unknown port openers. While the scripts do their best to identify what had a tcp or udp port
  open (by going as far if needed to parse the output of the recorded 'ps -ef' to locate the
  actual process running with the pid reported by 'netstat -anp') in some cases a 'netstat -anp'
  will just report the pid of a process as '-' (single dash) making it impossible to
  identify what is opening a port; a best guess can be made by matching a port with what is
  in /etc/services which may not be correct and in many cases listening ports are in ranges
  outside what is in /etc/services anyway.
* I always use SSH sessiosn with X11 forwarding, that opens a port range only when a 
  ssh session is active (shows as a process like 'sshd: mark@pts/0' so I have an include file
  for servers that provide X11 forwarding in the examples that allows the ports to be open
  in the firewall but also allows it to be OK is they are not in use that you will have to
  customise for your own port ranges if you use X11 forwarding in your own environment
