Two scripts are provided for use by batch shell scripts
to make it easier to raise and cancel alerts. These
are in the scripts folder under the installation directory
and are raise_alert.sh and end_alert.sh.
A third is provided, clear_all_alerts.sh that should be used as one of
your rc scripts during a machine shutdown. It clears all alerts for the host,
as if the host is supposed to be down there is no point in having alerts left
in the system.
They use a configuration file in the toolkit installation etc directory to determine where to send the alerts so the users don't need to know. You as the administrator should ensure the configuration file config.txt is correct for your site(s).
The syntax is:
raise_alert.sh key "text" [ critical | warning [AUTOMATE parms] ] end_alert.sh key ["text"] clear_all_alerts.sh |
Note that when raising an alert you can specify whether it is a critical alert or a warning alert. If you do not provide the third parameter the script defaults to critical.
To use the AUTOMATE option you must have enables and configured the automation toolkit in the config.txt file and added an automation rule for the alert event. Any parameters after the AUTOMATE keyword will be passed to the script you have defined in the automation rule.
Note: The
Example:
raise_alert.sh FS_var "Filesystem /var > 90% full" critical (fix the problem) end_alert.sh FS_var "/var now < 90% full" |
raise_alert.sh will generate critical or warning events,
end_alert.sh generates a harmless to cancel any previous,
clear_all_alerts.sh deletes all alerts generated by the server
alert.
Raising the alerts manually allows two additional message types to be forwarded. These are informational and log-only messages. The supplied scripts do not allow that as you should't be annoying operations with informational messages they are to ignore, or they might start ignoring them all; and log-only is something the operators would never see so why raise it from batch jobs for an error condition, raise a visible alert.
There are a few key points to be aware of if you want to generate informational messages
so don't use informational messages for anything you want the people monitoring the alert screens to actually see.
The log-only type messages may be usefull for hearbeat logging, where
each remote machine can run a script to log a message every hour or so. You could
write a script to check you get them on the main server if you wanted.
However as a general rule there's not much point as I'm sure the users will
let you know if an entire machine goes down, this toolkit isn't really
designed for that.
To raise alerts manually with the supplied program the syntax is:
raise_alert portnum ipaddr \
critical|warning|harmless|info|logonly|clearall \
key "text"
Example: raise_alert 9003 127.0.0.1 warning TEST_1 "Test Only"
|
The key should be unique for each alert
from a server. Not only does the server overwrite
duplicate keys but the key is used to delete alerts.
The server itself uses the generating hostname as
part of the key, so the same key can be used on
different hosts without conflict (ie: you can copy
your batch jobs between systems without any issues).
Important: harmless should only be used to cancel any
messages previously raised, use info for non-alert
messages, being aware they may not display as noted above.
Harmless messages never display, they are specifically for cancelling prior
alerts.
The clearall option should only be used from an rc script during server
shutdown.
There are some sample scripts provided with the toolkit that demonstrate the use of raising and cancelling alerts. These are covered in the next chapter.