2. Configuration

Important Considerations

2.1 General Considerations

You should as a general rule not run the alert_collector_server on a system with both internal and external network interfaces, but on an internal network server. If you must run it on a system with an external interface to the big wide world ensure that when you start the server you specify the ip-address of one of the interfaces connected to the internal network. This will prevent any external traffic from accessing the application.

2.2 Configuration/Installation considerations

The port number and ipaddress you will be starting the server to use need to be known to any remote servers that will be forwarding alerts to this server. Please record them before starting the alert_collector_server process. You will need to configure this information on each server you install the scripts into for alert forwarding.
Do read the entire chapter, there is a very important note on customising the startup script after the dicussion on configuring the main configuration file.

Configuring the toolkit - etc/config.txt file

Under the directory you installed the toolkit to there is a directory etc which contains all the configuration information used by the toolkit in the file config.txt. All the supplied scripts use this file by refering to it using relative paths from where they are installed.

As long as you get this file correct, it can be copied to every other server you install the toolkit on and the supplied scripts to raise/cancel alerts and run the monitoring interfaces (both vt100 and java GUI) will work correctly.
Note: if on the alert collector server you use the generic localhost address to force it to listen on all interfaces you will need to put the appropriate ip-address of the alert collector server into the config.txt SERVER_IPADDR of each remote server that will forward alerts to it rather just just copying the master, so it can find the correct server.

The config.txt file contains the following information (this sample is the configuration from my servers). The fields are explained below.

# ----------------------------------------------------------------------------- # Common to all scripts # ----------------------------------------------------------------------------- PORT=9003 # tcp port number collector server listens on # SERVER_IPADDR=127.0.0.1 # Use localhost address to listen on all interfaces SERVER_IPADDR=169.254.218.183 # ip addr of server with collector server running SERVER_NAME=falcon # ip addr of server with collector server running INSTALLED_DIR="/opt/dickinson/alert_management" # where the binaried reside LOGPATH="/opt/dickinson/alert_management/logs" # where the forwarding log files reside RUNPID_DIR="/opt/dickinson/alert_management/run" # Where pid files are stored # ----------------------------------------------------------------------------- # Used by the supplied scripts for forwarding alerts from batch jobs # ----------------------------------------------------------------------------- ALERTPROG="raise_alert" # supplied program, in $INSTALLED_DIR LOG_FILE="forwarded_alerts.log" # history file for batch alerts generated, in $LOGPATH # ----------------------------------------------------------------------------- # Used by the console application to override server defaults # ----------------------------------------------------------------------------- CONSOLE_LINE_LEN="LINELEN=118" # passed as a parm from console.sh (max 127) CONSOLE_NUM_LINES="NUMLINES=24" # passed as a parm from console.sh (max 99) # CONSOLE_BLINK="ON" # set to OFF is you don't want blinking acked alerts CONSOLE_BLINK="OFF" # set to OFF is you don't want blinking acked alerts on tty # ----------------------------------------------------------------------------- # Used for automation of alert events # Optional (free) plugin for this tool. # ----------------------------------------------------------------------------- AUTOMATION_ENABLED="TRUE" # Change to FALSE if you don't have the plugin AUTOMATION_DIR="/opt/dickinson/alert_management/automation" # Automation tools AUTOMATION_PROG="actioncmd.sh" # Does automation checks and tasks AUTOMATION_TABLE="automation_table"

Common fields

The connon fields block are for variables used by pretty much all of the supplied scripts. Used by the alert raise/cancel scripts and the monitoring scripts to locate the alert_collector server to connect to when they are requested to do anything. Also used by the supplied startup/shutdown scripts.

PORT=nnnn is the port number the alert collector server is to listen for requests on
SERVER_IPADDR=nnn.nnn.nnn.nnn is the ip-address of the interface the server will bind to and listen on. If this is set to 127.0.0.1 then the supplied startup scriot will start the server listening on all interfaces of the machine. This dotted format is used by all the supplied C programs, and the scripts that wrap around them
SERVER_NAME=hostname is the hostname of the machine running the alert collector server. This is used by the java GUI so ensure it is in the hosts file (or DNS) of each remote machine or PC that will be running the java GUI
INSTALLED_DIR="fullpath-no-trailing/" is the directory you installed the toolkit into
LOGPATH="fullpath-no-trailing/" is where you wish the supplied forwarding scripts to place action and error messages. This does not affect where the alert collector server writes log files, that will alsways write into the directory logs/ under the directory you were in when you started the server. This is why you should always use the supplied startup script, and why I recomend you put the forwarding logs in the same place
RUNPID_DIR="fullpath-no-trailing/" is the directory where the pid of the alert collector will be stored. The pis is written here by the supplied startup script and used by the script diring shutdown if a clean shutdown fails and the task must be killed.
This is also used by the sample log tailing script so it knows what it needs to stop on a shutdown.

Alert forwarding parameters

These are used by the supplied scripts that raise and cancel alerts. You should not need to change these. The alertprog must be in the installation directory.

ALERTPROG="raise_alert" is the binary program supplied to provide the interface between your scripts and the alert collector server.
It must be in the installation directory for the supplied scripts to work. If you put it anywhere else you will not be able to use the supplied scripts and will need to refer to the chapter on generating alerts to see how to write your own scripts
LOG_FILE="filenameonly-no-path" is the filename to be used to record alert activity. This is a place where the supplied alert raise/cancel scripts can record activity and errors. It will be in the LOGPATH directory.
Note: You should roll or otherwise manage this log regularly as it will happily grow forever.

Console application parameters

These control the operation of the vt100 alert monitoring interface. As users generally prefer different window sizes and displays these should be set as a generic default for most of the users, or for the console. Users can always customise there own display formats as documented in the chapter on the vt100 interface as needed.

CONSOLE_LINE_LEN="LINELEN=nnn" controls the width of the line displayed, if this is too big for a user the lines may wrap rendering the top alerts displayed off the top of the screen, if too small they will be unable to see some of the information. The LINELEN= bit is required as it is passed that way to the vt100 program. The maximum value is 127.
CONSOLE_NUM_LINES="NUMLINES=nn" controls the number of lines displayed. The maximum is 99
CONSOLE_BLINK="OFF" may be set to ON or OFF. This controls whether acknowledged alerts blink or not. On a black and white display it's hard to distiguish acknowledged alerts from informational ones so you can make them blink if you wish. I likes it at the start but eventually found it annoying so provided (and use) the ability to turn blinking off.

Automation parameters

These are not used by the alert server itself, they are used by the supplied scripts and are expected to be used on the server generating the alerts, not the central server collecting the alerts.

A sample simple automation structure is provided that will allow you to automate the recovery of some alerts at the time they are generated rather than just forwarding an alert to the central monitoring server for somebody else to deal with. The use of this facility is covered in the chapter on automation tools. The parameters here are

AUTOMATION_ENABLED="TRUE" may be TRUE or FALSE. If false no automation will be permitted by the supplied scripts
AUTOMATION_DIR="fullpath-no-trailing/" is the directory the automation samples have been installed into. As shipped this will be the directory automation under the installation directory
AUTOMATION_PROG="actioncmd.sh" should not be changed unless you have replaced it with your own automation scripts. This is the bash script that will be invoked to search for an automation rule match and action it
AUTOMATION_TABLE="automation_table" should not be changed unless you have replaced the actioncmd.sh above with your own automation scripts. This is the ruleset file that contains the automation rules.

Customising the startup script

The supplied startup script will need to be customised. As this is unlikely to be run from the scripts directory as supplied, but more likely to be run from the init.d directory, the startup script does not make any assumptions as to where the config.txt file will actually live. It may even be that you run multiple copies for some reason.

In the installation directory there is a file scripts/start_alertserver.sh that is used to start and stop the alert server task. The first few lines after the comments have explicit pathnames to where the config.txt file is expected to be. If you have installed the toolkit anywhere but /opt/dickinson/alert_management or if you are running multiple copies of the alert collector server on a machine you must customise this.
There is also a start_server_and_tailtask.sh script that I use personally, that starts the server and a tail task against /var/log/messages using the sample log processing scripts (although my message rules are different). You can use this instead of the script above if you will always be using the tail task.
Both these scripts are covered in chapter 3.

Summary

To keep the supplied scripts simple to use they all obtain there information from etc/config.txt under the installation directory. That should be the only file (with the exception of the supplied startup script) you need to customise on each server to use the application.