So you need to start with something like
timelapseinlog.pl
A utility script that can carry out a time-based filter on log files with various timestamp formats
#!/usr/bin/perl -ws
# This script parse logfiles for a specific period of time
sub usage {
printf "Usage: %s -s=<start time> [-e=<end time>] <logfile>\n";
die $_[0] if $_[0];
exit 0;
}
use Date::Parse;
usage "No start time submited" unless $s;
my $startim=str2time($s) or die;
my $endtim=str2time($e) if $e;
$endtim=time() unless $e;
usage "Logfile not submited" unless $ARGV[0];
open my $in, "<" . $ARGV[0] or usage "Can't open '$ARGV[0]' for reading";
$_=<$in>;
exit unless $_; # empty file
# Determining regular expression, depending on log format
my $logre=qr{^(\S{3}\s+\d{1,2}\s+(\d{2}:){2}\d+)};
$logre=qr{^[^\[]*\[(\d+/\S+/(\d+:){3}\d+\s\+\d+)\]} unless /$logre/;
while (<$in>) {
/$logre/ && do {
my $ltim=str2time($1);
print if $endtim >= $ltim && $ltim >= $startim;
};
};
user_alert.conf
this is the custom user alert that looks for logged events
alert syslog_status_trigger_action "your critical message" {
exec command="/shared/trigger_action.sh";
exec command="/bin/logger -p local0.info 'action_triggered'"
}
Then in your bash script trigger_action.sh:
#!/bin/bash
TRIGGER_LEVEL=3
if [ ! -e /tmp/trigger_action_count ]; then
# filter ltm1 and ltm into a single file so we don't miss events around log-rotation
grep 'trigger_action' /var/log/ltm.1 > /tmp/trigger_action_count
grep 'trigger_action' /var/log/ltm >> /tmp/trigger_action_count
# set your time period below - I have chosen 10 minutes
COUNT=$(/shared/timelapseinlog.pl -s=`/bin/date -d'now-10 minutes' +%H:%M` /tmp/trigger_action_count | wc -l)
rm -f /tmp/trigger_action_count
if [ -p $NP ] && [ "$COUNT" -gt "$TRIGGER_LEVEL" ]; then
# do something here
fi
fi
Note that there are some things that will not run in the alertd SELinux context (tcpdump, for example).
You may need to run a script on startup that waits on a named pipe, and then the above trigger script signals using the named pipe.
Nimbostratus
