Forum Discussion

Arron_1084's avatar
Arron_1084
Icon for Nimbostratus rankNimbostratus
Jun 04, 2014

Stacked monitoring?

Hello all, I'm fairly sure someone on here has done this before and am hoping its fairly easy to implement. I'm trying to use the LTM's monitors as a way to take offline non-functioning web servers. I've built many monitors in the past to do simple website checks, however the website I'm tasked with now has multiple sub-components that all must be online and available.

 

Systems:

 

Web tier: Has a http/s get monitor looking for specific text that is not application driven (simple are you up) Application tier: Has multiple pools created per service using a http post monitor to check on various web services components as well as LDAP monitors for other components that make up the complete application.

 

The interaction between the two tiers is each web server is configured to connect to the various VIP's that are tied to the application tier web services.

 

I need the ability to take the front end web server offline and there for cause a global failover via a GTM if any of the sub components OR the web tier is down.

 

Is there any sort of monitor that I can put in place on the web server pool that can also get the status of the other sub components without having to perform an additional check on them? I already know I can perform an additional check via using the alias address, however if I do it in this manner the service is being checked twice, once at the individual pool for the web service and an additional check against the web tier pool. I was hoping there was a way to have a monitor that can just read the existing status that the LTM has for the pool.

 

Thank you all in advanced

 

8 Replies

  • Hi!

    How about using an external monitor and use tmsh to get the status of the pools?

    tmsh show ltm pool PatrikIIS-8080-8081_pool field-fmt | grep status.availability-state
        status.availability-state available
    

    /Patrik

  • Hi Patrik,

    Thanks for the reply. I've been playing with a bash script to do this but am having some problems that maybe someone here could help with.

    "mypool1" is offline, "mypool2" is online. When I run this script with only mypool1 the result I get is 0 (offline) and when I run it with only mypool2 I get 1 (online) and when I run it with both commands I get 1 (online) when I would expect it to be 0.

    If I run those commands by themselves for mypool1 I get the text "offline" (expected) and with mypool2 it doesn't return any text (expected).

    !/bin/sh
    
    pidfile="/var/run/pinger.$1..$2.pid"
    
    if [ -f $pidfile ]
    then
       kill -9 `cat $pidfile` > /dev/null 2>&1
    fi
    
    echo "$$" > $pidfile
    
    node_ip=`echo $1 | sed 's/::ffff://'`
    
    tmsh show ltm pool Mypool1_Stg_pool field-fmt | grep -o offline > /dev/null
    tmsh show ltm pool Mypool2_Stg_pool field-fmt | grep -o offline > /dev/null
    
    status=$?
    
    echo "$status"
    
  • Hi!

     

    $? would give the status of the last command so you need to test them independently.

     

    tmsh show ltm pool Mypool1_Stg_pool field-fmt | grep -o offline > /dev/null
    tmsh show ltm pool Mypool2_Stg_pool field-fmt | grep -o offline > /dev/null
    
    status=$?

    This would only return the exit code of:

     

    tmsh show ltm pool Mypool2_Stg_pool field-fmt | grep -o offline > /dev/null

    Example:

     

    This command works:
    [root@PatrikLab:Active] config  ls bigip.conf > /dev/null
    [root@PatrikLab:Active] config  echo $?
    0
    
    This command does not work:
    [root@PatrikLab:Active] config  ls nonexistingfile > /dev/null
    ls: nonexistingfile: No such file or directory
    [root@PatrikLab:Active] config  echo $?
    2
    
    Run them together:
    [root@PatrikLab:Active] config  ls nonexistingfile > /dev/null
    ls: nonexistingfile: No such file or directory
    [root@PatrikLab:Active] config  ls bigip.conf > /dev/null
    [root@PatrikLab:Active] config  echo $?
    0

    So maybe you'd want to do something like this?

     

    !/bin/sh
    
    pidfile="/var/run/pinger.$1..$2.pid"
    
    if [ -f $pidfile ]
    then
       kill -9 `cat $pidfile` > /dev/null 2>&1
    fi
    
    echo "$$" > $pidfile
    
    node_ip=`echo $1 | sed 's/::ffff://'`
    
    status="UP"
    
    tmsh show ltm pool Mypool1_Stg_pool field-fmt | grep -o offline > /dev/null
    if [ $? ]; then
        status="DOWN";
    fi
    tmsh show ltm pool Mypool2_Stg_pool field-fmt | grep -o offline > /dev/null
    if [ $? ]; then
        $status="DOWN"
    fi
    
    echo $status

    /Patrik

     

  • I ended up doing this which seems to work when I run the script from the command line. I get "up" if the all of the pools are available, and I get no output if one or more of them is offline.

    I applied the monitor to a test pool, and even if the output of the script is null which should mark the test pool offline it doesn't and i'm not sure why...

    !/bin/sh
    
     $1 = IP (nnn.nnn.nnn.nnn notation or hostname)
     $2 = port (decimal, host byte order)
     $3 and higher = additional arguments
    
     In this sample script, $3 is the regular expression
    
    
    pidfile="/var/run/pinger.$1..$2.pid"
    
    if [ -f $pidfile ]
    then
       kill -9 `cat $pidfile` > /dev/null 2>&1
    fi
    
    echo "$$" > $pidfile
    
    node_ip=`echo $1 | sed 's/::ffff://'`
    
    pool1=$(tmsh show ltm pool 1_pool field-fmt | grep -o offline)
    pool2=$(tmsh show ltm pool 2_pool field-fmt | grep -o offline)
    pool3=$(tmsh show ltm pool 3_pool field-fmt | grep -o offline)
    pool4=$(tmsh show ltm pool 4_pool field-fmt | grep -o offline)
    pool5=$(tmsh show ltm pool 5_pool field-fmt | grep -o offline)
    
    if [ -z "$pool1" ]&&[ -z "$pool2" ]&&[ -z "$pool3" ]&&[ -z "$pool4" ]&&[ -z "$pool5" ];
    then
         echo "up"
      fi
    
    rm -f $pidfile
    
  • Hi!

    There's no need to save the node ip since you're only checking the pools. Also, the removal of the pid file should be placed before the "echo up" as the "up" string makes the ltm exit the script.

    Here's a version that worked for me at least. Change the pool names with those of your own.

    !/bin/sh
    
     Name of the pidfile
    pidfile="/var/run/$MONITOR_NAME.$1..$2.pid"
    
     Send signal to the process group to kill our former self and any children 
     as external monitors are run with SIGHUP blocked
    if [ -f $pidfile ]
    then
       kill -9 -`cat $pidfile` > /dev/null 2>&1
    fi
    
    echo "$$" > $pidfile
    
    pool1=$(tmsh show ltm pool SorryPool field-fmt | grep -o offline)
    pool2=$(tmsh show ltm pool PajoIIS-8080-8081_pool field-fmt | grep -o offline)
    
    if [ -z $pool1 ] && [ -z $pool2 ];
    then
         rm -f $pidfile
         echo "up"
    fi
    

    You might want to change the check from offline to accomodate the scenario where the members are disabled as well?

    !/bin/sh
    
     Name of the pidfile
    pidfile="/var/run/$MONITOR_NAME.$1..$2.pid"
    
     Send signal to the process group to kill our former self and any children
     as external monitors are run with SIGHUP blocked
    if [ -f $pidfile ]
    then
       kill -9 -`cat $pidfile` > /dev/null 2>&1
    fi
    
    echo "$$" > $pidfile
    
    status=0
    
    tmsh show ltm pool SorryPool field-fmt | grep -o "status.status-reason The pool is available" > /dev/null
    if [ $? != 0 ]; then
        status=1;
    fi
    tmsh show ltm pool PajoIIS-8080-8081_pool field-fmt | grep -o "status.status-reason The pool is available" > /dev/null
    if [ $? != 0 ]; then
        status=1;
    fi
    
    if [ $status == 0 ];
    then
         rm -f $pidfile
         echo "up"
    fi
    

    Note that this will also bring the pool down if the pools being checked are in an unknown state though. If you want to take that into account I'd recommend writing a function checking both monitor state and enabled state.

    /Patrik

  • Ah that's a good point, I'll make the last part of the script be:

    if [ -z $pool1 ] && [ -z $pool2 ];
    then
         rm -f $pidfile
         echo "up"
    fi
    

    This is the first external monitor I've played with so it's new territory for me. What I read in the help docs is that if there's any output from the script the F5 interprets it as a positive response (system up) and if the script doesn't output anything then it's negative (system down).

    The script as-is works in this manor right now - if all pools are up, the script outputs the text "up" and if any pool is down the script doesn't output anything. I expected my test pool with this monitor applied to go down when I brought down one of the pools but it didn't and if I run the script manually it's acting as it should, not emitting any text. Am I doing something wrong?

    Here's a screen shot of the monitor within the F5, fairly basic...

    Thank you again for all the help!

  • Actually never mind, I just fixed it. I had to put in:

    export REMOTEUSER="root"

    and it started working correctly.

    Thank you again for all your help!