Forum Discussion

FraserK_151071's avatar
FraserK_151071
Icon for Nimbostratus rankNimbostratus
Jun 17, 2014

External monitor "state returned down" response from incorrect IP

I have a combined GTM/LTM system running BIG-IP Version 11.4.1 Build 608.0 Final (Let's call this Device 1).

 

I have several pools of virtual servers in the Global Traffic component, and I have created an external monitor to check the availability of these virtual servers, and hence marking them online/offline accordingly. The external monitor currently checks a static web page hosted on a web server via port 80. The monitor greps for the string "=up" in order to mark the virtual server as available.

 

This works as expected on a BIG-IP GTM Virtual Edition running v11.3.0 Build 2806.0 Final (no LTM component, let's call this Device 2). I can run a tcpdump packet capture on the external interface and see the device sending HTTP requests to the IP address of the web server, as expected.

 

However, on Device 1, the same tcpdump returns nothing. Not even for any of the other interfaces. I can run a cURL request from the console successfully, which verifies that there are no firewall issues - but I can't work out why I'm not seeing the GTM try to reach the web server.

 

On Device 1, when I go to Global Traffic > Pools > Pool List and check out the members of a pool, I can mouse over the red icon in the Status column, and get one of the two below tooltips:

 

Offline (Enabled) - Monitor /Common/HTTPMonitorcURLGET from 192.168.1.125 : state returned down

 

Offline (Enabled) - Monitor /Common/HTTPMonitorcURLGET from 192.168.0.251 : state returned down

 

These are not the IP addresses the monitor should be using! As far as I understand, from GTM, Know Thyself, the self-IP of the GTM's internal interface should be displayed here (xxx.xxx.xx.126 = 192.168.0.240). The correct IP is shown for Device 2.

 

This is the Listener and Server Object configuration for Device 1:

 

Device 1 External UDP listener Destination: 192.168.0.240

 

Device 1 Internal UDP listener Destination: 192.168.1.125

 

Device 1 GTM Self-IPs: 192.168.1.125, xxx.xxx.xx.126 = 192.168.0.240

 

Device 1 LTM Self-IPs: 192.168.0.251

 

Note that the GTM has both the Internal and External interfaces set as self-IPs.

 

My question is: how do I force the monitor HTTPMonitorcURLGET to use the 192.168.0.240 (external) interface, instead of the internal GTM or LTM interfaces! Why does a packet capture not even show packets going to the web server from the internal interface? Device 1 is a live system, so I can't change things on a whim without knowing (a) it won't break anything else and (b) that it will work, beyond reasonable doubt!

 

HTTPMonitorcURLGET script:

 

!/bin/sh

 These arguments are supplied automatically for all external monitors:
 $1 = IP address of the virtual server (IPV6 notation. IPv4 addresses are
      passed in the form ::ffff:w.x.y.z, where "w.x.y.z" is the IPv4 address)
 $2 = port of the virtual server (decimal, host byte order)

 Name/Value pairs may also be specified in the monitor template.
 This example expects the following Name/Value pairs:
   URI  = the URI to request from the server (e.g. "/testpath/testfile.html")
   RECV = the expected response (not case sensitive) e.g. "UP"
   B    = web server IPv4 addresses, separated by a ":"
   DEBUG= enable logging to local0.debug if set to 1, no logging if set to 0

 Log debug to local0.debug
if [-n "$DEBUG" ]
  then
    if [ $DEBUG -eq 1]
      then
        echo "EAV `basename $0`: \$DEBUG: $DEBUG" | logger -p local0.debug
    fi
  else
     If the monitor configuration didn't specify debug, enable/disable it here
    DEBUG=0
fi

 Remove IPv6/IPv4 compatibility prefix (GTM passes addresses in IPv6 format)
IP=`echo $1 | sed 's/::ffff://'`

 Save the port for use in the shell command
PORT=$2

 Check if there is a prior instance of the monitor running
PIDFILE="/var/run/`basename $0`.$IP.$PORT.pid" 
 kill off the last instance of this monitor if hung
if [ -f $PIDFILE ]  if file $PIDFILE exists and is not a directory
  then
    echo "EAV `basename $0`: exceeded monitor interval, needed to kill"\
    "${IP}:${PORT} with PID `cat $PIDFILE`" | logger -p local0.error
     interface with syslog module with priority (-p) "error" in local0 facility
    kill -9 `cat $PIDFILE` > /dev/null 2>&1  9 is the signal number
fi

 Add the current PID to file
echo "$$" > $PIDFILE  $$ is the PID of the current process

 Debug
if [ $DEBUG -eq 1 ]
  then
    echo "EAV `basename $0`: Running for ${IP}:${PORT} using custom command" \
    | logger -p local0.debug
fi

 Split the B value into individual IP addresses
bld=$(echo $B | tr ":" "\n")

for i in $bld  cycle through each web page, exit if "UP" is found
  do
    curl --connect-timeout 30 -fNs http://${i}:80${URI} | grep -i \
    "${IP}=${RECV}" 2>&1 > /dev/null
    if [ $? -eq 0 ]  $? is the return code of the last executed command
      then
        rm -f $PIDFILE
        if [ $DEBUG -eq 1 ]
          then
            echo "EAV `basename $0`: Succeeded for ${IP}:${PORT}" | logger -p \
            local0.debug
        fi
        echo "UP"
        exit  exit from the script early if an "UP" is found
      else
        if [ $DEBUG -eq 1 ]
          then
            echo "EAV `basename $0`: Failed for ${IP}:${PORT}" | logger -p \
            local0.debug
        fi
    fi
  done
rm -f $PIDFILE

1 Reply

  • So, somewhat embarrassingly, the error was in the monitor script. I'd put in backslashes to try to keep lines within 80 characters width, and apparently the F5 can't handle that. Rather than copy the working monitor from Device 2 to Device 1, I'd copied the script I'd written up in the documentation (which is broken)!

    !/bin/sh
    .....
    curl --connect-timeout 30 -fNs http://${i}:80${URI} | grep -i \
    "${IP}=${RECV}" 2>&1 > /dev/null
    

    should be

    !/bin/sh
    .....
    curl --connect-timeout 30 -fNs http://${i}:80${URI} | grep -i "${IP}=${RECV}" 2>&1 > /dev/null
    

    The self-IP, listeners, etc. were set up correctly. I'll leave this question up here just in case anyone else has the same brain fart and needs a helping hand!