Learn F5 Technologies, Get Answers & Share Community Solutions Join DevCentral

Filter by:
  • Solution
  • Technology
Answers

Open SSL error on ltm logs since v11 upgrade

Hello,

We upgraded our viprion 4802 from 10.2.4 to 11.2.1. Since this upgrade, we see ltm logs concerning Open SSL error every 5 seconds:

Sep 13 04:02:14 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure. Sep 13 04:02:14 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol. Sep 13 04:02:14 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol. Sep 13 04:02:14 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol. Sep 13 04:02:16 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol. Sep 13 04:02:17 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol. Sep 13 04:02:17 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol. Sep 13 04:02:17 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol. Sep 13 04:02:18 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol. Sep 13 04:02:19 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure. Sep 13 04:02:19 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol. Sep 13 04:02:19 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol. Sep 13 04:02:19 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol. Sep 13 04:02:21 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol. Sep 13 04:02:22 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol. Sep 13 04:02:22 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol. Sep 13 04:02:22 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol. Sep 13 04:02:23 slot1/A-EB2-BIGIP-DMZ-BCK err bigd[10121]: 01060111:3: Open SSL error - error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol.

Did anyone ever faced of this problem ?

Regards,

Hassan Hireche

0
Rate this Question

Answers to this Question

placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

In our case the poolmembers were just marked down by the monitor as they did not enter the SSL handshake. So parsing the list of all poolmembers for their status was answering the question for us.
In case this is a intermittent problem, watching the /var/log/ltm should reveal suspect poolmembers.
Finally a specific tcpdump proved the assumption.
Btw, 1.200 pools is really a lot. If each of these is having i.e. 2 poolmembers to be monitored with https I would also be a bit concerned about scalability of the BIG-IP´s monitoring capacity. You are using a very short monitoring interval?

0
Comments on this Answer
Comment made 16-Sep-2013 by Hassan Hireche 77
Hello Stephan, Thank you for your answer. I managed to find which pool has generated this error using "tmsh list ltm pool" command associated with grep. The error was due to a monitor https applied on a port 80 on the node. There is still one problematic pool, because the error happens now every 5 seconds only (Vs every 2 or 3 seconds before pool modification). I need to go further in the configuration file to match the bad couple monitor/port. Thank you all for your help and suggestions.
0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

usually related to using wrong SSL version or send normal http traffic instead of https.

is it clear to you which virtual server causes this? does it work normally for you?

has all config been checked? nothing new been added? every 5 seconds might indicate a monitor which causes this.

0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

I saw this problem related to monitoring with an HTTPS monitor.
The poolmember accepted the 3way handshake, but rejected the CLIENT HELLO coming from the bigd.
We removed the HTTPS monitor for the relevant poolmembers.

0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Thank you for your answers. We suspected a monitor problem, but we have more than 1200 pools, this is very difficult to know which ones are generating this kind of log. Is there a way to activate debug on these log? Thank you for your help.

0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

You could set SSL logging to debug and see if that gives you more information, but don't leave it in debug mode for very long. Turn it on only for as long as you need to gather information on the problem and then set it back to its default.

To turn on debug logging: tmsh modify sys db log.ssl.level value "Debug"

To set it back to the default: tmsh modify sys db log.ssl.level value "Error"

0
Comments on this Answer
Comment made 13-Sep-2013 by boneyard 5579
for me that doesn't help anything, doesn't point out the monitor causing the issue.
0
Comment made 13-Sep-2013 by Hassan Hireche 77
Effectively, it doesn't offer more informations about the pool causing the error
0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Hi Hassan,

the tmsh 'one-line' option is very helpful in case you want to grep.

Let´s sort out your https based monitors first:

tmsh list ltm monitor one-line | grep https | awk '{print $4}'

Now you can look for all poolmembers configured on port 80|http in state down with one of your suspect monitors:

tmsh list ltm pool one-line | grep -E ':(80|http) .*session monitor-enabled state down.*(your_monitor_here)' | awk '{print $3}'

Good luck,

Stephan

0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Putting it into a script should make life easier:

#! /bin/bash

i=0
for line in `tmsh list ltm monitor one-line | grep https | awk '{print $4}'`
do
  arr[$i]=$line
  i=`expr $i + 1`
done

for i in "${arr[@]}"
do
   tmsh list ltm pool one-line | grep -E ":(80|http) .*session monitor-enabled state down.*${i}" | awk '{print $3}'
done
0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Or a bit shorter:

#! /bin/bash

for line in `tmsh list ltm monitor one-line | grep https | awk '{print $4}'`
do
  tmsh list ltm pool one-line | grep -E ":(80|http) .*session monitor-enabled state down.*${line}" | awk '{print $3}'
done
0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

I have written a fairly comprehensive script that should root out these troubled pool members. I have included it in the DevCentral CodeShare in hopes that it can help others out. Let me know if you have any issues with the script.

Clean Up Monitor Error: "SSL23_GET_SERVER_HELLO:unknown protocol" https://devcentral.f5.com/wiki/TMSH.Clean-Up-Monitor-Error-SSL23-GET-SERVER-HELLO-unknown-protocol.ashx

0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

I've seen multiple solutions that require to run multiple tmsh command on the running BigIP.

I've made a (dirty) script that parse the bigip.conf and return a CSV list of node on port 80 with a https monitor. The only downside is all the monitor doing https check must have the word "https" in it.

#! /usr/bin/sh

nbl=0 ;

runningPool=""
runningIRule="" 
runningVS=""
runningNode=""
while IFS= read -r bigipconf
do

    if [[ $bigipconf == ltm\ pool\ \/Common\/* ]]
    then
        runningPool=$(echo $bigipconf | awk '{print $3}' | awk -F '/' '{print $3}')
        #echo $runningPool
        poolNode=""
    fi

    if [[ $runningPool != "" ]] 
    then
        if [[ $bigipconf == *\/Common\/*\:80* ]] 
        then
            runningNode=$(echo $bigipconf | awk -F '/' '{print $3 }' | awk '{ print $1 }')
            poolNode=$(echo "$poolNode;$runningNode") 
        fi
        if [[ $bigipconf == *monitor*  ]]
        then
            monitor=$(echo $bigipconf   | awk -F '/' '{ print $3 }')
            if [[ $monitor == *https* ]] 
            then
                echo -e "$runningPool;$monitor;$poolNode" | grep ':'

            fi
        fi

    fi

    if [[ $bigipconf == ltm\ virtual\ \/Common\/* ]]
    then
        runningPool=""
    fi
done < input/bigip.conf
0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

We had the same problem at my office and it turns out that it was a Cipher issue with the HTTPS monitor. As part of web server hardening, everything but TLS1.2 was disabled on the web servers. The HTTPS monitor did have "DEFAULT:+SHA:+3DES:+kEDH" in the ciphers list and the 'DEFAULT' group does also include TLSv1.2 cipher suites but somehow this was causing problems.

I changed the cipher list to TLSv1_2+AES and this immediately fixed the problem - though it did create other problems as I did have a pool where some servers were hardened while others weren't. Adding this cipher suites caused the pool members pointing to the unhardened servers to go down.

In the end I couldn't find a compromise so I just put it to "tcp". Not the best solution but I need to get the server and application guys to fix it on the servers.

0