Forum Discussion

seilemor_131269's avatar
seilemor_131269
Icon for Altostratus rankAltostratus
Aug 29, 2016

SSL errno 104 through F5 (vip), directly with curl ok

Hello community,

 

I've the following configuration design within a virtual server configuration:

 

- A virtual server will route the traffic based on the hostname within the request to different pools.
- Client- and server SSL are enabled
- The routing will been done with LTM policys instead of iRules.
- There are five backend systems which should be accessible over the vs
- SNAT is enabled

Now, the problem is that from the five backends are only three backend systems are accessible. The other two systems don't work. Within the LTM policy I've additional enabled the "log" option to make sure that the routing will work.

 

To find out what is going wrong, I've executed a curl and openssl query direct to the backend system from the F5 console. Connection can be established and I receive the 200 status code. If I do the same over the VIP of the configuration I receive the following error code:

 

read:errno=104

These are my teststrings:

 

openssl s_client -connect vip:443
GET /dialin/ HTTP/1.1
Host: lyncpool1 or 2 or 3 ...

With lyncpool1 as the value for the host it will work and with lyncpool2 I receive the error code from above. The client SSL settings from the vip will been displayed. It seams that there is an issue/problem while talking SSL with the backend system which will not work. If I do the same with the original IP of the system instead of the VIP, each system are working fine.

 

Have anybody an idea? I've just readed some threads within devcentral to similar problems but nothing helped my until now.

 

Regards seilemor

 

20 Replies

  • Hi Seilemor,

     

    CURL, the F5 Monitors and the Virtual Servers are using independent SSL settings. So its not uncommon that one method may work and the other doesn't...

     

    To troubleshoot your issue further, you may enable LTM SSL Debug logging to see if any SSL related errors are getting raised. Go to Logs\Configuration\Options and then set SSL Logging to "DEBUG". Then take a look to the Local Traffic logfile...

     

    Note: You may also temporary switch the Server SSL profile to "serverssl" or even "serverssl-insecure-compatible" to identify server side SSL problems. If those generic SSL profiles are working for you, then you have alredy identify the cause (aka. custom SSL settings on each individual pool member.

     

    Cheers, Kai

     

  • Hey Kai,

    thanks for the very fast answer. I already activaed the standard serversll profile to avoid problems. After activating the SSL debug level I see the following error within "\var\log\ltm"

    Aug 29 12:44:01 o01006670 info tmm2[5320]: 01260013:6: SSL Handshake failed for TCP from 172.21.x.x:45218 to 10.235.x.x:4443
    

    The 172.21.x.x IP address is the floating one from the F5 cluster. The IP address 10.235.x.x are the backend system.

    Ok - the handshake failed. What now?

    Regards seilemor

  • Hi Seilemor,

    the next step would be to activate the "serverssl-insecure-compatible" Profile and to TCPDUMP the SSL handshake between your Floating IP and the individual Backend servers and then inspect/compare the collected SSL handshakes using e.g. WireShark.

    tcpdump -i YOUR_VLAN_LABEL -w /var/tmp/capture.cap dst host 10.235.x.x and dst port 4443

    Cheers, Kai

  • Ok - I've used the article "how to use ssldump utility" for the next steps. After performing tcpdump and ssldump I will receive the following lines:

    For description:

    - x.x.x.200 is the non floating IP address which execute the monitor check.

    - x.x.x.201 is the floating IP address for the regular request

    Within the connection of the floating IP address I see just nothing. I already used the insecude SSL server profile, too. To check the system I've also reconfigured the vip to use the non nloating IP (x.x.x.200).

    New TCP connection 2: 172.21.254.200(61407) <-> 10.235.96.29(4443)
    2 1  0.1926 (0.1926)  C>S  Handshake
          ClientHello
            Version 3.3
            cipher suites
            TLS_RSA_WITH_RC4_128_MD5
            TLS_RSA_WITH_RC4_128_SHA
            TLS_RSA_WITH_AES_128_CBC_SHA
            TLS_RSA_WITH_AES_256_CBC_SHA
            TLS_RSA_WITH_3DES_EDE_CBC_SHA
            TLS_RSA_WITH_DES_CBC_SHA
            TLS_RSA_WITH_AES_128_CBC_SHA256
            TLS_RSA_WITH_AES_256_CBC_SHA256
            Unknown value 0xc013
            Unknown value 0xc014
            Unknown value 0xc012
            Unknown value 0xff
            compression methods
                      NULL
    2    0.5735 (0.3808)  S>C  TCP FIN
    2    0.5736 (0.0001)  C>S  TCP RST
    New TCP connection 3: 172.21.254.200(59346) <-> 10.235.96.29(4443)
    3 1  0.1939 (0.1939)  C>S  Handshake
          ClientHello
            Version 3.3
            cipher suites
            TLS_RSA_WITH_RC4_128_MD5
            TLS_RSA_WITH_RC4_128_SHA
            TLS_RSA_WITH_AES_128_CBC_SHA
            TLS_RSA_WITH_AES_256_CBC_SHA
            TLS_RSA_WITH_3DES_EDE_CBC_SHA
            TLS_RSA_WITH_DES_CBC_SHA
            TLS_RSA_WITH_AES_128_CBC_SHA256
            TLS_RSA_WITH_AES_256_CBC_SHA256
            Unknown value 0xc013
            Unknown value 0xc014
            Unknown value 0xc012
            Unknown value 0xff
            compression methods
                      NULL
    New TCP connection 4: 172.21.254.201(1978) <-> 10.235.96.29(4443)
    4    0.2052 (0.2052)  C>S  TCP FIN
    3    0.5759 (0.3820)  S>C  TCP FIN
    3    0.5760 (0.0000)  C>S  TCP RST
    4    0.5828 (0.3776)  S>C  TCP FIN
    New TCP connection 5: 172.21.254.200(34721) <-> 10.235.96.29(4443)
    5 1  0.1934 (0.1934)  C>S  Handshake
          ClientHello
            Version 3.3
            cipher suites
            TLS_RSA_WITH_RC4_128_MD5
            TLS_RSA_WITH_RC4_128_SHA
            TLS_RSA_WITH_AES_128_CBC_SHA
            TLS_RSA_WITH_AES_256_CBC_SHA
            TLS_RSA_WITH_3DES_EDE_CBC_SHA
            TLS_RSA_WITH_DES_CBC_SHA
            TLS_RSA_WITH_AES_128_CBC_SHA256
            TLS_RSA_WITH_AES_256_CBC_SHA256
            Unknown value 0xc013
            Unknown value 0xc014
            Unknown value 0xc012
            Unknown value 0xff
            compression methods
                      NULL
    5    0.5740 (0.3806)  S>C  TCP FIN
    5    0.5741 (0.0001)  C>S  TCP RST
    New TCP connection 6: 172.21.254.200(46619) <-> 10.235.96.29(4443)
    6 1  0.1980 (0.1980)  C>S  Handshake
          ClientHello
            Version 3.3
            cipher suites
            TLS_RSA_WITH_RC4_128_MD5
            TLS_RSA_WITH_RC4_128_SHA
            TLS_RSA_WITH_AES_128_CBC_SHA
            TLS_RSA_WITH_AES_256_CBC_SHA
            TLS_RSA_WITH_3DES_EDE_CBC_SHA
            TLS_RSA_WITH_DES_CBC_SHA
            TLS_RSA_WITH_AES_128_CBC_SHA256
            TLS_RSA_WITH_AES_256_CBC_SHA256
            Unknown value 0xc013
            Unknown value 0xc014
            Unknown value 0xc012
            Unknown value 0xff
            compression methods
                      NULL
    6    0.5788 (0.3807)  S>C  TCP FIN
    6    0.5789 (0.0001)  C>S  TCP RST
    New TCP connection 7: 172.21.254.200(3858) <-> 10.235.96.29(4443)
    7 1  0.1927 (0.1927)  C>S  Handshake
          ClientHello
            Version 3.3
            cipher suites
            TLS_RSA_WITH_RC4_128_MD5
            TLS_RSA_WITH_RC4_128_SHA
            TLS_RSA_WITH_AES_128_CBC_SHA
            TLS_RSA_WITH_AES_256_CBC_SHA
            TLS_RSA_WITH_3DES_EDE_CBC_SHA
            TLS_RSA_WITH_DES_CBC_SHA
            TLS_RSA_WITH_AES_128_CBC_SHA256
            TLS_RSA_WITH_AES_256_CBC_SHA256
            Unknown value 0xc013
            Unknown value 0xc014
            Unknown value 0xc012
            Unknown value 0xff
            compression methods
                      NULL
    7    0.5746 (0.3818)  S>C  TCP FIN
    7    0.5747 (0.0001)  C>S  TCP RST
    
    • Kai_Wilke's avatar
      Kai_Wilke
      Icon for MVP rankMVP

      Hi Seilemor,

      sorry my fault. Change the TCPDUMP filters to the example below. It will allow you to capture bidirectional traffic...

      tcpdump -i YOUR_VLAN_LABEL -w /var/tmp/capture.cap host 10.235.x.x and port 4443

      But anyhow. In your last capture the F5 Floating hasn't even send a Client Hello to the problematic servers. But on the other hand the F5 Floating was able to send a Client Hello (and successfully negotiate a SSL connection) to the remaining servers using the identical pool and SSL Profiles?

      Thats a somewhat strange behavior... 😞

      Cheers, Kai

  • Hey Kai,

     

    I have executed the tcpdump, which was the base for the ssldump, without the "dst" option to receive also the answer packages. That the floating IP does not send correctly the clienthello I've also noticed... sadly I have no answer for these situation :(

     

    What I've also checked is what happen if I create a virtual server with only the problem server and without policy routing. With this configuration I also have no success.

     

    • Kai_Wilke's avatar
      Kai_Wilke
      Icon for MVP rankMVP

      I guess this is more a node specific problem then? Configuration differences? Wrong Routing Tables? Bad SSL Chiphers? Some FW access-lists in transit? To get sure, I'd like to recomment to TCPDUMP directly on the affected nodes (or use a SPAN port on your switches)...

       

      Cheers, Kai

       

  • here comes the next frustrating thing: I've tested the configuration within my lab version with the result that it work there how it was designed. The lab version is installed with version 12 and the productive system has 11.4.1

     

    • Kai_Wilke's avatar
      Kai_Wilke
      Icon for MVP rankMVP

      Oh gosh... its getting even more weird. :-( But keep in mind, that the Lab and Live environment are not using the same Source IPs. I'd like to recommend to check the intermediate routers / firewall systems in place for missing configurations... :-(

       

      Cheers, Kai

       

  • Uh, my fault: the IP address x.x.x.200 is the floating one, the 201 is the non floating which execute the monitor checks.

    I've now disabled the monitor and executed again the tcpdump and ssldump. Following the output which will displayed 10 times:

    New TCP connection 1: 172.21.254.200(12938) <-> 10.235.96.29(4443)
    1 1  0.2097 (0.2097)  C>S  Handshake
          ClientHello
            Version 3.3
            cipher suites
            TLS_RSA_WITH_RC4_128_SHA
            TLS_RSA_WITH_AES_128_CBC_SHA
            TLS_RSA_WITH_AES_256_CBC_SHA
            TLS_RSA_WITH_3DES_EDE_CBC_SHA
            TLS_RSA_WITH_AES_128_CBC_SHA256
            TLS_RSA_WITH_AES_256_CBC_SHA256
            Unknown value 0xc013
            Unknown value 0xc014
            Unknown value 0xc012
            Unknown value 0xff
            compression methods
                      NULL
    1    0.5910 (0.3812)  S>C  TCP FIN
    1    0.5911 (0.0001)  C>S  TCP RST
    
  • Now I have somehting positive: if I change the cipher within the monitor check to "DEFAULT:+SHA:+3DES:+kEDH", the monitor checks are working. Within a tcpdump/ssldump I see some client and server hello.

    The problem: I can not use these cipher list within the SSL server profile. If I try to use it I receive the error message that the value "kEDH" is unknown.

    I think that the problem between the F5 and the Backend are the ciphers which can be used and which can not be used. Additional I think that the normal openssl stack which can executed within a normal console is not the same as the stack which will been used within the bigip daemon self.

    With a script I've checked which ciphers are provided from the backend system. Here is the output:

     Cipher order
         SSLv3:     DES-CBC3-SHA RC4-SHA RC4-MD5
         TLSv1:     ECDHE-RSA-AES256-SHA ECDHE-RSA-AES128-SHA DHE-RSA-AES256-SHA DHE-RSA-AES128-SHA AES256-SHA AES128-SHA DES-CBC3-SHA RC4-SHA RC4-MD5
         TLSv1.1:   ECDHE-RSA-AES256-SHA ECDHE-RSA-AES128-SHA DHE-RSA-AES256-SHA DHE-RSA-AES128-SHA AES256-SHA AES128-SHA DES-CBC3-SHA RC4-SHA RC4-MD5
         TLSv1.2:   ECDHE-RSA-AES256-SHA384 ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES256-SHA ECDHE-RSA-AES128-SHA DHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-SHA DHE-RSA-AES128-SHA AES256-GCM-SHA384 AES128-GCM-SHA256 AES256-SHA256 AES128-SHA256 AES256-SHA AES128-SHA textDES-CBC3-SHA RC4-SHA RC4-MD5
    

    I've tested a little bit around with the cipher lists which the F5 can use (native, compat, default, ...). But sadly nothing has solved my issue.

  • I've changed the cipher like it was described in the KB article. Unfortunately it has also not solved the problem.

     

    After I've disabled the monitor of the pool I can see Client Hello messages if I try to reconnect to the site. The result/output from the tcpdump is somewhere in the answers above.

     

    • Kai_Wilke's avatar
      Kai_Wilke
      Icon for MVP rankMVP

      Hi Seilemor,

       

      unfortunately I can't help you further unless you start to TCPDUMP/SSLDUMP on the application servers.

       

      Cheers, Kai

       

    • seilemor_131269's avatar
      seilemor_131269
      Icon for Altostratus rankAltostratus

      Hey, I will perform an tcpdump on the application server today at 2pm...

       

    • seilemor_131269's avatar
      seilemor_131269
      Icon for Altostratus rankAltostratus

      hmmm... now I have the tcpdump from the application server. Sadly I can not see anything. If I put the pcap file into the ssldump utility I can also only see the client hello commands from the F5 :(

       

  • Ok - here are some new tcpdumps. On the first screenshot you can see the tcpdump if I perform manually the command

    openssl s_client -connect 10.235.96.29:4443

    The second screenshot is the tcpdump if I execute the test through the F5 vip. The connection try including only five packages. After the F5 receive the RST, ACK from the application server the F5 start a new try.

    The only thing which I can see at the moment is that the length of the 4th package is different.

    • seilemor_131269's avatar
      seilemor_131269
      Icon for Altostratus rankAltostratus

      Ok - I've solved the problem. I've disabled TLS features within the server SSL profile. I think it is a "feature" of the old 11.4.1 version which we are use. I'll now plan a update of the system to the current BigIP version.

       

      Thanks for your support and your helpful comments ;)

       

    • Kai_Wilke's avatar
      Kai_Wilke
      Icon for MVP rankMVP

      Hi seilemor,

       

      good to hear that you've managed to solve your issue! ;-)

       

      Cheers, Kai