Learn F5 Technologies, Get Answers & Share Community Solutions Join DevCentral

Filter by:
  • Solution
  • Technology
Answers

BIG-IP Proxy SSL 12.1 Handshake Failure

I set up SSL Proxy in order to do client certificate authentication on my IIS web server on LTM 12.1 firmware. The setup is working fine on Firefox version 43, IE 10 and OpenSSL but it fails on Chrome 51, Firefox 47 and IE 11.

I've captured the packets. Clients use TLS1 or TLS1.2 using the same ciphersuite of TLS_RSA_WITH_AES_256_CBC_SHA (0x0035), the same process takes place for the passing and failing cases.

  1. Client Hello
  2. Server Hello, Certificate, Server Hello Done
  3. Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message 4. 4.1 Either Server sends Change Cipher Spec and then Application Data gets transfered Or 4.2 The server sends Alert level: Fatal, Descrition: Handshake Failure

So I suspect the BIG-IP fails to decrypt the handshake sent by the client in some cases but I can't figure out why because there's nothing different between failing and passing tests.

ssldump using Firefox 47 (Fails):

    New TCP connection #1: 192.168.100.125(55041) <-> 192.168.100.231(443)

1 1  0.0027 (0.0027)  C>S  Handshake

      ClientHello

        Version 3.3 

        cipher suites

        TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

        Unknown value 0xcca9

        Unknown value 0xcca8

        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA

        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

        TLS_DHE_RSA_WITH_AES_128_CBC_SHA

        TLS_DHE_RSA_WITH_AES_256_CBC_SHA

        TLS_RSA_WITH_AES_128_CBC_SHA

        TLS_RSA_WITH_AES_256_CBC_SHA

        TLS_RSA_WITH_3DES_EDE_CBC_SHA

        compression methods

                  NULL

1 2  0.0033 (0.0005)  S>C  Handshake

      ServerHello

        Version 3.3 

        session_id[32]=

          d9 0a 00 00 3e 11 22 ac e2 c2 00 f5 9a 41 35 53 

          43 6a 9e a5 e0 26 32 e4 f8 38 2e ca 72 3c fb 93 

        cipherSuite         TLS_RSA_WITH_AES_256_CBC_SHA

        compressionMethod                   NULL

      Certificate

      ServerHelloDone

1 3  0.0185 (0.0151)  C>S  Handshake

      ClientKeyExchange

1 4  0.0185 (0.0000)  C>S  ChangeCipherSpec

1 5  0.0185 (0.0000)  C>S  Handshake

1 6  0.0196 (0.0011)  S>C  Alert

    level           fatal

    value           handshake_failure

1    0.0197 (0.0000)  S>C  TCP FIN

1    0.0205 (0.0008)  C>S  TCP FIN

New TCP connection #2: 192.168.100.125(55042) <-> 192.168.100.231(443)

2 1  0.0005 (0.0005)  C>S  Handshake

      ClientHello

        Version 3.2 

        cipher suites

        Unknown value 0x5600

        Unknown value 0xcca9

        Unknown value 0xcca8

        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA

        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

        TLS_DHE_RSA_WITH_AES_128_CBC_SHA

        TLS_DHE_RSA_WITH_AES_256_CBC_SHA

        TLS_RSA_WITH_AES_128_CBC_SHA

        TLS_RSA_WITH_AES_256_CBC_SHA

        TLS_RSA_WITH_3DES_EDE_CBC_SHA

        compression methods

                  NULL

2 2  0.0010 (0.0005)  S>C  Handshake

      ServerHello

        Version 3.2 

        session_id[32]=

          85 48 00 00 8f 2a ae 80 b8 d7 e9 e2 47 c0 15 4e 

          e8 af 69 6f 2d b9 b8 d6 ed d5 29 3c a3 a3 44 b3 

        cipherSuite         TLS_RSA_WITH_AES_256_CBC_SHA

        compressionMethod                   NULL

      Certificate

      ServerHelloDone

2 3  0.0145 (0.0134)  C>S  Handshake

      ClientKeyExchange

2 4  0.0145 (0.0000)  C>S  ChangeCipherSpec

2 5  0.0145 (0.0000)  C>S  Handshake

2 6  0.0158 (0.0013)  S>C  Alert

    level           fatal

    value           handshake_failure

2    0.0158 (0.0000)  S>C  TCP FIN

2    0.0162 (0.0003)  C>S  TCP FIN

New TCP connection #3: 192.168.100.125(55043) <-> 192.168.100.231(443)

3 1  0.0005 (0.0005)  C>S  Handshake

      ClientHello

        Version 3.1 

        cipher suites

        Unknown value 0x5600

        Unknown value 0xcca9

        Unknown value 0xcca8

        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA

        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

        TLS_DHE_RSA_WITH_AES_128_CBC_SHA

        TLS_DHE_RSA_WITH_AES_256_CBC_SHA

        TLS_RSA_WITH_AES_128_CBC_SHA

        TLS_RSA_WITH_AES_256_CBC_SHA

        TLS_RSA_WITH_3DES_EDE_CBC_SHA

        compression methods

                  NULL

3 2  0.0010 (0.0004)  S>C  Handshake

      ServerHello

        Version 3.1 

        session_id[32]=

          aa 41 00 00 04 82 07 3f ed 35 96 49 e2 c5 ba 79 

          f8 39 5a f2 d2 41 19 33 8e 5b 05 5e 2f d1 ca 24 

        cipherSuite         TLS_RSA_WITH_AES_256_CBC_SHA

        compressionMethod                   NULL

      Certificate

      ServerHelloDone

3 3  0.0141 (0.0131)  C>S  Handshake

      ClientKeyExchange

3 4  0.0141 (0.0000)  C>S  ChangeCipherSpec

3 5  0.0141 (0.0000)  C>S  Handshake

3 6  0.0155 (0.0013)  S>C  Alert

    level           fatal

    value           handshake_failure

3    0.0155 (0.0000)  S>C  TCP FIN

3    0.0165 (0.0009)  C>S  TCP FIN

ssldump using Firefox 43 (Passes):

    New TCP connection #1: 192.168.100.125(55099) <-> 192.168.100.231(443)

    1 1  0.0007 (0.0007)  C>S  Handshake

  ClientHello

    Version 3.3 

    cipher suites

    TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

    TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

    TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

    TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA

    TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

    TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

    TLS_DHE_RSA_WITH_AES_128_CBC_SHA

    TLS_DHE_RSA_WITH_AES_256_CBC_SHA

    TLS_RSA_WITH_AES_128_CBC_SHA

    TLS_RSA_WITH_AES_256_CBC_SHA

    TLS_RSA_WITH_3DES_EDE_CBC_SHA

    compression methods

              NULL

    1 2  0.0012 (0.0004)  S>C  Handshake

  ServerHello

    Version 3.3 

    session_id[32]=

      0f 16 00 00 ec 24 3b 75 10 f0 53 c4 45 d3 df ef 

      97 91 f0 9a b8 fe c2 98 5d 15 fd 11 ed 2f 55 58 

    cipherSuite         TLS_RSA_WITH_AES_256_CBC_SHA

    compressionMethod                   NULL

  Certificate

  ServerHelloDone

    1 3  0.0031 (0.0018)  C>S  Handshake

  ClientKeyExchange

    1 4  0.0031 (0.0000)  C>S  ChangeCipherSpec

    1 5  0.0031 (0.0000)  C>S  Handshake

    1 6  0.0053 (0.0022)  S>C  ChangeCipherSpec

    1 7  0.0056 (0.0002)  S>C  Handshake

    1 8  0.2922 (0.2865)  C>S  application_data

    1 9  0.3330 (0.0408)  S>C  Handshake

    1 10 0.3337 (0.0006)  C>S  Handshake

    1 11 0.3368 (0.0031)  S>C  Handshake

    1 12 0.3473 (0.0104)  C>S  Handshake

    1 13 0.3473 (0.0000)  C>S  ChangeCipherSpec

    1 14 0.3473 (0.0000)  C>S  Handshake

    1 15 0.3500 (0.0026)  S>C  ChangeCipherSpec

    1 16 0.3501 (0.0001)  S>C  Handshake

    1 17 0.3512 (0.0011)  S>C  application_data

    1 18 0.3779 (0.0266)  C>S  application_data

    New TCP connection #2: 192.168.100.125(55102) <-> 192.168.100.231(443)

    2 1  0.0008 (0.0008)  C>S  Handshake

  ClientHello

    Version 3.3 

    resume [32]=

      b3 15 00 00 94 41 0f d7 0f ce 39 45 82 5e 53 85 

      b4 4f de 6d 1c f7 23 16 c6 8b bb d6 96 d9 53 c5 

    cipher suites

    TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256

    TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

    TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA

    TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA

    TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA

    TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA

    TLS_DHE_RSA_WITH_AES_128_CBC_SHA

    TLS_DHE_RSA_WITH_AES_256_CBC_SHA

    TLS_RSA_WITH_AES_128_CBC_SHA

    TLS_RSA_WITH_AES_256_CBC_SHA

    TLS_RSA_WITH_3DES_EDE_CBC_SHA

    compression methods

              NULL

    2 2  0.0011 (0.0003)  S>C  Handshake

  ServerHello

    Version 3.3 

    session_id[32]=

      b3 15 00 00 94 41 0f d7 0f ce 39 45 82 5e 53 85 

      b4 4f de 6d 1c f7 23 16 c6 8b bb d6 96 d9 53 c5 

    cipherSuite         TLS_RSA_WITH_AES_256_CBC_SHA

    compressionMethod                   NULL

    2 3  0.0012 (0.0000)  S>C  ChangeCipherSpec

    2 4  0.0018 (0.0006)  S>C  Handshake

    1 19 0.3804 (0.0025)  S>C  application_data

    2 5  0.0025 (0.0006)  C>S  ChangeCipherSpec

    2 6  0.0025 (0.0000)  C>S  Handshake

    2 7  0.0033 (0.0007)  C>S  application_data

    2 8  0.0057 (0.0023)  S>C  Handshake

    2 9  0.0062 (0.0005)  C>S  Handshake

    2 10 0.0072 (0.0010)  S>C  Handshake

    2 11 0.0210 (0.0137)  C>S  Handshake

    2 12 0.0210 (0.0000)  C>S  ChangeCipherSpec

    2 13 0.0210 (0.0000)  C>S  Handshake

    2 14 0.0246 (0.0035)  S>C  ChangeCipherSpec

    2 15 0.0246 (0.0000)  S>C  Handshake

    2 16 0.0250 (0.0003)  S>C  application_data
0
Rate this Question

Answers to this Question

placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

I figured out the issue I was facing. It had to do with TLS Extended Master Secret and the BIG-IP was failing to decrypt the handshake. The extended master secret changes the way pre-master secret is generated for TLS sessions and I suspect BIG-IP fails to detect its presence and calculates the pre-master secret as if extended master secret is not in place, anyways I've written my experience in a couple of blog posts in case someones willing to get into the details: TLS Extended Master Secret Breaking SSL Proxies.

Solution

As for the solution, until BIG-IP adds this feature (decrypting sessions where extended master secret is used) I disabled it on my web server (The threat it was mitigating was minimal in my case when the choice is between having a WAF or having extended master secret enabled, it basically prevents rogue CAs to create bogus certificates and use them to MITM live TLS sessions, more details in the blog post).

Disabling TLS Extended Master Secret in Windows Server/IIS:

For IIS you'd have to go into registry and under SCHANNEL configurations add the following key:

Under HKLM\System\CurrentControlSet\Control\SecurityProviders\Schannel:
Add DisableServerExtendedMasterSecret as REG_DWORD with the value of 1 (anything other than 0 works)
The setting applies immediately and you don't need to restart the server.

0
Comments on this Answer
Comment made 24-Aug-2017 by mfkk531 206

I'm experiencing similar issue after upgrade to v12.1.2 Final.
Any recommendation on how to fix this if the pool members are Apache servers ?

Thanks!

0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

AskF5 Support Article "K34019109: SSL handshakes using TLS extended master secret extensions fail when Proxy SSL is enabled" with this information can be found at https://support.f5.com/csp/article/K34019109

1
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Try setting the SSL logging level to "Debug" on the LTM and see what debug info it logs in the /var/log/ltm for the failure case. It may give more clue. After the testing, make sure to change the logging level back to the original value.

What is the version of IIS server/OS are you using?

0
Comments on this Answer
Comment made 06-Jul-2016 by Saravanan M K
Also try upgrading the box to 12.1-HF1 and see whether the issue still persists.
0
Comment made 06-Jul-2016 by Babak AA 69
I already have Hotfix-BIGIP-12.1.0.1.0.1447-HF1 installed.
0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

ltm logs say:

srv warning tmm[12511]: 01260009:4: Connection error: hud_ssl_handler:1224: codec alert(20)

srv info tmm[12511]: 01260013:6: SSL Handshake failed for TCP 192.168.100.215:443 -> 192.168.100.230:51271

srv info tmm[12511]: 01260013:6: SSL Handshake failed for TCP 192.168.100.230:51271 -> 192.168.100.215:443

I'm using windows server 2012 and IIS 8.

Let me point out again that old web browsers and openssl cli connects through the LTM/ASM correctly but modern browsers fail to do so and get a handshake error. In order to isolate the issue I made sure all my tests are under same protocol/cipher suite (TLS_RSA_WITH_AES_256_CBC_SHA).

I looked at packet captures from BIG-IP's point of view and web servers point of view. The cases where there's a handshake failure alert at BIG-IP, the webserver sees a FIN followed by a RST after client sends change cipher spec and it's the BIG-IP resetting the connection due to handshake failure (I suspect it fails to decrypt the handshake for some reason). But in the case where openssl cli or older browsers are used the BIG-IP happily decrypts the handshake and the connection goes through it.

0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Check if your client-auth certificate, client-ssl certificate and all certificates in CA intermediary certificate bundle are signed with modern SHA algorithms, or if there are any certs signed by the deprecated SHA1. Note that the very root certificate of your CA may still remain SHA1-signed, but any intermediary certificates above it must be SHA256-signed (or stronger). Any of this does not apply to SSL handshakes and SHA in a cipher suite is still OK.

Newer browsers are becoming more and more restrictive with SHA1 certificates and this could be one potential cause to your issue. This OpenSSL command can help you get started:

openssl x509 -text -in '/Path/To/Cert.crt' | grep "Signature Algorithm"

Signature Algorithm: sha256WithRSAEncryption
0
Comments on this Answer
Comment made 06-Jul-2016 by Babak AA 69
Apart from the root certificate which was SHA1, the rest of the certificates where using SHA256. but the fact that the BIG-IP sends a handshake failure and resets the connection has to do something with an issue on BIG-IP side rather than clients browser being unhappy, correct me if this assumption is incorrent.
0
Comment made 07-Jul-2016 by Hannes Rapp 3890
You may be right, but we cannot really tell, as in your own words, there are browsers with which the whole solution works. To further isolate the issue, and eliminate variables, perhaps disabling Client-Auth for a test would be worth it. If the SSL handshake then completes successfully, it can at least be confirmed that it's a SSL-MA problem, and maybe tweaking some settings will do the trick. After all, it's also possible you're looking to solve a problem caused by a bug, so I recommend to open a support case with F5.
0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

You can remove ECDHE-RSA-AES256-CBC-SHA from F5's cipher suite in your client-side SSL profile and see if that makes a difference.

0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

Hello,

I have the same problem, you have a solution or not ? I have this problem since few days.

Thanks.

0
placeholder+image
USER ACCEPTED ANSWER & F5 ACCEPTED ANSWER

No more errors in logs with 12.1.3.

0