Forum Discussion

RiverFish's avatar
RiverFish
Icon for Altostratus rankAltostratus
Aug 16, 2017

App auth breaks when put behind F5

We have an internal IIS 7 site that uses kerberos for auto login. When we put it behind the F5 and point DNS to the VIP the seamless kerberos auth breaks (there's just the one server in the pool). Is this normal? Is APM the only solution? Obviously we want to keep the auth seamless. Thanks for any advice.

 

13 Replies

  • Hi,

     

    You may have several issues here :

     

    • Invalid SPN configuration,
    • Kerberos token too big (> 32 Kb),
    • ...

    What is the behavior observed when putting the bigip device in front of the application ?

     

    If you start a packet capture or an HTTP trace on the browser (Burp, httpwatch, fiddler, Developer tools, ...), you may see if you get a response 401, a tcp reset, a basic fallback or some other things you may find.

     

    APM module is required when you require to terminate Kerberos authentication on the bigip device which is not the use case you described. So APM is not required in your situation

     

    Yann

     

    • RiverFish's avatar
      RiverFish
      Icon for Altostratus rankAltostratus

      Sorry for the delay. Started a new job and all I have is Chrome tools (chrome://net-internals). The behavior observed when putting the bigip in front of the app is we get a login dialog box. Below is the conversation:

      925: URL_REQUEST
      https://i2ddev.ab.abc.com/
      Start Time: 2017-08-17 14:19:33.657
      t=120636 [st=   0] +REQUEST_ALIVE  [dt=?]
                          --> priority = "HIGHEST"
                          --> url = "https://i2ddev.ab.abc.com/"
      t=120636 [st=   0]    DELEGATE_INFO  [dt=1]
                            --> delegate_blocked_by = "NavigationResourceThrottle"
      t=120637 [st=   1]    URL_REQUEST_DELEGATE  [dt=0]
      t=120637 [st=   1]    URL_REQUEST_START_JOB  [dt=0]
                            --> load_flags = 37120 (MAIN_FRAME_DEPRECATED | MAYBE_USER_GESTURE | VERIFY_EV_CERT)
                            --> method = "GET"
                            --> url = "https://i2ddev.ab.abc.com/"
      t=120637 [st=   1]   +URL_REQUEST_START_JOB  [dt=?]
                            --> load_flags = 37120 (MAIN_FRAME_DEPRECATED | MAYBE_USER_GESTURE | VERIFY_EV_CERT)
                            --> method = "GET"
                            --> url = "https://i2ddev.ab.abc.com/"
      t=120637 [st=   1]      URL_REQUEST_DELEGATE  [dt=0]
      t=120637 [st=   1]      HTTP_CACHE_GET_BACKEND  [dt=0]
      t=120637 [st=   1]      HTTP_CACHE_OPEN_ENTRY  [dt=1]
                              --> net_error = -2 (ERR_FAILED)
      t=120638 [st=   2]      HTTP_CACHE_CREATE_ENTRY  [dt=0]
      t=120638 [st=   2]      HTTP_CACHE_ADD_TO_ENTRY  [dt=0]
      t=120638 [st=   2]     +HTTP_STREAM_REQUEST  [dt=154]
      t=120638 [st=   2]        HTTP_STREAM_JOB_CONTROLLER_BOUND
                                --> source_dependency = 928 (HTTP_STREAM_JOB_CONTROLLER)
      t=120792 [st= 156]        HTTP_STREAM_REQUEST_BOUND_TO_JOB
                                --> source_dependency = 929 (HTTP_STREAM_JOB)
      t=120792 [st= 156]     -HTTP_STREAM_REQUEST
      t=120792 [st= 156]     +HTTP_TRANSACTION_SEND_REQUEST  [dt=0]
      t=120792 [st= 156]        **HTTP_TRANSACTION_SEND_REQUEST_HEADERS**
                                --> GET / HTTP/1.1
                                    Host: i2ddev.ab.abc.com
                                    Connection: keep-alive
                                    Upgrade-Insecure-Requests: 1
                                    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36
                                    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
                                    Accept-Encoding: gzip, deflate, br
                                    Accept-Language: en-US,en;q=0.8
                                    Cookie: [76 bytes were stripped]
      t=120792 [st= 156]     -HTTP_TRANSACTION_SEND_REQUEST
      t=120792 [st= 156]     +HTTP_TRANSACTION_READ_HEADERS  [dt=481]
      t=120792 [st= 156]        HTTP_STREAM_PARSER_READ_HEADERS  [dt=480]
      t=121272 [st= 636]        **HTTP_TRANSACTION_READ_RESPONSE_HEADERS**
                                --> HTTP/1.1 401 Unauthorized
                                    Content-Type: text/html
                                    Server: 
                                    WWW-Authenticate: Negotiate
                                    WWW-Authenticate: NTLM
                                    strict-transport-security: max-age=31536000
                                    X-Frame-Options: SAMEORIGIN
                                    X-Content-Type-Options: nosniff
                                    X-XSS-Protection: 1; mode=block
                                    X-UA-Compatible: IE=edge
                                    Date: Thu, 17 Aug 2017 17:19:34 GMT
                                    Content-Length: 1293
      t=121273 [st= 637]        AUTH_CHANNEL_BINDINGS
      t=121273 [st= 637]     -HTTP_TRANSACTION_READ_HEADERS
      t=121273 [st= 637]      URL_REQUEST_DELEGATE  [dt=0]
      t=121273 [st= 637]      URL_REQUEST_DELEGATE  [dt=0]
      t=121273 [st= 637]      HTTP_TRANSACTION_DRAIN_BODY_FOR_AUTH_RESTART  [dt=0]
      t=121273 [st= 637]      HTTP_TRANSACTION_DRAIN_BODY_FOR_AUTH_RESTART  [dt=0]
      t=121273 [st= 637]     +AUTH_SERVER  [dt=537]
      t=121273 [st= 637]       +HOST_RESOLVER_IMPL_REQUEST  [dt=1]
                                --> address_family = 0
                                --> allow_cached_response = true
                                --> host = "i2ddev.ab.abc.com:0"
                                --> is_speculative = false
      t=121273 [st= 637]          HOST_RESOLVER_IMPL_IPV6_REACHABILITY_CHECK
                                  --> cached = true
                                  --> ipv6_available = false
      t=121273 [st= 637]          HOST_RESOLVER_IMPL_CREATE_JOB
      t=121273 [st= 637]          HOST_RESOLVER_IMPL_JOB_ATTACH
                                  --> source_dependency = 931 (HOST_RESOLVER_IMPL_JOB)
      t=121274 [st= 638]       -HOST_RESOLVER_IMPL_REQUEST
      t=121810 [st=1174]     -AUTH_SERVER
      t=121810 [st=1174]     +HTTP_TRANSACTION_SEND_REQUEST  [dt=1]
      t=121810 [st=1174]        **HTTP_TRANSACTION_SEND_REQUEST_HEADERS**
                                --> GET / HTTP/1.1
                                    Host: i2ddev.ab.abc.com
                                    Connection: keep-alive
                                    Authorization: Negotiate [7424 bytes were stripped]
                                    Upgrade-Insecure-Requests: 1
                                    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36
                                    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
                                    Accept-Encoding: gzip, deflate, br
                                    Accept-Language: en-US,en;q=0.8
                                    Cookie: [76 bytes were stripped]
      t=121811 [st=1175]     -HTTP_TRANSACTION_SEND_REQUEST
      t=121811 [st=1175]     +HTTP_TRANSACTION_READ_HEADERS  [dt=281]
      t=121811 [st=1175]        HTTP_STREAM_PARSER_READ_HEADERS  [dt=281]
      t=122092 [st=1456]        **HTTP_TRANSACTION_READ_RESPONSE_HEADERS**
                                --> HTTP/1.1 401 Unauthorized
                                    Content-Type: text/html
                                    Server: 
                                    WWW-Authenticate: Negotiate
                                    WWW-Authenticate: NTLM
                                    strict-transport-security: max-age=31536000
                                    X-Frame-Options: SAMEORIGIN
                                    X-Content-Type-Options: nosniff
                                    X-XSS-Protection: 1; mode=block
                                    X-UA-Compatible: IE=edge
                                    Date: Thu, 17 Aug 2017 17:19:34 GMT
                                    Content-Length: 1293
      t=122092 [st=1456]        AUTH_CHANNEL_BINDINGS
      t=122092 [st=1456]     -HTTP_TRANSACTION_READ_HEADERS
      t=122092 [st=1456]      URL_REQUEST_DELEGATE  [dt=0]
      t=122093 [st=1457]      URL_REQUEST_DELEGATE  [dt=0]
      
    • Yann_Desmarest's avatar
      Yann_Desmarest
      Icon for Cirrus rankCirrus

      It looks like the user provides a Kerberos token but the backend server refuses it.

       

      What is the kerberos configuration you have done on the Active Directory and on the IIS server to make Kerberos works ?

       

      Did you change the DNS name or you just changed the DNS record to point to the VIP ?

       

      What is the configuration of your Virtual Server ?

       

      I saw that NTLM is also supported but require a OneConnect profile and NTLM profile applied on a Virtual Server to make it works through BIG-IP.

       

    • RiverFish's avatar
      RiverFish
      Icon for Altostratus rankAltostratus

      I will have to reach out to the app owner to get the AD and IIS kerb config. Will get back to you. I modified local etc/hosts file so FQDN has not changed. Here is the VIP:

      ltm virtual i2ddev.ab.abc.com-443-vs {
          destination 10.10.10.1:https
          ip-protocol tcp
          mask 255.255.255.255
          pool i2ddev.ab.abc.com-443-pl
          profiles {
              clientssl-i2ddev.ab.abc.com {
                  context clientside
              }
              http { }
              serverssl {
                  context serverside
              }
              tcp { }
          }
          source 0.0.0.0/0
          source-address-translation {
              type automap
          }
          translate-address enabled
          translate-port enabled
          vs-index 6
      }
      
  • Hi,

     

    You may have several issues here :

     

    • Invalid SPN configuration,
    • Kerberos token too big (> 32 Kb),
    • ...

    What is the behavior observed when putting the bigip device in front of the application ?

     

    If you start a packet capture or an HTTP trace on the browser (Burp, httpwatch, fiddler, Developer tools, ...), you may see if you get a response 401, a tcp reset, a basic fallback or some other things you may find.

     

    APM module is required when you require to terminate Kerberos authentication on the bigip device which is not the use case you described. So APM is not required in your situation

     

    Yann

     

    • RiverFish's avatar
      RiverFish
      Icon for Altostratus rankAltostratus

      Sorry for the delay. Started a new job and all I have is Chrome tools (chrome://net-internals). The behavior observed when putting the bigip in front of the app is we get a login dialog box. Below is the conversation:

      925: URL_REQUEST
      https://i2ddev.ab.abc.com/
      Start Time: 2017-08-17 14:19:33.657
      t=120636 [st=   0] +REQUEST_ALIVE  [dt=?]
                          --> priority = "HIGHEST"
                          --> url = "https://i2ddev.ab.abc.com/"
      t=120636 [st=   0]    DELEGATE_INFO  [dt=1]
                            --> delegate_blocked_by = "NavigationResourceThrottle"
      t=120637 [st=   1]    URL_REQUEST_DELEGATE  [dt=0]
      t=120637 [st=   1]    URL_REQUEST_START_JOB  [dt=0]
                            --> load_flags = 37120 (MAIN_FRAME_DEPRECATED | MAYBE_USER_GESTURE | VERIFY_EV_CERT)
                            --> method = "GET"
                            --> url = "https://i2ddev.ab.abc.com/"
      t=120637 [st=   1]   +URL_REQUEST_START_JOB  [dt=?]
                            --> load_flags = 37120 (MAIN_FRAME_DEPRECATED | MAYBE_USER_GESTURE | VERIFY_EV_CERT)
                            --> method = "GET"
                            --> url = "https://i2ddev.ab.abc.com/"
      t=120637 [st=   1]      URL_REQUEST_DELEGATE  [dt=0]
      t=120637 [st=   1]      HTTP_CACHE_GET_BACKEND  [dt=0]
      t=120637 [st=   1]      HTTP_CACHE_OPEN_ENTRY  [dt=1]
                              --> net_error = -2 (ERR_FAILED)
      t=120638 [st=   2]      HTTP_CACHE_CREATE_ENTRY  [dt=0]
      t=120638 [st=   2]      HTTP_CACHE_ADD_TO_ENTRY  [dt=0]
      t=120638 [st=   2]     +HTTP_STREAM_REQUEST  [dt=154]
      t=120638 [st=   2]        HTTP_STREAM_JOB_CONTROLLER_BOUND
                                --> source_dependency = 928 (HTTP_STREAM_JOB_CONTROLLER)
      t=120792 [st= 156]        HTTP_STREAM_REQUEST_BOUND_TO_JOB
                                --> source_dependency = 929 (HTTP_STREAM_JOB)
      t=120792 [st= 156]     -HTTP_STREAM_REQUEST
      t=120792 [st= 156]     +HTTP_TRANSACTION_SEND_REQUEST  [dt=0]
      t=120792 [st= 156]        **HTTP_TRANSACTION_SEND_REQUEST_HEADERS**
                                --> GET / HTTP/1.1
                                    Host: i2ddev.ab.abc.com
                                    Connection: keep-alive
                                    Upgrade-Insecure-Requests: 1
                                    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36
                                    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
                                    Accept-Encoding: gzip, deflate, br
                                    Accept-Language: en-US,en;q=0.8
                                    Cookie: [76 bytes were stripped]
      t=120792 [st= 156]     -HTTP_TRANSACTION_SEND_REQUEST
      t=120792 [st= 156]     +HTTP_TRANSACTION_READ_HEADERS  [dt=481]
      t=120792 [st= 156]        HTTP_STREAM_PARSER_READ_HEADERS  [dt=480]
      t=121272 [st= 636]        **HTTP_TRANSACTION_READ_RESPONSE_HEADERS**
                                --> HTTP/1.1 401 Unauthorized
                                    Content-Type: text/html
                                    Server: 
                                    WWW-Authenticate: Negotiate
                                    WWW-Authenticate: NTLM
                                    strict-transport-security: max-age=31536000
                                    X-Frame-Options: SAMEORIGIN
                                    X-Content-Type-Options: nosniff
                                    X-XSS-Protection: 1; mode=block
                                    X-UA-Compatible: IE=edge
                                    Date: Thu, 17 Aug 2017 17:19:34 GMT
                                    Content-Length: 1293
      t=121273 [st= 637]        AUTH_CHANNEL_BINDINGS
      t=121273 [st= 637]     -HTTP_TRANSACTION_READ_HEADERS
      t=121273 [st= 637]      URL_REQUEST_DELEGATE  [dt=0]
      t=121273 [st= 637]      URL_REQUEST_DELEGATE  [dt=0]
      t=121273 [st= 637]      HTTP_TRANSACTION_DRAIN_BODY_FOR_AUTH_RESTART  [dt=0]
      t=121273 [st= 637]      HTTP_TRANSACTION_DRAIN_BODY_FOR_AUTH_RESTART  [dt=0]
      t=121273 [st= 637]     +AUTH_SERVER  [dt=537]
      t=121273 [st= 637]       +HOST_RESOLVER_IMPL_REQUEST  [dt=1]
                                --> address_family = 0
                                --> allow_cached_response = true
                                --> host = "i2ddev.ab.abc.com:0"
                                --> is_speculative = false
      t=121273 [st= 637]          HOST_RESOLVER_IMPL_IPV6_REACHABILITY_CHECK
                                  --> cached = true
                                  --> ipv6_available = false
      t=121273 [st= 637]          HOST_RESOLVER_IMPL_CREATE_JOB
      t=121273 [st= 637]          HOST_RESOLVER_IMPL_JOB_ATTACH
                                  --> source_dependency = 931 (HOST_RESOLVER_IMPL_JOB)
      t=121274 [st= 638]       -HOST_RESOLVER_IMPL_REQUEST
      t=121810 [st=1174]     -AUTH_SERVER
      t=121810 [st=1174]     +HTTP_TRANSACTION_SEND_REQUEST  [dt=1]
      t=121810 [st=1174]        **HTTP_TRANSACTION_SEND_REQUEST_HEADERS**
                                --> GET / HTTP/1.1
                                    Host: i2ddev.ab.abc.com
                                    Connection: keep-alive
                                    Authorization: Negotiate [7424 bytes were stripped]
                                    Upgrade-Insecure-Requests: 1
                                    User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.90 Safari/537.36
                                    Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
                                    Accept-Encoding: gzip, deflate, br
                                    Accept-Language: en-US,en;q=0.8
                                    Cookie: [76 bytes were stripped]
      t=121811 [st=1175]     -HTTP_TRANSACTION_SEND_REQUEST
      t=121811 [st=1175]     +HTTP_TRANSACTION_READ_HEADERS  [dt=281]
      t=121811 [st=1175]        HTTP_STREAM_PARSER_READ_HEADERS  [dt=281]
      t=122092 [st=1456]        **HTTP_TRANSACTION_READ_RESPONSE_HEADERS**
                                --> HTTP/1.1 401 Unauthorized
                                    Content-Type: text/html
                                    Server: 
                                    WWW-Authenticate: Negotiate
                                    WWW-Authenticate: NTLM
                                    strict-transport-security: max-age=31536000
                                    X-Frame-Options: SAMEORIGIN
                                    X-Content-Type-Options: nosniff
                                    X-XSS-Protection: 1; mode=block
                                    X-UA-Compatible: IE=edge
                                    Date: Thu, 17 Aug 2017 17:19:34 GMT
                                    Content-Length: 1293
      t=122092 [st=1456]        AUTH_CHANNEL_BINDINGS
      t=122092 [st=1456]     -HTTP_TRANSACTION_READ_HEADERS
      t=122092 [st=1456]      URL_REQUEST_DELEGATE  [dt=0]
      t=122093 [st=1457]      URL_REQUEST_DELEGATE  [dt=0]
      
    • Yann_Desmarest_'s avatar
      Yann_Desmarest_
      Icon for Nacreous rankNacreous

      It looks like the user provides a Kerberos token but the backend server refuses it.

       

      What is the kerberos configuration you have done on the Active Directory and on the IIS server to make Kerberos works ?

       

      Did you change the DNS name or you just changed the DNS record to point to the VIP ?

       

      What is the configuration of your Virtual Server ?

       

      I saw that NTLM is also supported but require a OneConnect profile and NTLM profile applied on a Virtual Server to make it works through BIG-IP.

       

    • RiverFish's avatar
      RiverFish
      Icon for Altostratus rankAltostratus

      I will have to reach out to the app owner to get the AD and IIS kerb config. Will get back to you. I modified local etc/hosts file so FQDN has not changed. Here is the VIP:

      ltm virtual i2ddev.ab.abc.com-443-vs {
          destination 10.10.10.1:https
          ip-protocol tcp
          mask 255.255.255.255
          pool i2ddev.ab.abc.com-443-pl
          profiles {
              clientssl-i2ddev.ab.abc.com {
                  context clientside
              }
              http { }
              serverssl {
                  context serverside
              }
              tcp { }
          }
          source 0.0.0.0/0
          source-address-translation {
              type automap
          }
          translate-address enabled
          translate-port enabled
          vs-index 6
      }
      
  • Hi,

    The problem with kerberos is to know which SPN is used to encrypt ticket!

    • Kerberos protocol allows a user to request a ticket provided by Kerberos Distribution Center to access the target resource. The ticket can only be decrypted by the owner of the resource. The owner of a resource is the unique AD element with the servicePrincipalName of the resource.
    • Web services (with or without ssl) Service Principal Names always starts with “http/” pattern. By default, a machine object have servicePrincipalNames host/, host/ and each service type hosted by the machine (ex : IIS  http/ and http/)

    In AD the resource owner can be:

    • Machine hosting web application
    • Web Application service account

    If the resource owner is the machine, there is no solution. if you have 2 servers, you have 2 different resource owner and the client won't know which one will decrypt the token.

    the solution is to be sure the Web application account is the same on the 2 IIS server, have the SPN defined like HTTP/myapp.company.com

    powershell Command to configure it on AD (same as setspn)

    Set-AdUser -Identity svc_app1 -ServicePrincipalNames @{Add="host/myapp.company.local"} 
    

    make sure the web app pool is defined to decrypt token with web app account and no the machine account (option

    useAppPoolCredentials
    )

    Powershell to configure it on IIS server

    Set-WebConfigurationProperty /system.webServer/security/authentication/windowsAuthentication -Name useAppPoolCredentials -value $true -PSPath IIS:\ -Location SITE-myapp.company.com