Forum Discussion

Steven_89435's avatar
Steven_89435
Icon for Nimbostratus rankNimbostratus
Apr 08, 2015

Blackboard ssl-offload using x-forwarded-proto unsuccessful

We are trying to off-load SSL processing to our LTM for Blackboard Learn 9.1.201404.160205 and with the use of the header “X-Forwarded-Proto: https.” To paraphrase, Blackboard documentation states that when the header “X-Forwarded-Proto: https” is provided with an unencrypted session to Blackboard, Blackboard knows the SSL processing has been offloaded upstream and the session between Blackboard and the LTM continues unencrypted.

 

Below is a crude attempt at representing the desired flow: USER -> F5 (dst Port 80)

 

User <- F5 redirect client to use https (dst Port 443)

 

USER -> F5 https (dst port 443)

 

F5 -> (has header X-Forwarded-Proto: https inserted) -> Appserver (dst port 8081 http)

 

F5 <- Appserver (src port 8081 http)

 

USER <- F5 (src port 443 https)

 

What we are experiencing is clients providing the X-Forwarded-Proto: https header connecting to the Blackboard application servers using http are being redirected to the https site of the server by the application server.

 

With the VIP configured to connect to the pool using http and inserted header X-Forwarded-Proto: https, the application server redirects the client to https. The LTM passes this redirect to the user, user connects to https VIP, the VIP inserts X-Forwarded-Proto: https header and connects to application server http, the application server sends a redirect back to client/user,…. until user’s browser presents too many redirects message.

 

F5 support provided an iRule to log header information sent to the application server. Below is the log viewed from the LTM CLI, “X-Forwarded-Proto: https” is being provided.

 

Apr 1 15:35:00 slot1/ocs-vip02 info tmm7[10918]: Rule /Common/Header_logger : Accept-Language: en-US,en;q=0.8

 

Apr 1 15:35:00 slot1/ocs-vip02 info tmm7[10918]: Rule /Common/Header_logger : X-Forwarded-For: aa.bb.cc.dd

 

Apr 1 15:35:00 slot1/ocs-vip02 info tmm7[10918]: Rule /Common/Header_logger : X-Forwarded-Proto: https

 

Apr 1 15:35:00 slot1/ocs-vip02 info tmm7[10918]: Rule /Common/Header_logger :

Apr 1 15:35:04 slot1/ocs-vip02 info tmm[10910]: Rule /Common/Header_logger : Accept-Language: en-US,en;q=0.8

 

Apr 1 15:35:04 slot1/ocs-vip02 info tmm[10910]: Rule /Common/Header_logger : X-Forwarded-For: aa.bb.cc.dd

 

Apr 1 15:35:04 slot1/ocs-vip02 info tmm[10910]: Rule /Common/Header_logger : X-Forwarded-Proto: https

 

Apr 1 15:35:04 slot1/ocs-vip02 info tmm[10910]: Rule /Common/Header_logger : =============================================

 

Apr 1 15:41:05 slot1/ocs-vip02 info tmm7[10918]: Rule /Common/Header_logger : Accept-Encoding: gzip, deflate, sdch

 

Apr 1 15:41:05 slot1/ocs-vip02 info tmm7[10918]: Rule /Common/Header_logger : Accept-Language: en-US,en;q=0.8

 

Apr 1 15:41:05 slot1/ocs-vip02 info tmm7[10918]: Rule /Common/Header_logger : X-Forwarded-For: aa.bb.cc.dd

 

Apr 1 15:41:05 slot1/ocs-vip02 info tmm7[10918]: Rule /Common/Header_logger : X-Forwarded-Proto: https

 

Apr 1 15:41:05 slot1/ocs-vip02 info tmm7[10918]: Rule /Common/Header_logger : =============================================

 

Output from using cURL from the LTM CLI to connect to the Blackboard application server. The output shows the “X-Forwarded-Proto: https” header is being provided, the Blackboard application server is replying with redirect.

 

[] config curl -v -H "X-Forwarded-Proto:https" -H "X-Forwarded-For: aa.bb.cc.dd" * About to connect() to ww.xx.yy.zz port 8081 (0) * Trying ww.xx.yy.zz... connected * Connected to ww.xx.yy.zz (ww.xx.yy.zz) port 8081 (0)

 

GET /webapps/portal/healthCheck HTTP/1.1 User-Agent: curl/7.19.7 (i686-redhat-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8y zlib/1.2.3 libidn/0.6.5 Host: ww.xx.yy.zz:8081 Accept: / X-Forwarded-Proto:https X-Forwarded-For: aa.bb.cc.dd

 

< HTTP/1.1 301 Moved Permanently < Server: Apache-Coyote/1.1 < Location: https://ww.xx.yy.zz/webapps/portal/healthCheck < Content-Length: 0 < Date: Wed, 01 Apr 2015 21:13:07 GMT < Connection: close < * Closing connection 0 [] config

 

Output from using cURL from a windows PC to connect to the Blackboard application server. The output shows the “X-Forwarded-Proto: https” header is being provided, the Blackboard application server is replying with redirect. C:\curl_741_0_ssh2_ssl>curl --http1.1 -v -S - -k -H "X-Forwarded-Proto:https" -H "X-Forwarded-For: aa.bb.cc.dd" * Trying ww.xx.yy.zz... * Connected to ww.xx.yy.zz (ww.xx.yy.zz) port 8081 (0)

 

GET /webapps/portal/healthCheck HTTP/1.1 User-Agent: curl/7.41.0 Host: ww.xx.yy.zz:8081 Accept: / X-Forwarded-Proto:https X-Forwarded-For: aa.bb.cc.dd

 

< HTTP/1.1 301 Moved Permanently < Server: Apache-Coyote/1.1 < Location: https://ww.xx.yy.zz/webapps/portal/healthCheck < Content-Length: 0 < Date: Thu, 02 Apr 2015 15:21:32 GMT < Connection: close < * Closing connection 0

 

Are there any known successfully Blackboard/F5 configurations that take advantage of the F5 SSL offload with the use of the X-Forwarded-Proto header?

 

What could be causing the application server to not accept the F5 VIP, F5 cURL (or windows desktop cURL) provided X-Forwarded-Proto header?

 

Are there any other users experiencing this issue? If so, how is it being addressed? One work around is to encrypt the traffic user to VIP and VIP to application server, no ssl offload.

 

Are there any possible solutions? Successfully configurations for both the LTM and windows/blackboard server are greatly appreciated.

 

application delivery
blackboard
deployment
design
LTM
x-forwarded-proto

9 Replies

Sort By
  • amolari's avatar
    amolari
    Icon for Cirrus rankCirrus
    Apr 08, 2015

    It seems to me that the backend server is doing what you're asking for: it writes its URLs (here 301 status) with https scheme. Do the clients get the https://<> redirection ? Your VS is listening on 443 and has a client ssl profile. Does it loop on all URIs (wouldn't be a bad idea to log the request HOST+URI in your iRule for troubleshooting)?

     

  • Steven_89435's avatar
    Steven_89435
    Icon for Nimbostratus rankNimbostratus
    Apr 08, 2015

    The application server is unwantingly redirecting http to https. The application server is to accept http traffic when the Proto header is present.

     

    The clients get the redirect from the application server resulting in the loop.

     

    The VS/VIP is listening on both 80 and 443. Incoming connections to http:80 on the VIP are redirected by the VIP to https:443 VIP, this is the only redirect that is desired.

     

    Regardlesss of the client (F5 vip, f5 cURL, or windows desktop cURL) the application servers redirect to https. We see the requests and responses in the cURL output.

     

  • amolari's avatar
    amolari
    Icon for Cirrus rankCirrus
    Apr 08, 2015

    the app server performs the redirect to https because you're asking him to, with the x-forwarded-proto: https

    RFC 7239 :

    5.4. Forwarded Proto
   The "proto" parameter has the value of the used protocol type.  The
   syntax of a "proto" value, after potential quoted-string unescaping,
   MUST conform to the URI scheme name as defined in Section 3.1 in
   [RFC3986] and registered with IANA according to [RFC4395].  Typical
   values are "http" or "https".

   For example, in an environment where a reverse proxy is also used as
   a crypto offloader, this allows the origin server to rewrite URLs in
   a document to match the type of connection as the user agent
   requested, even though all connections to the origin server are
   unencrypted HTTP.
  • Steven_89435's avatar
    Steven_89435
    Icon for Nimbostratus rankNimbostratus
    Apr 09, 2015

    Is there a more layman’s version of this? It seems that the purpose of the proto is for the server to edit URLs contained within a served document to the URI format the end client believes it is communicating with. Am I understanding this correctly?

     

    I’m not sure if Blackboard’s use of this header is the same as its intended purpose.

     

    In our case the Blackboard servers are listening http:8081. Connections to http:8081 are redirected before any content is provided. Clients are redirected whether the proto is https or http.

     

    C:\curl_741_0_ssh2_ssl>curl --http1.1 -v -S - -k -H "X-F orwarded-Proto: https" -H "X-Forwarded-For: aa.bb.cc.dd" * Trying ww.xx.yy.zz... * Connected to ww.xx.yy.zz (ww.xx.yy.zz) port 8081 (0)

     

    GET /webapps/portal/healthCheck HTTP/1.1 User-Agent: curl/7.41.0 Host: ww.xx.yy.zz:8081 Accept: / X-Forwarded-Proto: https X-Forwarded-For: aa.bb.cc.dd

     

    < HTTP/1.1 301 Moved Permanently < Server: Apache-Coyote/1.1 < Location: < Content-Length: 0 < Date: Thu, 09 Apr 2015 05:22:41 GMT < Connection: close < * Closing connection 0

     

    C:\curl_741_0_ssh2_ssl>curl --http1.1 -v -S - -k -H "X-F orwarded-Proto: http" -H "X-Forwarded-For: aa.bb.cc.dd" * Trying ww.xx.yy.zz... * Connected to ww.xx.yy.zz (ww.xx.yy.zz) port 8081 (0)

     

    GET /webapps/portal/healthCheck HTTP/1.1 User-Agent: curl/7.41.0 Host: ww.xx.yy.zz:8081 Accept: / X-Forwarded-Proto: http X-Forwarded-For: aa.bb.cc.dd

     

    < HTTP/1.1 301 Moved Permanently < Server: Apache-Coyote/1.1 < Location: < Content-Length: 0 < Date: Thu, 09 Apr 2015 05:22:51 GMT < Connection: close < * Closing connection 0

     

  • Arald_den_Brabe's avatar
    Arald_den_Brabe
    Icon for Nimbostratus rankNimbostratus
    Sep 16, 2015

    Are you sure you have confgured Bb to accept the inserted headers from the F5? in the bb-config.properties file there is a line where you have to specify an regex with the pattern of the IP adresses used by your F5 so Blackboard knows that haeder insertions from these IP's are valid.

     

  • Steven_89435's avatar
    Steven_89435
    Icon for Nimbostratus rankNimbostratus
    Sep 16, 2015

    The issue was resolved. The "web servers" installed with Blackboard did accept connections although the web servers were also configured to redirect despite the x-forwarded-proto intending to stop the redirect. The web servers were changed to not redirect 8081 connections.

     

    • Shawn_OBrien_84's avatar
      Shawn_OBrien_84
      Icon for Nimbostratus rankNimbostratus
      Oct 09, 2015
      Steven, I am going through the same issue that you presented here. Blackboard is saying that the issue is with the F5 and I say otherwise. Just so I'm certain that the iRule I have for X-Forward-Proto is correct for Blackboard, would you mind sharing what you have in place. They say they don't have a known good example. Here's what I have! ltm rule /Common/X-Forwarded-For { when HTTP_REQUEST { if {[HTTP::header exists X-Forwarded-For]}{ HTTP::header replace X-Forwarded-For "[HTTP::header X-Forwarded-For], [IP::client_addr]" } else { HTTP::header insert X-Forwarded-For [IP::client_addr] } } ltm rule /Common/X-Forwarded-Proto { when CLIENT_ACCEPTED { Check the requested port switch [TCP::local_port] { 80 { set proto http } 443 { set proto https } default { Drop the request drop } } }
    • Shawn_OBrien_84's avatar
      Shawn_OBrien_84
      Icon for Nimbostratus rankNimbostratus
      Oct 09, 2015
      My issue, was resolved when they correctly changed their bbconfig to this value bbconfig.frontend.trusted=\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\\\d{1,3}
  • EDHEC-DSI_33880's avatar
    EDHEC-DSI_33880
    Icon for Nimbostratus rankNimbostratus
    Oct 25, 2017

    We've got the same issue with loop redirection of http 8081 in BB Learn. I'm not sure if it's a BB bug or not but BB support tell me to check my LB settings. I try to stop redirect in tomcat server.xml file, but it dosen't change anything :( Steven, if you follow this thread, i would appreciate more details on you solution. Thanks