Forum Discussion

Dec 26, 2013

BIG-IP : redirect loop : debugging techniques

BIG-IP 11.4.0 Build 2384.0 Final

In my hosts file I point

www.mydomain.com

to my web-server IP, and in my client-browser I submit the request

http://www.mydomain.com/natural-language-url-1

and the page is returned as expected..

In BIG-IP , I have configured a single virtual-server

vip-01

with a single iRule

irule-01

when HTTP_REQUEST {
  log local0. "HTTP_REQUEST received by irule-01"
  log local0. "host = [HTTP::host]"
  log local0. "uri = [HTTP::uri]"
  switch -glob [string tolower [HTTP::host]] {
    "www.mydomain.com" {
      switch -glob [string tolower [HTTP::uri]] {
        "/api/*" {
          pool service-pool-01
        }
        default {
          log local0. "matched default pool"       
          pool default-pool-01
        }
      }
    }
  }
}

Now, in my hosts file I point

www.mydomain.com

vip-01

, and in my client-browser I submit the request

http://www.mydomain.com/natural-language-url-1

and the client-browser reports an infinite-redirect error

"This webpage has a redirect loop"

BIG-IP > System > Logs > Local Traffic

Rule /Common/irule-01 : HTTP_REQUEST received by irule-01`
Rule /Common/irule-01 : host = www.mydomain.com
Rule /Common/irule-01 : uri = /natural-language-url-1
Rule /Common/irule-01 : matched default pool

... repeat 10 times ...

Fiddler shows that 21 302-redirects have occurred :

Result          302
Protocol        HTTP
Host            www.mydomain.com
URL             /natural-language-url-1
Body            185
Caching         private
Content-Type    text/html; charset=utf-8
Process         chrome:7516
Comments
Custom

NOTE 1 : on vip-01, Source Address Translation = Auto-Map

NOTE 2 : I have complete control over the web-server ( remote access with ability to modify configuration, run diagnostics, perform local tests, etc. )

NOTE 3 : The web-server expects the request to include the domain. Sending a request with host = {web-server-host-name} , or host = {web-server-ip} will throw an error because IIS will be unable to locate the virtual app.

What are some debugging techniques I can deploy to further diagnose the underlying cause of the redirect loop ?

28 Replies

John_Alam_45640
Historic F5 Account
Dec 26, 2013
The server is redirecting, probably because it does not like something. Check with server admin.

For debugging in any other case use: curl httpwatch tcpdump

In this case these tools will not tell you too much.

Working with server logs is probably best.

HTH
- BaltoStar_12467
  Dec 26, 2013
  Thanks for the response John. Could you please elaborate on how the server could be redirecting ? I don't see how that's possible because in my test with hosts pointing www.mydomain.com to the web-server IP, the redirect loop error does not occur. The error only occurs when BIG-IP is introduced into the picture.
Kevin_Stewart
Employee
Dec 26, 2013
I completely agree with John here. There's absolutely nothing in your iRule that could be generating a redirect. I'd be willing to bet that if you did a tcpdump between the F5 and the web server, you'd see this 302 message coming from the server itself. I know you're generalizing the code for forum view, but is there anything different between accessing directly and through the VIP, other than the host name? If not, try adding this to your code:

HTTP::header replace Host "my-real-server-hostname"
where "my-real-server-hostname" is the name you use when you're accessing the web server directly.
John_Alam_45640
Historic F5 Account
Dec 27, 2013
Do curl -v -H servername.com http://...... To both the virtual and the server. See if either succeeds, and if the results are different.

Post here the entire redirect response So we see the server header.

Check server events log it should tell you why it is redirecting.

It is also possible that the server is not accepting connections from the selfip subnet. You have SNAT automap on the VIP.
- BaltoStar_12467
  Dec 27, 2013
  Thanks John. I'm not sure what you mean by "servername.com" ? The web-server FQDN = "web-server-1.devtest.mydomain.com" , but the F5 BIG-IP VIP is just an IP with no corresponding hostname.
- BaltoStar_12467
  Dec 27, 2013
  Here is curl request direct to web-server : C:\Windows\System32>curl -v -H web-server-1.devtest.mydomain.com www.mydomain.com/natural-language-url-1 |more * About to connect() to www.mydomain.com port 80 (0) * Trying 10.23.0.2... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* connected * Connected to www.mydomain.com (10.23.0.2) port 80 (0) > GET /natural-language-url-1 HTTP/1.1 > User-Agent: curl/7.27.0 > Host: www.mydomain.com > Accept: */* > < HTTP/1.1 200 OK I'm leaving the rest out because it's a few thousand lines of content ( headers, cookies, html ).
- BaltoStar_12467
  Dec 27, 2013
  Here is curl request to virtual-server vip : C:\Windows\System32>curl -v -H 10.25.0.2 www.mydomain.com/natural-language-url-1 |more * About to connect() to www.mydomain.com port 80 (0) * Trying 10.25.0.2... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* connected * Connected to www.mydomain.com (10.25.0.2) port 80 (0) > GET /natural-language-url-1 HTTP/1.1 > User-Agent: curl/7.27.0 > Host: www.mydomain.com > Accept: */* > < HTTP/1.1 302 Found < Date: Fri, 27 Dec 2013 02:42:52 GMT < Server: Microsoft-IIS/6.0 < P3P: CP="CAO PSA OUR" < Etag: < X-mydomainHost: XX-XX-XX-01 < X-Powered-By: ASP.NET < PICS-Label: (PICS-1.0 "http://www.rsac.org/ratingsv01.html" l by "support@mydomain.com" on "2008.12.01T18:21-0800" exp "2035.12.31T12:00-0800" r (v 0 s 0 n 0 l 0)) < X-AspNet-Version: 4.0.30319 < Location: http://www.mydomain.com/natural-language-url-1 < Set-Cookie: lsCKE=ors=otcomqanew; domain=.mydomain.com; path=/ < Cache-Control: private < Content-Type: text/html; charset=utf-8 < Content-Length: 185 < { [data not shown] 100 185 100 185 0 0 2371 0 --:--:-- --:--:-- --:--:-- 2983 * Connection 0 to host www.mydomain.com left intact *Object moved Object moved to here. Closing connection 0
nitass_89166
Noctilucent
Dec 27, 2013
can you run curl on f5 and post the output?

curl -v -H "Host: www.mydomain.com" http://pool_member_ip/natural-language-url-1 curl -v -H "Host: www.mydomain.com" http://virtual_server_ip/natural-language-url-1
- BaltoStar_12467
  Dec 27, 2013
  C:\Windows\System32>curl -v -H "Host: www.mydomain.com" http://10.23.0.2/natural-language-url-1 |more * About to connect() to 10.23.0.2 port 80 (0) * Trying 10.23.0.2... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* connected * Connected to 10.23.0.2 (10.23.0.2) port 80 (0) > GET /natural-language-url-1 HTTP/1.1 > User-Agent: curl/7.27.0 > Accept: */* > Host: www.mydomain.com > < HTTP/1.1 200 OK I'm leaving the rest out because it's a few thousand lines of content ( headers, cookies, html ).
- nitass_89166
  Noctilucent
  Dec 27, 2013
  would you mind running curl on f5 command line please?
- BaltoStar_12467
  Dec 27, 2013
  C:\Windows\System32>curl -v -H "Host: www.mydomain.com" http://10.25.0.2/natural-language-url-1 |more * About to connect() to 10.25.0.2 port 80 (0) * Trying 10.25.0.2... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* connected * Connected to 10.25.0.2 (10.25.0.2) port 80 (0) > GET /natural-language-url-1 HTTP/1.1 > User-Agent: curl/7.27.0 > Accept: */* > Host: www.mydomain.com > < HTTP/1.1 302 Found < Date: Fri, 27 Dec 2013 03:04:10 GMT < Server: Microsoft-IIS/6.0 < P3P: CP="CAO PSA OUR" < Etag: < X-mydomainHost: XX-XX-XX-01 < X-Powered-By: ASP.NET < PICS-Label: (PICS-1.0 "http://www.rsac.org/ratingsv01.html" l by "support@mydomain.com" on "2008.12.01T18:21-0800" exp "2035.12.31T12:00-0800" r (v 0 s 0 n 0 l 0)) < X-AspNet-Version: 4.0.30319 < Location: http://www.mydomain.com/natural-language-url-1 < Set-Cookie: lsCKE=ors=otcomqanew; domain=.mydomain.com; path=/ < Cache-Control: private < Content-Type: text/html; charset=utf-8 < Content-Length: 185 < { [data not shown] 100 185 100 185 0 0 2371 0 --:--:-- --:--:-- --:--:-- 2936 * Connection 0 to host 10.25.0.2 left intact *Object moved Object moved to here. Closing connection 0
nitass
Employee
Dec 27, 2013
can you run curl on f5 and post the output?

curl -v -H "Host: www.mydomain.com" http://pool_member_ip/natural-language-url-1 curl -v -H "Host: www.mydomain.com" http://virtual_server_ip/natural-language-url-1
- BaltoStar_12467
  Dec 27, 2013
  C:\Windows\System32>curl -v -H "Host: www.mydomain.com" http://10.23.0.2/natural-language-url-1 |more * About to connect() to 10.23.0.2 port 80 (0) * Trying 10.23.0.2... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* connected * Connected to 10.23.0.2 (10.23.0.2) port 80 (0) > GET /natural-language-url-1 HTTP/1.1 > User-Agent: curl/7.27.0 > Accept: */* > Host: www.mydomain.com > < HTTP/1.1 200 OK I'm leaving the rest out because it's a few thousand lines of content ( headers, cookies, html ).
- nitass
  Employee
  Dec 27, 2013
  would you mind running curl on f5 command line please?
- BaltoStar_12467
  Dec 27, 2013
  C:\Windows\System32>curl -v -H "Host: www.mydomain.com" http://10.25.0.2/natural-language-url-1 |more * About to connect() to 10.25.0.2 port 80 (0) * Trying 10.25.0.2... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* connected * Connected to 10.25.0.2 (10.25.0.2) port 80 (0) > GET /natural-language-url-1 HTTP/1.1 > User-Agent: curl/7.27.0 > Accept: */* > Host: www.mydomain.com > < HTTP/1.1 302 Found < Date: Fri, 27 Dec 2013 03:04:10 GMT < Server: Microsoft-IIS/6.0 < P3P: CP="CAO PSA OUR" < Etag: < X-mydomainHost: XX-XX-XX-01 < X-Powered-By: ASP.NET < PICS-Label: (PICS-1.0 "http://www.rsac.org/ratingsv01.html" l by "support@mydomain.com" on "2008.12.01T18:21-0800" exp "2035.12.31T12:00-0800" r (v 0 s 0 n 0 l 0)) < X-AspNet-Version: 4.0.30319 < Location: http://www.mydomain.com/natural-language-url-1 < Set-Cookie: lsCKE=ors=otcomqanew; domain=.mydomain.com; path=/ < Cache-Control: private < Content-Type: text/html; charset=utf-8 < Content-Length: 185 < { [data not shown] 100 185 100 185 0 0 2371 0 --:--:-- --:--:-- --:--:-- 2936 * Connection 0 to host 10.25.0.2 left intact *Object moved Object moved to here. Closing connection 0
BaltoStar_12467
Dec 27, 2013
Sorry ok now I get it : you want curl from f5 command line.
- BaltoStar_12467
  Dec 27, 2013
  f5 command line curl direct to web-server pool ip : curl -v -H "Host: www.mydomain.com" http://10.23.0.2/natural-language-url-1 |less * About to connect() to 10.23.0.2 port 80 (0) * Trying 10.23.0.2... connected * Connected to 10.23.0.2 (10.23.0.2) port 80 (0) > GET /natural-language-url-1 HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8y zlib/1.2.3 libidn/0.6.5 > Accept: */* > Host: www.mydomain.com > % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 etc.
- BaltoStar_12467
  Dec 27, 2013
  f5 command line curl to virtual-server vip : curl -v -H "Host: www.mydomain.com" http://10.25.0.2/natural-language-url-1 |less * About to connect() to 10.25.0.2 port 80 (0) * Trying 10.25.0.2... connected * Connected to 10.25.0.2 (10.25.0.2) port 80 (0) > GET /natural-language-url-1 HTTP/1.1 > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8y zlib/1.2.3 libidn/0.6.5 > Accept: */* > Host: www.mydomain.com > % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0< HTTP/1.1 302 Found < Date: Fri, 27 Dec 2013 03:24:06 GMT < Server: Microsoft-IIS/6.0 < P3P: CP="CAO PSA OUR" < Etag: < X-mydomainHost: XX-XX-XX-01 < X-Powered-By: ASP.NET < PICS-Label: (PICS-1.0 "http://www.rsac.org/ratingsv01.html" l by "support@mydomain.com" on "2008.12.01T18:21-0800" exp "2035.12.31T12:00-0800" r (v 0 s 0 n 0 l 0)) < X-AspNet-Version: 4.0.30319 < Location: http://www.mydomain.com/natural-language-url-1 < Set-Cookie: lsCKE=ors=otcomqanew; domain=.mydomain.com; path=/ < Cache-Control: private < Content-Type: text/html; charset=utf-8 < Content-Length: 185 < { [data not shown] 185 185 185 185 0 0 6585 0 --:--:-- --:--:-- --:--:-- 7708* Connection 0 to host 10.25.0.2 left intact * Closing connection 0 Object moved Object moved to here. (END)
nitass_89166
Noctilucent
Dec 27, 2013
you want curl from f5 command line. But how to page the output ? because it's overflowing the putty window - even with |more

you can use -I to request only header (i.e. HEAD method).

e.g.

curl -Iv -H "Host: www.mydomain.com" http://ip_address/natural-language-url-1

based on the output, requests seem to be same but responses are different. is there any specific configuration on server?

would it be possible to not use snat as John asked?
- BaltoStar_12467
  Dec 27, 2013
  I am not a networking engineer, but here is my understanding : This is not an isolated test environment, and to construct one is prohibitively complex & time-consuming. The dev-test F5 ( which I am using to develop the new iRules ) is on the same internal engineering subnet as other non-prod F5s. To be functional, the dev-test website must talk to shared db-servers and web-services, and therefore I have the web-server gateway set to a VIP of one of those other F5s ( otherwise I would have to do a total network configuration on the dev-test F5 - as opposed to just what I need to test natural-language URLs ). I added SNAT to the dev-test F5 virtual-server because the only other option to ensure responses flowed back through the F5 was to add a static route to the web-server, and this was not representative of how it would work in a prod environment. One thing I have noticed is that some of the virtual addresses ( different from those used by the vips the test harness hits ) are red, indicating they are not available, but I don't know why. I am wondering if one of these vips could be the self-ip and this is the underlying cause of the redirect loop error ?
nitass
Employee
Dec 27, 2013
you want curl from f5 command line. But how to page the output ? because it's overflowing the putty window - even with |more

you can use -I to request only header (i.e. HEAD method).

e.g.

curl -Iv -H "Host: www.mydomain.com" http://ip_address/natural-language-url-1

based on the output, requests seem to be same but responses are different. is there any specific configuration on server?

would it be possible to not use snat as John asked?
- BaltoStar_12467
  Dec 27, 2013
  I am not a networking engineer, but here is my understanding : This is not an isolated test environment, and to construct one is prohibitively complex & time-consuming. The dev-test F5 ( which I am using to develop the new iRules ) is on the same internal engineering subnet as other non-prod F5s. To be functional, the dev-test website must talk to shared db-servers and web-services, and therefore I have the web-server gateway set to a VIP of one of those other F5s ( otherwise I would have to do a total network configuration on the dev-test F5 - as opposed to just what I need to test natural-language URLs ). I added SNAT to the dev-test F5 virtual-server because the only other option to ensure responses flowed back through the F5 was to add a static route to the web-server, and this was not representative of how it would work in a prod environment. One thing I have noticed is that some of the virtual addresses ( different from those used by the vips the test harness hits ) are red, indicating they are not available, but I don't know why. I am wondering if one of these vips could be the self-ip and this is the underlying cause of the redirect loop error ?
Kevin_Stewart
Employee
Dec 27, 2013
Does data travel the same path to the shared db and web services when accessing directly or via VIP? If all of that is the same, I'm not sure it matters. Have you determined where the 302 is actually coming from (F5 or web server)?
Kevin_Stewart
Employee
Dec 27, 2013
The web-server gateway IP is set to a VIP on the internal engineering network F5. This F5 routes traffic to a pool of DNS that resolve the hostnames to the appropriate server IPs.

Does that mean that the data path is the same for both? In other words, assuming the web server is talking to another web service and database in relation to a client request, does the web server communicate with these services in the same way regardless of how the client connects to the web server?

How would I determine whether the 302 is originating from the web-server or the F5 ? As I demonstrated above, curl from the F5 command-line direct to the web-server IP returns the entire page content as expected.

Completely understand that, and perhaps I'm grasping at straws here, but in the absence of anything else that might make sense, it would be helpful to know definitely which device is actually sending the redirect. So, if you run a tcpdump on the interface between the f5 and the web server and use the -Xs0 option, you'll see the HTTP payload coming from the web server (assuming it's not encrypted).

tcpdump -lnni [internal interface] -Xs0 [and any additional filters]

This will ultimately tells us where to look for the problem.