Forum Discussion

JPMarine_163325's avatar
JPMarine_163325
Icon for Nimbostratus rankNimbostratus
Jun 19, 2018

Load Balancing through another brand load balancer

Hi,

 

We're working on an application that uses NGINX Load Balancers for their DB connections, but F5 (at the moment), for their web front end. The vendor apparently has convinced them to keep NGINX for the DB connections ( I tried ). The problem now is we're injecting an X-Forward header through so the DB can see the connection, but I feel we're still fighting a double NAT. They're getting ready to use the NGINX for the web front end too as the connection is breaking. My guess is the double NAT is doing it, but still not sure. Anyone have experience with these two solutions working together? We're in a pretty flat L2 environment so turning it off isn't an option. Maybe an Irule?

 

Thanks,

 

8 Replies

  • When you say “the connection is breaking”, are you referring to the entire application? I’m not clear on the issue.

     

    From your explanation, the first thing that jumps out is the nginx box may not be configured to pass the x-forwarded-for header to the backend dB servers. Even if they decide to put nginx in front of the webservers instead of the F5, they may encounter similar issues. Every device in the stack has to be aware and able to extract x-forwarded-for headers if app logic is built on the real client IP address.

     

  • Thanks for responding. The application once it gets to the DB is getting DB timeouts unable to connect to the DB. Below is a snippit from the NGINX configuration

     

    Set up X-Forwarded headers so that WildFly interprets the client's IP correctly Note that X-Forwarded-For is set to a specific address, rather than adding the address to a list. This prevents an attack whereby a bogus X-Forwarded-For could be supplied in the initial request (overriding the client's real IP). proxy_set_header X-Forwarded-For $remote_addr;

    proxy_set_header X-Forwarded-Proto https;

     

    Pass on Host header so that upstream servers see the load balancer

    proxy_set_header Host $http_host;

     

    I'm not sure if there's a similar checkbox enabled for the x-forward profile on F5. Just to have an "apples to apples" config. I'll make sure the DB server can accept X-Fwd, but I think that only works for Windows Servers....(I could be wrong). I've just never dealt with two different vendor load balancers in line so to speak in an application.

     

    Thanks,

     

  • X-forwarded-for is typically implemented on any web based application: 99.9% of the time, this means web servers (Apache, IIS, Tomcat).

     

    Doesn’t really matter the number or type of load balancers involved, each connection is a separate distinct session. So if the F5 was load balancing both the web and dB apps, chances are you’d have the same problems if the entire stack isn’t able to handle x-forwarded-for headers.

     

    Have the nginx team create a vip for the web servers. Then use localhost entries for name resolution. Pretty certain they will have similar problems.

     

    I’m a Linux guy so work primarily with MySQL and postgresql. That said, products like OneProxy help address the client IP address problem you’re facing.

     

  • To eliminate the F5 as the culprit, create a local host entry on your workstation so the application url points to a pool member rather than the F5 VIP.

     

  • By the way, what do you have configured for the SNAT setting on the virtual server?

     

  • Thanks, oddly enough, they built a parallel instance with nginx as the web load balancer and db load balancer and it worked for them. The SNAT configuration is with a pool of 4 IP's in the VLAN. We're on 12.1.1. But when they had F5 to Nginx, it didn't work for them, so there's some setting I'm missing somewhere. Below is the configuration of the nginx config they built in parallel to the F5 that was working. The IP's have been changed to protect the innocent. The F5 essentially was LAN/WAN protocol profile, x-forward http profile, cookie persistence, nothing special really. Not sure if you can see anything unique I might be missing. No irules, nothing.

     

    upstream ewb_web { zone upstream_ewb_web 64k;

     

     List the E-WorkBook servers for handling web request (including web services requests)
    server 1.1.1.1:8443;
    server 1.1.1.1:8443;
    
    sticky cookie srv_id path=/;
    
    keepalive 32;
    

    }

     

    upstream ewb_desktop { zone upstream_ewb_desktop 64k;

     

     List the E-WorkBook servers for handling desktop client requests
    server 1.1.1.1:8443;
    server 1.1.1.1:8443;
    

    }

     

    upstream ewb_web_pubsub { zone ewb_web_pubsub 64k;

     

    server 1.1.1.1:8443;
    server 1.1.1.1:8443;
    

    }

     

    upstream ewb_web_ir { zone ewb_web_ir 64k;

     

    server 1.1.1.1:8443;
    server 1.1.1.1:8443;
    ip_hash;
    

    }

     

    We only set the "Connection" header to upgrade if the "Upgrade:" header is present (as it will be for web sockets and EWB Desktop Client connections)

    map $http_upgrade $connection_upgrade { default upgrade; '' ''; }

     

    Set a variable for whether a response code is considered a failure (and will not be cached)

    map $status $status_is_failure { 200 0; 301 0; 302 0; default 1; }

     

    proxy_cache_path /tmp/nginx-ewb-cache keys_zone=ewb-cache:1m inactive=1d; proxy_no_cache $status_is_failure;

     

    proxy_http_version 1.1; proxy_pass_header Server;

     

    Pass on http Upgrade headers (WebSockets/EWB Desktop Client) so that protocol upgrades work.

    proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection $connection_upgrade;

     

    Set up X-Forwarded headers so that WildFly interprets the client's IP correctly Note that X-Forwarded-For is set to a specific address, rather than adding the address to a list. This prevents an attack whereby a bogus X-Forwarded-For could be supplied in the initial request (overriding the client's real IP). proxy_set_header X-Forwarded-For $remote_addr;

    proxy_set_header X-Forwarded-Proto https;

     

    Pass on Host header so that upstream servers see the load balancer

    proxy_set_header Host $http_host;

     

    Default timeouts of 20 minutes

    proxy_read_timeout 20m;

     

    proxy_send_timeout 20m; send_timeout 20m;

     

    server { listen 443 ssl default_server; listen 8443 ssl default_server; server_name x.x.x.x.abc.com; status_zone eworkbook;

     

    ssl_certificate      /etc/nginx/ssl/x.x.x.x.crt;
    ssl_certificate_key  /etc/nginx/ssl/x.x.x.x.key;
    
    ssl_session_cache    shared:SSL:1m;
    ssl_session_timeout  10m;
    
    ssl_ciphers  HIGH:!aNULL:!MD5;
    ssl_prefer_server_ciphers  on;
    
    location = / {
        proxy_pass https://ewb_desktop;
        proxy_read_timeout 1h;
        proxy_send_timeout 1h;
    }
    
    location / {
        proxy_pass https://ewb_web;
        proxy_redirect https://ewb_web $scheme://$host:$server_port;
        proxy_cache ewb-cache;
        client_max_body_size 0;
    }
    
    location /EWorkbookWebApp/pubsub {
        proxy_pass https://ewb_web_pubsub/EWorkbookWebApp/pubsub;
        proxy_buffering off;
        proxy_ignore_client_abort off;
    }
    
    location ~/instruments {
        proxy_pass https://ewb_web_ir;
        proxy_buffering off;
        proxy_ignore_client_abort off;
    }
    
    location /status {
         If required, add allow/deny or password directives to restrict access to this status information
        status;
    }
    
    location = /status.html {
        alias /usr/share/nginx/html/status.html;
    }
    

    }

     

  • Does the web application support x-forwarded-for or does it expect to obtain the client ip via tcp?

     

    When the web app makes calls to the db VIP, nginx will see the IP address of the web app device as the source.

     

    What are the ip addresses of all the requisite pieces:

     

    1. F5 VIP
    2. Web app servers
    3. Db VIP
    4. Db servers
    5. Default gateways of all of the above

    If you have a network diagram, that would be helpful as well.

     

  • Just looked through the nginx config. What are the web app servers using as default gateway?