Search
Joe Pruitt - A Software Architect's take on Network Security
You are here: DevCentral > Weblogs

posted on Wednesday, August 19, 2009 11:16 AM

For those that don't know what X-Forwarded-For is, then you might as well close your browser because this post likely will mean nothing to you…

A Little Background

Now, if you are still reading this, then you likely are having issues with determining the origin client connections to your web servers.  When web requests are passed through proxies, load balancers, application delivery controllers, etc, the client no longer has a direct connection with the destination server and all traffic looks like it's coming from the last server in the chain.  In the following diagram, Proxy2 is the last hop in the chain before the request hits the destination server.  Relying on connection information alone, the server thinks that all connections come from Proxy2, not from the Client that initiated the connection.

TrafficFlow The only one in the chain here who knows who the client really is (as determined by it's client IP Address, is Proxy1.  The problem is that application owners rely on source client information for many reasons ranging from analyzing client demographics to targeting Denial of Service attacks.

That's where the X-Forwarded-For header comes in.  It is non-RFC standard HTTP request header that is used for identifying the originating IP address of a client connecting to a web server through a proxy.  The format of the header is:

X-Forwarded-For: client, proxy1, proxy, …

X-Forwarded-For header logging is supported in Apache (with mod_proxy) but Microsoft IIS does not have a direct way to support the translation of the X-Forwarded-For value into the client ip (c-ip) header value used in its webserver logging.

Back in September, 2005 I wrote an ISAPI filter that can be installed within IIS to perform this transition.  This was primarily for F5 customers but I figured that I might as well release it into the wild as others would find value out of it.

Recently folks have asked for 64 bit versions (especially with the release of Windows 2008 Server).  This gave me the opportunity to brush up on my C skills.  In addition to building targets for 64 bit windows, I went ahead and added a few new features that have been asked for.

Proxy Chain Support

The original implementation did not correctly parse the "client, proxy1, proxy2,…" format and assumed that there was a single IP address following the X-Forwarded-For header.  I've added code to tokenize the values and strip out all but the first token in the comma delimited chain for inclusion in the logs.

Header Name Override

Others have asked to be able to change the header name that the filter looked for from "X-Forwarded-For" to some customized value.  In some cases they were using the X-Forwarded-For header for another reason and wanted to use iRules to create a new header that was to be used in the logs.  I implemented this by adding a configuration file option for the filter.  The filter will look for a file named F5XForwardedFor.ini in the same directory as the filter with the following format:

[SETTINGS]
HEADER=Alternate-Header-Name

The value of "Alternate-Header-Name" can be changed to whatever header you would like to use. 

Download

I've updated the original distribution file so that folks hitting my previous blog post would get the updates.  The following zip file includes 32 and 64 bit release versions of the F5XForwardedFor.dll that you can install under IIS6 or IIS7.download_buttonInstallation

Follow these steps to install the filter.

  1. Download and unzip the F5XForwardedFor.zip distribution.
  2. Copy the F5XForwardedFor.dll file from the x86\Release or x64\Release directory (depending on your platform) into a target directory on your system.  Let's say C:\ISAPIFilters.
  3. Ensure that the containing directory and the F5XForwardedFor.dll file have read permissions by the IIS process.  It's easiest to just give full read access to everyone.
  4. Open the IIS Admin utility and navigate to the web server you would like to apply it to.
  5. For IIS6, Right click on your web server and select Properties.  Then select the "ISAPI Filters" tab.  From there click the "Add" button and enter "F5XForwardedFor" for the Name and the path to the file "c:\ISAPIFilters\F5XForwardedFor.dll" to the Executable field and click OK enough times to exit the property dialogs.  At this point the filter should be working for you.  You can go back into the property dialog to determine whether the filter is active or an error occurred.
  6. For II7, you'll want to select your website and then double click on the "ISAPI Filters" icon that shows up in the Features View.  In the Actions Pane on the right select the "Add" link and enter "F5XForwardedFor" for the name and "C:\ISAPIFilters\F5XForwardedFor.dll" for the Executable.  Click OK and you are set to go.

I'd love to hear feedback on this and if there are any other feature request, I'm wide open to suggestions. The source code is included in the download distribution so if you make any changes yourself, let me know!

Good luck and happy filtering!

-Joe



Feedback

8/19/2009 10:38 PM
Gravatar Thanks for sharing. This is a common topic that comes up when I converse w/ customers about their approach to site analytics. Any licensing restrictions on the usage of the DLLs?
Catherine Liao
8/19/2009 11:40 PM
Gravatar No, I think the thanks go to you Catherine for linking on your appgirl.net blog.

There are absolutely no licensing restrictions on the filter. I've provided the filter along with the source AS-IS so use it to your hearts content. The only thing I ask is that you don't blame me if something goes wrong B-).
Joe Pruitt
8/23/2009 4:00 AM
Gravatar Huge thanks!
Grzegorz
9/1/2009 8:04 AM
Gravatar Thanks a lot, this is coming in very handy!
Ernest Mueller
9/2/2009 7:41 AM
Gravatar Has anyone used this new filter with IIS 5? I cannot seem to make it work on IIS 5 and I cannot find the old one here.
Stephen Brown
11/16/2009 4:22 PM
Gravatar Hello: This is an amazing piece of work. It solves one of the biggest problems that clients have with SNATs. Also it is very light weight and makes no impact to performance, atleast in my case.

Appreciate the good work!!
Chenna
11/16/2009 11:04 PM
Gravatar Thanks Chenna, glad it's working out for you!

-Joe
Joe Pruitt
11/17/2009 12:13 PM
Gravatar Hi there. We have run into an issue with analytics after putting a the front end servers in our SharePoint farm behind an F5. Since then our web logs have not showed the true client IP address, which has caused an issue for our marketing department with analytics. I followed your instructions to install the ISAPI on our servers, and everything shows green like it is working but the logs are still displaying the IP of the F5 instead of the clients. Is there anything else that needs to be done in order for this to work? Thanks for any assistance.
Tray Harrison
11/17/2009 1:28 PM
Gravatar Sorry Joe, I meant to add....does something need to be enabled on the F5 in order for this to work? I'm just the web server guy, so if something needs to be enabled on the network device then I'll need to let that team know. Thanks again!
Tray Harrison
11/17/2009 2:07 PM
Gravatar Hey Tray, you will need to make sure that you have the X-Forwarded-For header passed through. The filter will look for the existence of that header and then replace the "c-ip" IIS log value with that supplied HTTP header. Depending on the version of BIG-IP you are using, this should be an option in the virtual server. If not, then you'll have to write an iRule to inject that header. Let me know if you need to go that route and I'll help you out.

-Joe
Joe Pruitt
11/17/2009 3:00 PM
Gravatar @Stephen - There is nothing IIS specific in the filter so I don't know why it wouldn't work with IIS5. It's just a Win32 .dll with a few exported functions. In the latest build I put up I've compiled 32 and 64 bit versions but that's the only difference. I've seen some situations where file system permissions cause the filter to not be able to be loaded. What issue are you having?

-Joe
Joe Pruitt
11/23/2009 9:18 AM
Gravatar Joe,

Any chance this will eventually move from an ISAPI filter to an IIS module for IIS7/7.5?

I ask because we are eliminating ISAPI plug-ins on our Windows 2008/R2 farms based in part on this guidance:
mvolo.com/.../...elopment-is-better-with-IIS7.aspx

EDF
11/23/2009 9:57 AM
Gravatar I'll take a look. In the meantime, the source is available if anyone wants to take a stab at it.

-Joe
Joe
11/23/2009 9:58 AM
Gravatar Joe,

Any idea if there are issues when IIS7 is in "integrated pipeline mode"? I've had reports there are issues with ISAPI filters and this mode option.
Wes Robertson
11/23/2009 10:06 AM
Gravatar No clue Wes. Maybe going the module route will avoid those issues. If anyone sees issues with the XFF filter and IIS7 Integrated pipeline mode, let me know...

-Joe
Joe
11/30/2009 8:11 PM
Gravatar Tested the 64 bit version with IIS7 in integrated pipeline mode, works fine.
Wes Robertson
12/23/2009 1:00 PM
Gravatar X-Forwarded-For HTTP Module For IIS7, Source Included!
Joe Pruitt
12/27/2009 12:54 AM
Gravatar Thanks, I would like to test it soon.
slomah
12/2/2009 10:21 AM
Gravatar For those interested, be on the lookout soon for a native IIS XFF module. Working on packaging right now.
Joe
12/9/2009 11:16 AM
Gravatar For those interested, I've got a native IIS7 Http Module that implements the features of the X-Forwareded-For ISAPI filter. Contact me directly at Joe -at- F5 dot com and I'll send you the bits to test out.

-Joe
Joe
2/2/2010 2:25 PM
Gravatar Will this work with SSL connections? We have BigIP in front of Exchange 2007 OWA/CAS which requires SSL so we are not terminating the cert on the BigIP, we are passing traffic through and the cert is on the CAS. I have the older version of the ISAPI filter installed but its still reporting the proxy's IP.
Sandy
2/4/2010 5:05 PM
Gravatar This will work as long as the X-Forwarded-For header is passed through to the web server. Since you aren't terminating SSL connections on the BIG-IP, you can't inject it there.

-Joe
Joe
4/6/2010 2:29 AM
Gravatar Hello, I have installed F5X Module but still do not see source IP in the W3SVC1 log files. Could anybody help please?

thank you
Pavel
3/3/2010 1:56 PM
Gravatar I have the old version running for some time successfully.

I installed the newer version because I want to use an alternate header.

but the newer version does not write any ip info (no +...) in the log file.

this is even without a ini file custom header.

mendel

P.S. my apologies if this is a double post, but I did not get any indication that my post attempt was successful.
mendel
4/5/2010 12:04 AM
Gravatar Is it safe to copy F5XForwardedFor.dll in "\inetpub\wwroot\" directory on IIS and enable ISAPI Filter ?

Thanks & regards,

Prasad.
Prasad Kulkarni
4/5/2010 8:41 AM
Gravatar @Mendel, not sure what changed between the older and newer versions. The source is available if you want to take a look.

@Prasad, shouldn't be an issue as long as IIS has read access on the filter.
Joe
4/16/2010 1:52 AM
Gravatar Hi,


I have a F5 machine, configured to use snat, connected to a ISA Server working as Web Publication.

In the ISAS logs the F5 IP is saved and I would like to save the client IP that is in the x-forwarded-for header.

It's possible to do this with the IIS x-forwarderd-for ISAPI?


Thanks.
Jon
4/16/2010 2:23 AM
Gravatar I have seen the Pure and Tilt in the USA, but is the HD2 even available in the USA from any of the carriers? I know you can get it at NewEgg.com and other for $800+ but I don't believe any carrier in the USA even has them at the moment...so I assume when it does officially launch in with be as a WinMo7 phone.....
jouer gratis
4/22/2010 8:52 AM
Gravatar Hi,
Done all the above (i added also the ini file).
does not work for me.

please help
Michael
5/13/2010 11:58 PM
Gravatar I am glad to come across this posting. I have had some difficulties in getting the Client IP address. As it's currently, the Weblogic access.log it's only showing the load balancer's addresses. I found your website perfect for my needs. It contains wonderful and helpful posts.
giochi del casinò gratuiti
5/18/2010 12:42 AM
Gravatar Hi,

our customer is using Joes ISAPI filter and is experiencing some problems. Quite often the wrong IP address gets logged to the IIS log. It seems to happen when objects get requested which do not exist (e.g. image.giff instead of image.gif). If such a request happens, the IP address of the request just happened before gets logged, instead of the IP address from the current request. Sometimes it even logs the BigIP source address (which is one from the SNAT pool)

We already figured out that the BigIP is doing everything fine, tcpdumps show that the BigIP is inserting X-Forwarded-For in the right way, so the problem happens on the server / filter side.

Does anyone have an idea? the Server version is:
MS Windows 2003 enterprise english with internet information server 6.

isapi filter for loging is F5XForwardedFor.dll with following versionsinfos:

file version 1.0.0.1
descr: F5 HTTP X-Forwarded-For proxy ISAPI Logging Filter
copyright: Copyright (c) 1996-2005 F5 Networks, Inc.
product version: 9, 2, 0, 0

Please help if possible

Many thanks
Roman
Roman
6/17/2010 2:29 PM
Gravatar Joe,

We are planing on using this ISAPI filter so we can enable SNAT on our LTM's, but our applications require the client IP address. I'm new to ISAPI, and not a developer, so I don't completely understand what is happening here, but everything in your posts says this filter will put the client IP in the IIS log. My developers tell me that they need the client IP in the "server variables" remote_addr property. Will this filter do that, or do I need something else?

Thanks

Shawn
Shawn
6/17/2010 3:01 PM
Gravatar Shawn, the filter doesn't modify any inbound variables. On the inbound stream in IIS before it get's to the web application, it stores the value contained in the "X-Forwarded-For" HTTP header value (if it exists). Then after the server responds and IIS is about to add an entry into the IIS HTTP Logs, it replaces the value of the internal "c-ip" value (client ip) with the value contained in the X-Forwarded-For header. the REMOTE_ADDR server variable isn't a HTTP header, it's a value IIS sets based on the clients address that established the connection (in this case the LTM). As far as I know, the REMOTE_ADDR server variable is read-only from the context of a filter.

If you are flowing through a BIG-IP, then you can always write an iRule that creates a custom HTTP header containing the value of the client's address on the virtual but it won't replace the REMOTE_ADDR variable because, as I said, that isn't a HTTP header, just a server variable inside of IIS.

Hope this helps...

-Joe
Joe
6/22/2010 9:46 PM
Gravatar Hi Joe,

One of our customers is using your XFF logging filter.

They claim that two IP adresses are being logged after applying our CDN service.

They say the version number is 1.0.0.1, but I'm not sure if they're using the latest version. Is 1.0.0.1 the latest version currently?

Thanks!

- Charlie
Charlie
6/22/2010 10:34 PM
Gravatar Oh, let me add one more thing.

The size of DLL file they're using is 45,056 bytes, which looks different from that of this version.

Thanks!
Charlie
6/26/2010 4:30 AM
Gravatar published an ISAPI filter to work with our BIG-IP's and have just updated it with full x86 and x64 support. Check out the blog post I wrote on it a while ago. Just copy the enclosed F5XForwardedFor.dll to your server and reference it in the ISAPI section of your virtual site...
Bestes Internet Kasino
10/10/2010 12:40 PM
Gravatar Is there any solution to the TMG server to allow it to read the client IP address coming from earlier proxy.
Ahmed Nabil
11/3/2010 1:24 PM
Gravatar Joe,
can we change the location of the F5LogHeader.log file? If not, where can we find the config for it (we'd like to get it off of the C:\drive as it's filling up pretty quickly).

Some IT Guy
11/4/2010 8:37 AM
Gravatar How can we turn off the logging or at least the log location of the log?
Jim
11/29/2010 4:45 AM
Gravatar Thanks a lot, it works fine.

Vince
vince
12/13/2010 3:36 AM
Gravatar This filter is very useful and we have applied in all the web servers (both 32 and 64 bit). It works in all except one case where 64 bit IIS7 is being used. Actually on another server with 64 bit OS it is working correctly.

The server where it is not working is used for Exchange 2007 OWA. We terminate the SSL on F5-LTM and F5-LTM in turn will create a new SSL connection to the Exchange web server. By decrypting the tcp-dump traffic towards the server we have confirmed that F5-LTM is sending the correct client IP address as X-Forwarded-For tag on the http header. But on the IIS7 log files we are getting either the IP address of the F5-LTM or 127.0.0.1 as the client IP address. Any help or suggestion is very much appreciated in order to get the correct client IP address on the IIS7 log files.
Rajendran
1/14/2011 12:38 PM
Gravatar Odd; it seems that this won't work for me. It stays at "*unknow* state.

There are some articles on MSDN that refer to "flags not being set' because the user might not have permissions.

True, this isapi worked under the default user, but now that the app pool runs under a different user, the isapi stays at *unknown* status and the client IP is not there. I did a TCPDump and I can see that my LTM has injected the x-forwarded-for client IP.

Has anyone run into a fix for this? To where does the user need permissinos to write to that flag, if that's the hurdle?
Curious,
Steve in SD
Steven Scarbrough
2/4/2011 9:41 AM
Gravatar I'm running into an odd situation where I have multiple web sites hosted on my .NET farm (Win 2k3, IIS6) - and the ISAPI filter is installed at the web server level. Log files for some of the web sites are reflecting the c-ip values correctly; others are just showing the backend IP address of the LTM. Has anyone run into this?

Also, I would love to get the debug version of this DLL working to create the F5LogHeader.log file, but so far I haven't been able to build a version that creates the log file. Has anyone successfully generated the .log file?

Thanks,
Jen
Jen
2/11/2011 1:42 PM
Gravatar It seems that on IIS6 this works well only when the default (network service) user is running the app pool. If one uses a domain\user, one gets web info, URI, but still doesn't record the IP of origin. Is there a way to keep the domain\user and get the client IP address as well?

Curious,
Steve in SD
Steve Scarbrough
2/17/2011 1:51 PM
Gravatar Steve, thanks for posting that - we are running into the exact same scenario! We have a great deal of application pools on our cluster, and the ones that run under Network Service have their client IP logged correctly; all the domain user application pools are recording the backend of the LTM. Any ideas?

Thanks,
Jen
Jen
2/25/2011 9:02 AM
Gravatar We found the problem - using the Metabase Explorer tool from the IIS resource kit, we found that the AD group containing the domain accounts did not have permissions to the ISAPI filter DLL. Once we added read/write permissions and reset IIS, all was well. Hope that helps!
Jen
3/31/2011 10:59 AM
Gravatar I've been using the old version on a 32-bit OS with no problems. I just recently had the need to install the filter on 64-bit Windows Web Server 2008 R2 with SP1. But it won't work. I get it installed, but it won't show the true client IP in the logs. Does anybody else have it working with IIS 7.5? I put it in the windows/system32/inetsrv folder, so there shouldn't be a rights issue. Any thoughts?
Mark
4/26/2011 2:33 AM
Gravatar Is it possible also apply X-Forwarded-For ip to IIS "IP address and domain name restrictions"?
hshh
5/13/2011 6:28 AM
Gravatar I have installed both 32 and 64 bit successfully on several of our F5 load-balanced web farms. I am having an issue with the current install. It is 2003 Sp2 32 bit. 2 of the 3 nodes check out fine. On node 3, the IIS mgr ISAPI Filters tab still shows the X-Forwarded-For as Priority *Unknown* with no status.
The logs indicate that the filter is working because client IPs are being passed thru.
I have added and removed the filter several time, reset IIS, downloaded another file etc...
Any ideas?
Marino
5/19/2011 1:48 PM
Gravatar Hi,
I'm having problems with my network and IIS is logging one fixed ip and not the client ip. I tried to use this ISAPI module but I think it's not working. How can I enable debbuging for this dll?

stackoverflow.com/.../cannot-get-user-ip-addres...
HasanG
8/26/2011 12:33 AM
Gravatar It is working great in a setup using nginx as reverse proxy and Win2k3/IIS, thank you for sharing your work with us!
Cavaleru Ioan
9/5/2011 6:42 AM
Gravatar Hi,

I installed this filter, it works fine but I've a strange problem.

In the IIS log file, I've a mixed of my loadbalancer IP (192.168.0.3) but also the official IP (xxx.xxx.xxx.xxx), for example:

2011-09-05 13:33:21 192.168.0.61 GET /v8/ - 80 - 87.67.52.233 Opera/9.80+(Macintosh;+Intel+Mac+OS+X+10.7.1;+U;+en)+Presto/2.9.168+Version/11.51 302 0 0 93
2011-09-05 13:33:21 192.168.0.61 GET /v8/login.cfm - 80 - 192.168.0.3 Opera/9.80+(Macintosh;+Intel+Mac+OS+X+10.7.1;+U;+en)+Presto/2.9.168+Version/11.51 200 0 0 140
2011-09-05 13:33:21 192.168.0.61 GET /CFIDE/scripts/cfform.js - 80 - 192.168.0.3 Opera/9.80+(Macintosh;+Intel+Mac+OS+X+10.7.1;+U;+en)+Presto/2.9.168+Version/11.51 200 0 0 46
2011-09-05 13:33:21 192.168.0.61 GET /CFIDE/scripts/masks.js - 80 - 192.168.0.3 Opera/9.80+(Macintosh;+Intel+Mac+OS+X+10.7.1;+U;+en)+Presto/2.9.168+Version/11.51 200 0 0 15
2011-09-05 13:33:21 192.168.0.61 GET /v8/styles.css - 80 - 192.168.0.3 Opera/9.80+(Macintosh;+Intel+Mac+OS+X+10.7.1;+U;+en)+Presto/2.9.168+Version/11.51 200 0 0 31

Do you have an idea?

Thanks for your help.

Regards, Christophe
Christophe
9/6/2011 6:00 AM
Gravatar Hi,

I found a solution, I must modify the config of my loadbalancer.

Christophe
Christophe
11/29/2011 9:17 PM
Gravatar Hi,
I had configured XFF successfully earlier and also got the log reports as required, now we have moved to https from http. Since the changes i am not able to get the logs it only shows the C-ip address of the load balancer. Can anyone help me out with this.

Thanks,
Santosh
Santosh

Let Me Know What You Think


Please use the form below if you have any comments, questions, or suggestions.

Title:
 
Name:
 
Email: (so we can show your gravatar)
Website:
Comment: Allowed tags: blockquote, a, strong, em, p, u, strike, super, sub, code
 
Please add 6 and 5 and type the answer here:

Blog Stats

Posts:379
Comments:1066
Stories:1
Trackbacks:301
  

Article Categories

  iRules
  

Image Galleries

  

Joe's bookshelf: read

The Lost Gate
4 of 5 stars
This one started slow but I got really got into it about 1/3 of the way through. If you are an Ender's Game fan, you'll probably like this one as well.

goodreads.com


82,243 Members in 102 Countries and Growing!

Join DevCentral Today!

About DevCentral

DevCentral has been a successful, thriving community for many years. We have always strived to bring you the best technical documentation, discussion forums, blogs, media and much more that we can.

So dive in, get familiar with DevCentral. We hope you like it, we hope it makes your job easier, and lets you get that much more power out of the community. To learn more, make sure to check out the Getting Started section. And if you have any problems, or think something could be easier to use, drop us a line to let us know.

Got It !

We've received your comment and transmitted it directly to DevCentral HQ.

Thanks for taking time to let us know what's on your mind. At DevCentral | Community Matters!

Get In Touch With Us

Have questions, suggestions or just want to get something off your chest?

Use our handy form below to Direct Connect with DevCentral Mission Control.

Send Us Feedback       or