X-Forwarded-For HTTP Module For IIS7, Source Included!

posted on Wednesday, December 23, 2009 1:00 PM

For those who of you that are having problems with logging client addresses in their server logs because you are running your web servers behind a proxy of some sort, never fear, your solution is here. For those that don't, I already discussed in my previous posts about what the X-Forwarded-For header is so feel free to click back into those to read about it.

History

Back in September, 2005 I wrote and posted a 32-bit ISAPI filter that extracted the X-Forwarded-For header value and replaced the c-ip value (client ip) that is stored in the server logs. Lots of folks have found this useful over time and I was eventually asked for a 64-bit version which I posted about in August, 2009.

The Question

Well, it looks like it's time for the next generation for this filter… I received an email from a colleague here at F5 telling me that his customer didn't want to deploy any more ISAPI filters in their IIS7 infrastructure. IIS7 introduced the concept of IIS Modules that are more integrated into the whole pipeline and was told that Microsoft is recommending folks move in that direction. I was asked if I had plans to port my ISAPI filter into a HTTP Module.

The Answer

Well, the answer was "probably not", but now it's changed to a "yes"!

The Solution

In reading about IIS Module, I found that you can develop in managed (C#/VB) or Native (C++) code. I loaded up the test C# project to see if I could get it working. In a matter of minutes I had a working module that intercepted the event when logging occurs. The only problem was that from managed code, I could find no way to actually modify the values that were passed to the logging processor. This was a problem so I scrapped that and moved to a native C++ module. After a little while of jumping through the documentation, I found the things I needed and pretty soon I had a working HTTP module that implemented the same functionality as the ISAPI filter.

Download

The new Http Module hasn't had much testing done so please test it out before you roll it out into production. I've made the source available as well if you find an issue and care to fix it. Just make sure you pass back the fixes to me B-).

The filter will require installation into IIS in order for you to be able to add it to your applications. Both distributions include a readme.txt file with an example installation procedure. Make sure you use the Release builds for the appropriate platform (x86 or x64) unless you are in need of some troubleshooting as the Debug build will dump a lot of data to a log file. The module supports customizable headers if you are using something other than X-Forwarded-For. Instructions for using that are in the readme.txt file as well.

If you have any issues with using this, please let me know on this blog. Keep in mind that this is an unsupported product, but I'll do my best to fix any issues that come up. I'm even open to enhancements if you can think of any.

Enjoy!

-Joe

Technorati Tags: F5,DevCentral,IIS7,Modules,ISAPI,X-Forwarded-For,Joe Pruitt

Posted In: DevCentral, F5, Development and General, Microsoft Solutions,

33 Comments

Feedback

4/6/2010 1:55 AM
		Hi, I have problem with installation of this feature, could you help please? C:\HttpModules>> .\Install-XFF.ps1 -cmd install -module_path c:\HttpModules\F5XF FHttpModule.dll '-cmd' is not recognized as an internal or external command, operable program or batch file. thank you
Pavel

3/18/2010 3:22 AM
		Hi Joe, You mention this has had little testing and not to use in production without testing. Can you tell me if you received feedback and if you can approve the module further? James.
James Crawshaw

3/19/2010 8:58 AM
		Joe, One of the IIS team created a module to cover the x-forwarded for header that Microsofts Application Request Routing module puts in to the request. Here is the link, - blogs.iis.net/.../...nt-server-when-using-arr.aspx I will be testing that soon to see if it works for the LTM too. Carl
Carl

3/19/2010 1:11 PM
		@James - I've heard back from a few users that this is working fine. I've been using it internally on my test systems without any problem so I don't think there should be any issue with using it in prod.
Joe

3/19/2010 1:14 PM
		@Carl - Thanks for the pointer. My module will work with our without ARR in place so hopefully those that are using external load balancers will take a look at my module. I've also got the ISAPI filter for IIS6 and below available for folks to use. I've also made source available for all my distributions so anyone can take a look at what I've done. -Joe
Joe

4/9/2010 11:15 AM
		Hi Joe, I have tried both ISAPI filter and HTTP module. I see the following behaviour. Did anyone have resolution. When app pool is running in 64 bit mode (Enable 32-Bit Applications - False), the x64 ISAPI filter works fine. When app pool is running in 32 bit mode (Enable 32-Bit Applications - True), the x64 ISAPI filter does not replace the source IP with X-Forwarded-For. If the x86 version of the ISAPI is applied to a site running in an 32-bit app pool, the app pool will fail and throw 500 error.
Ram

6/3/2010 8:12 AM
		Hi, I installed the module, did an iisreset but I had no luck. I tried the debug release but I do not see any logs. Does the module need to appear in a specific order in the modules list (eg before the httploging module)? Does it need anything else in order to get enabled?
Vangelis

6/30/2010 12:41 PM
		Hi Joe, I am using your module on IIS behind a HAProxy load balancer and it is generally working great. However, sometimes we get the load balancer IP and not the client's IP. I have tested this and it seems to only happen for content that is cached by IIS. Any ideas how how I might fix this? Thank you, Kyle
Kyle Brandt

7/26/2010 9:51 AM
		Thanks for the reply Joe. I am referring to the installation steps in the README file, which say to copy the files to the 'httpmodules' directory, then run the following command to install the module and register it with IIS: .\Install-XFF.ps1 -cmd install -module_path c:\HttpModules\F5XFFHttpModule.dll The above command gives me no output and I do not see the module listed in IIS. Can you tell me how you're installing the module? As for the build, I am using 2008 R2, which is only 64-bit and I am using the 64-bit module. So, that shouldn't be the issue. Thanks for the help!
Kyle G

7/21/2010 9:48 AM
		I have issued the install command and it would seem to have worked (i.e., I get no output). But, I cannot see the module in IIS. Am I missing something? I am running 2008 R2 and using the 64-bit code and installing it via PowerShell. Any help in getting this module configured would be great.
Kyle G

7/21/2010 10:14 AM
		@Kyle B - not sure how to deal with that as I'd have to do some testing to try to simulate the IIS caching and I'm not sure how exactly to do that one. I'm no expert, I just patched this together and it worked for the test cases I had. Let me know if you find a solution to the server side caching and I'll get it integrated into the distribution. @Kyle G - What "install" command were you referring to? I've only tested it with the IIS admin tool and configuring it in there. Also, are you sure you are using the correct build (64-bit). I'd also check that the file on disk has the correct permissions to allow the IIS account to be able to read it. I've seen this issue before that by increasing the permissions on the file allowed it to be loaded into IIS.
Joe

8/25/2010 10:18 AM
		I installed the module and verified that is does log the XForwardedFor on 200 responses. It doesn't seem to work for a 404 request. It logs the load balancer's IP. I'm guessing other 4xx responses have the same behavior.
Ron

10/22/2010 1:32 PM
		I have installed the 64-bit module on 2008 R2/IIS 7.5 behind an F5 with some application pools which run as 32-bit applications. Both the 64-bit app pools and 32-bit app pools are capturing the source IP. I also tested requests which yield a 404, and those requests also capture the source IP. I opted to install the module manually as a Native module.
Eric P

11/2/2010 10:04 AM
		UPDATE - Server 2008 R2/IIS7.5: configuring both the x64 and x86 modules as native modules for the entire web server (thereby inherited by all sites), my tests show both x64 and x86 sites are capturing the source IP.
Eric P

12/2/2010 2:34 PM
		Guys has anybody used this with Amazon EC2 clould using their Elastic Load Balancer? I installed this on my IIS7 x64 it shows up in modules but in the logs I still get the load balancer local address instead of the client outside one... even though Amazon recommends using the X-FORWARDED-FOR. Thanks, remus
Remus

12/14/2010 11:32 PM
		Thanks ;)
Dhiraj Gupta

12/16/2010 9:35 AM
		I am having the same issue as remus. Running 2008 R2 and installed the x86 module but no go. When installing x64 version event log states only x86 version is supported. In event viewer app logs getting garbage. ------------------------------------------------------------------------------------ x64 version message: ------------------------------------------------------------------------------------ The Module DLL 'D:\HTTPModules\F5XFFHTTP\F5XFFHttpModule.dll' could not be loaded due to a configuration problem. The current configuration only supports loading images built for a x86 processor architecture. The data field contains the error number. To learn more about this issue, including how to troubleshooting this kind of processor architecture mismatch error, see http://go.microsoft.com/fwlink/?LinkId=29349. ------------------------------------------------------------------------------------ x86 version message: ------------------------------------------------------------------------------------ IISMANAGER_ERROR_LOADING_PROVIDER_TYPE IIS Manager could not load type 'Microsoft.Web.Deployment.UI.PackagingModuleProvider, Microsoft.Web.Deployment.UI.Server, Version=7.1.2.1, Culture=neutral, PublicKeyToken=31BF3856AD364E35' for module provider 'WebDeployment' that is declared in %windir%\system32\inetsrv\config\administration.config. Verify that the type is correct, and that the assembly that contains the module provider is in the Global Assembly Cache (GAC). Exception:System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Web.Deployment.UI.Server, Version=7.1.2.1, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified. File name: 'Microsoft.Web.Deployment.UI.Server, Version=7.1.2.1, Culture=neutral, PublicKeyToken=31bf3856ad364e35' at System.RuntimeTypeHandle._GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMark& stackMark, Boolean loadTypeFromPartialName) at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMark& stackMark) at System.RuntimeType.PrivateGetType(String typeName, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMark& stackMark) at System.Type.GetType(String typeName, Boolean throwOnError) at Microsoft.Web.Management.Server.AdministrationModuleProvider.GetModuleProvider(String userName, String connectionName) WRN: Assembly binding logging is turned OFF. To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1. Note: There is some performance penalty associated with assembly bind failure logging. To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog]. Process:InetMgr
Eric

12/27/2010 4:00 AM
		keno rules strategy I believe in God so god is control and what happens here is only the smallest glimpse of something far greater that He has in mind for us..
dimpe

2/3/2011 5:45 PM
		You really need to replace powershell based installation instructions with simple shell script below %windir%\system32\inetsrv\appcmd install module /name:F5XFF /image:%CD%\F5XFFHttpModule.dll Just save it with name .bat and put in the same folder as module and execute it. It's much simplier then powershell based method.
Gregory Suvalian

3/4/2011 10:40 AM
		I have the same problem as Eric; it does not work when using applicationpools enabled for 32-bit applications. Does anyone have a solution to this? Regards,
Frans Rampen

3/16/2011 12:29 PM
		After more research, I wrote a PowerShell script that will allow proper installation to handle 32-bit or 64-bit app pools. The key is passing a precondition when you install the module. Using this code, the both the 64-bit and the 32-bit modules will be installed and ready for use, but ONLY the proper module will be activated for 32-bit and 64-bit app pools. This is working like a charm for me...you set it once globally and it will automatically handle all sites on the server, regardless if you change the "bitness" of an application pool. Replace the <name-of-module> to specify the name you want the module to have, and replace the <location-of-module> to point to the fully qualified path of the 32-bit and 64-bit DLLs you copied to your server. c:\windows\system32\inetsrv\appcmd.exe install module /name:<name-of-module> /image:<location-of-module> /preCondition:"bitness32" /commit:apphost c:\windows\system32\inetsrv\appcmd.exe install module /name:<name-of-module> /image:<location-of-module> /preCondition:"bitness64" /commit:apphost Enjoy!
Eric P

4/1/2011 6:36 PM
		Does this only modify the X-Forwarded-For header that is stored in the IIS logs? Or if I use Request.ServerVariables["X-Forwarded-For"], will I also see the new value? What if the request is over SSL?
Jason

7/14/2011 11:59 AM
		I have installed the 64bit module and now the port 80 links pass the client-IP through and is logged the actual client IP. My problem is with port 443. The Cert is installed on both the LTM and the Web Servers. both before and after I installed the X-Forwarded-For HTTP Module I get one of the LTM's Self IPs!! What did I do wrong and/or what can I do to fix this problem with logging the port 443 visits?
Steve Burch

8/8/2011 10:37 AM
		F5XFFHttpModule not completely compatible with WCF We've been using the X-Forwarded-For HTTP Module For IIS7 (F5XFFHttpModule) for over a year without issue. However, some new WCF (.Net 4.0) web services started throwing errors: The server encountered an error processing the request. The exception message is 'Specified value has invalid Control characters. Parameter name: value'. See server logs for more details. The exception stack trace is: at System.Net.WebHeaderCollection.CheckBadChars(String name, Boolean isHeaderValue at System.Net.WebHeaderCollection.Add(String name, String value) at System.Collections.Specialized.NameValueCollection.Add(NameValueCollection c) at System.ServiceModel.Activation.HostedHttpContext.HostedRequestContainer.System.ServiceModel.Channels.HttpRequestMessageProperty.IHttpHeaderProvider.CopyHeaders(WebHeaderCollection headers) at System.ServiceModel.Channels.HttpRequestMessageProperty.get_Headers() at System.ServiceModel.Web.IncomingWebRequestContext.get_IfModifiedSince() It turns out that the new WCF API doesn't like the modification that the F5XFFHttpModule makes to the X-Forward-For header. The F5XFFHttpModule truncates the X-Forward-For Header with null string terminator. This is an invalid control character and will cause WCF to error if it tries to read the request headers: // Check for any additional proxy fields, // ie. X-Forwarded-For: 10.10.10.10, 1.2.3.4, 5.6.7.8, TCHAR sep = _tcschr(pszHeaderValue, _T(',')); if ( NULL != sep ) { sep = _T('\0'); // WCF will choke on this control char
Aheifetz

9/1/2011 3:26 AM
		I have installed this as per the combined instructions (for bitness) and for some reason its's replaced c-ip with the servers IP??? (IIS 7.0/Win2K8) People have mentioned similar issues, but nothing seems to be mentioned about a fix. It's not throwing errors. It's just putting the wrong value. It's running behind CloudFlare (and it was them that put me onto this) I've got compression/caching turned on, but I put the F5 module first. Anyone got any ideas/suggestions/
Malcolm White

9/8/2011 10:51 AM
		I am also having an issue with WCF - same as aheifetz. Has anybody come up with a work around or fix for this issue?
John

10/7/2011 10:39 AM
		Hi everyone, We installed this module on 2008 r2 x64 web servers. We are just using big-ip for load balancing at the moment. We noticed that some client IPs were coming in but many floating IP addresses were also showing in the logs. It appears the issue was related to IIS7.5 caching and the module handling cache hits differently than non-cache hits. I would clear the IIS cache and then hit a few pages and my client ip would show in the IIS logs. I would then force the same pages into the IIS cache by hitting them many times within a few seconds. Once the pages were cached in IIS then the client IP logged was the floating IP of the Virtual Server on big-ip. I disabled all caching in IIS 7.5 by adding an Output Caching entry for * and disabling all caching for that entry. Since then the logs are showing the client IP address.
Ed

10/7/2011 1:39 PM
		Just an update. Rather than add an * entry I tried disabling kernel cache and it seems to resolve the issue with seeing the floating ip (in ha setup) or self-ip (single device) in the IIS logs. Go to your site in IIS Manager -> Output Caching -> edit feature setting -> and uncheck "Enable kernel cache" I know that this might not be ideal solution as some folks may need to cache content for performance reasons, but for us the performance loss is minimal and the importance of having client IP addresses is significant.
Ed

10/11/2011 12:03 PM
		Sorry for posting 3 times in a row, but I have more to add. The disabling of the cache worked except for 404 and 302 that we are getting. I imagine it would happen for other non-200 codes as well. I tried moving the module to the top of the list in IIS 7, but that did not work either. I then delved into the source and read a bit on IIS 7 pipeline and here's what I am thinking. The events that the module are looking is too low in the piple line. In the code the following is found: // step 3: register for server events // TODO: register for more server events here DWORD dwRequestNotifications = RQ_ACQUIRE_REQUEST_STATE \| RQ_SEND_RESPONSE; DWORD dwPostRequestNotifications = 0; According to (learn.iis.net/.../aspnet-integration-with-iis-7...) AcquireRequestState is step 10 of the pipeline. However, the output cache is step 6 and skips other processing. This would explain why turning off cache resolves the issue with the floating ip showing in the IIS logs instead of the x-forwarded-for. I'm still not clear on why the 404 still show the Floating IP, but I would bet it is also related to where in the pipeline the f5 module is getting triggered. Hope this helps.
Ed Lewis

10/21/2011 6:49 AM
		Hi all. We are in the process of implementing a different module. This F5 module was never supported and it appears the individual who wrote it is either inactive in blogs now or no longer with F5. At any rate, we have had some success in our staging environment with a module written by an IIS developer at Microsoft. It's called ARR Helper. We still had to disable kernel caching to get good request to rewrite the client ip. However, unlike the F5 module, the ARR module writes the client IP address for 404s. Additionally it will make the F5 cookie available in the IIS logs as well. You can find the information on the module here: blogs.iis.net/.../...nt-server-when-using-arr.aspx Thanks.
Ed Lewis

10/24/2011 11:23 AM
		Hi all, I'm still here, just have let this post go a while too long. The XFF filter was originally written for us to use with DevCentral and I chose to make the source available to the public. I wrote the IIS Module version in request to those moving to IIS7. The source is available for those that are willing to modify it if you find any problems. Thanks for using it and I'll try to do better at replying to comments here. -Joe
Joe

11/18/2011 2:16 PM
		Joe, is it to be understood that if you're not terminating SSL at the LTM then there is no option for capturing the original source IP or c-ip in your IIS logs?
Steve

12/5/2011 10:46 AM
		Steve - You've probably already found your answer, but try this link: support.f5.com/.../sol4816.html "You can insert HTTP headers in HTTPS traffic only if the client connects to a BIG-IP virtual server configured with a Client SSL profile. When the BIG-IP terminates the SSL connection, it has access to the unencrypted HTTP data." Pretty much if you want to keep the traffic encrypted to the server but still use ASM or add headers like this you'll need to terminate the client side at the big-ip. You can encrypt again on the server side if you have a requirement to have encrypted data on the wire on the server side.
Ed

Let Me Know What You Think

Please use the form below if you have any comments, questions, or suggestions.

Title:

Name:

Email: (so we can show your gravatar)

Website:

Comment: Allowed tags: blockquote, a, strong, em, p, u, strike, super, sub, code

Remember Me?

Enter the code shown above:

Please add 3 and 8 and type the answer here:

		iControl
		WS-I
		BIG-IP
		Misc
		DevCentral
		F5
		iRules
		Conferences
		MIX06
		PDC
		Development and General
		Monitoring/Management
		XML
		Security
		ISV Solutions
		Microsoft Solutions
		Cloud Computing

		October, 2011 (1)
		October, 2010 (2)
		September, 2010 (1)
		July, 2010 (1)
		December, 2009 (1)
		October, 2009 (1)
		September, 2009 (2)
		August, 2009 (5)
		June, 2009 (4)
		May, 2009 (6)
		April, 2009 (13)
		March, 2009 (2)
		February, 2009 (6)
		January, 2009 (16)
		December, 2008 (14)
		November, 2008 (7)
		October, 2008 (17)
		September, 2008 (22)
		August, 2008 (5)
		July, 2008 (2)
		June, 2008 (3)
		May, 2008 (14)
		April, 2008 (17)
		March, 2008 (5)
		February, 2008 (3)
		January, 2008 (4)
		November, 2007 (4)
		September, 2007 (5)
		August, 2007 (2)
		July, 2007 (3)
		June, 2007 (2)
		May, 2007 (3)
		April, 2007 (3)
		January, 2007 (1)
		November, 2006 (2)
		October, 2006 (4)
		September, 2006 (4)
		August, 2006 (6)
		July, 2006 (2)
		June, 2006 (3)
		May, 2006 (4)
		April, 2006 (4)
		March, 2006 (9)
		February, 2006 (5)
		January, 2006 (7)
		December, 2005 (3)
		November, 2005 (4)
		October, 2005 (6)
		September, 2005 (7)
		August, 2005 (6)
		July, 2005 (18)
		June, 2005 (7)
		May, 2005 (7)
		April, 2005 (5)
		March, 2005 (3)
		February, 2005 (10)
		January, 2005 (4)
		December, 2004 (8)
		November, 2004 (15)
		October, 2004 (8)
		September, 2004 (5)
		August, 2004 (2)
		June, 2004 (1)
		April, 2004 (2)
		March, 2004 (3)
		January, 2004 (2)
		November, 2003 (4)
		October, 2003 (2)

		[f5] Colin Walker
		[f5] Dan Matte
		[f5] DCTV
		[f5] Jeff Browning
		[f5] Lori Mac Vittie
		Art Through RSS
		Becky Diaz
		C# FAQ
		Don Box
		Dylan Greene
		Engadget
		Hack-A-Day
		KungfuRobots.com
		MSDN Powertoys
		Nicholas Zurfluh (iRules)
		Robert Scoble
		Rory Blyth
		Scott Hanselman
		Slashdot
		SOAP::Lite
		The Darth Side
		WebServices.org