Learn F5 Technologies, Get Answers & Share Community Solutions Join DevCentral


Questions and Answers

Loading... Loading...

We are currently experiencing an issue in our data center. We have two F5 LTM’s with VIPs for our database cluster and webservers. In the data center, we run HP ProLiant Servers with Fault Tolerant Load Balancing (TLB) NIC teams and two Cisco 3560E switches with Etherchannel Layer 2 trunks. We have two F5 LTMs that are currently in an Active/Standby configuration. On the first LTM, we have interface 1.1 going to switch 1 and interface 1.2 going to switch 2. On the second LTM, we have the same configuration. Please see the simplified topology concerning the connection to the switches.

Over the weekend, we removed foundry switches and replaced those with the Cisco 3560E’s. Since this change over we have had a few issues with no resolution to date. First, when we try to access the VIP for our webservers on the F5 LTM by HTTP/HTTPS it does not resolve. If we try to access the LTMs HTTPS web address, it does not resolve either. However, we can access all servers using their physical address with HTTP/HTTPS. What is really weird is that we can ping the VIP and LTM address. We do not currently have an access-list on any device denying this traffic. Also, when we removed a NIC from the team, we could resolve the VIP and LTM by HTTP/HTTPs. The second issue is that spanning-tree is blocking the redundant interfaces on our second switch. Not sure why this is happening if the LTM is in an Active/Standby state and it must be noted that we are using STP pass through.

Hopefully someone reading this has experienced this before or has an idea/suggestion for a resolution. We have opened a ticket with F5, but no resolution yet. We opened a case with Cisco TAC and they have reviewed the switch configuration and everything looks good.

19 Answer(s):

Can you post your switch config and your bigip_base.conf file? Where do the web servers plug in here?

Not being able to resolve a hostname doesn't really fit into the picture you sent.. Do you have GTM(s) running anywhere? If it truly is a DNS resolution issue, what's your DNS architecture like? I think you might be using the word "resolve", but really meaning you can't bring up the content you're looking for when hitting the VIP address.. Correct?





DNS is working like is should, so no complaints here.  For example, I cannot go to http://172.23.10.10 (VIP) or https://172.23.10.8 (F5 LTM web management).

No GMT running in our network

Ports 1.1 & 1.1 from both F5 LTM connect to SW1
Ports 1.2 & 1.2 from both F5 LTM connect to SW2

See the attached switch configs
SW1-SW2 Etherchannel are ports 49-50 & F5 LTMs connect to ports 37-38 on both switches

I will work on getting you those config files from the F5s.


Issues:

1) When NICs are teamed with TLB, I cannot access the server VIPs or either LTM by HTTP/HTTPS.  (Remove one NIC from the team and it works...also works with NFT) I can ping when teamed though.

2) Spanning-tree has disabled ports 37 & 38 on SW2 for connections from 1.2 on both LTMs (not sure if it's suppose to do this or not when LTMs are in Active/Standby)


Additional Information:

Nothing has changed on the LTMs since the change or prior to.  I have not rebooted either LTM, but have cleared ARP
What type of NIC teaming configuration are your servers using? I believe in the HP teaming setup you can see this by selecting the team in the main screen and clicking "Properties". The two I'm interested in are "Team Type Selection" and "Transmit Load Balancing Method".

I'm not saying this is necessarily incorrect, but it may contribute depending on the setting.
The NIC teams are configured for as follows:



The team type changes to Transmit Load Balancing with Fault Tolerance.

When I change the Load Balancing method to Destination IP, it works.  This is the only method that allows access to any VIP. I just do not want to change items without knowing why and how it effects our network.

Please depict your servers on your drawing, and please post the config of the switches the servers are plugged into...

Thanks
Cisco's example documentation relating to NIC teaming compatibity with Etherchannel actually uses "802.3ad Dynamic with Fault Tolerance" as the Selection and "Destination IP" as the Method:See Here

The document above covers Catalyst switch configuration with HP NIC teaming, but you get the idea... I've not gone through your switch configurations yet, but this is something I've seen in other environments. When I get some extra time I'll scope out those configs.
iRuleYou,

I have attached a simple document showing the server connection to the switches.  We have one NIC attached to SW1 and the second NIC attached to SW2.  For example, the Management Server connects to both switches on Port 12.  What is interesting about the NIC teams is that I can access all resources including HTTP/HTTPs for all servers, but VIP or F5 Web management cannot be accessed.

Joel,

Thanks for the link, but we are not using any type of 802.3ad or Dual Channeling.  We are using a simple one NIC connection to our redundant switches.  I have provided a link to the document about HP NIC Teaming. 

ftp://ftp.compaq.com/pub/products/servers/networking/TeamingWP.pdf



Another interesting item to note was this was all functioning properly prior to the switch change over.  F5 support thinks that rebooting the LTMs will fix the issue, but I am not sure about that.
Just received word from F5 support that HP NIC teams that consist of TLB and SLB are not supported by F5 LTM.  Also, it looks as if the Cisco switches are doing their job by blocking the ports connected to the second switch.  Although it seems a bit weird that this would occur with the LTMs in pass through mode, but I guess the Cisco switch sees it differently.   I will team the NICs using Network Fault Tolerance and this should solve my issue with access to the VIPs.

If anyone has any other suggestions or believes otherwise, please let me know.  Thank you to all that have helped with this issue thus far.
OK...


Firstly... I'm assuming the etherchannel you're referring to in your original post is the one BETWEEN the pair of switches only. Because 3560's don't do VPC's. (You need 6500 VSS or Nexus for that kind of stuff).

Second... Your F5 interfaces are bridged? That's why your redundant interfaces are blocking... Because you have a bridging loop. So the switches HAVE to block one of them (Because you're passing through the STP or PVST packets - As you need to if you're bridging other packets).
Hmm.. Thinking... You're not trying to do a trunk WITHOUT signalling are you? That won't work. It never does. (Well, OK, it CAN but any failures or mis-cabling generally lead to the problems you describe. So I tend to discourage actively any attempts to do so (i.e. won't support it and tell people to fix their end - You can only support stuff you have faith in. Unless you like to make great hourly rates fixing other peoples mistakes).

TLB & SLB... Hmm... I'm not a great fan of HP's NIC teaming... Most of the options don't actually work very well in a multi-switch environment. Switches are simple beasts. They expect IP's to map to consistent MAC's. And they expect MAC's to be on ONE switchport, and not move around. The only really useful NIC teaming mode is 802.3ad (LACP). It works fine, as long as the load-balancing is set correctly. If it's not, then again, you get the problems you're describing.

In summary.
Could be trunking/etherchannel problem without signalling
Or incorrect load balancing modes and MAC addresses moving between switches.
Or something else... The full config and diagram would need to be provided to give much more info.
Edited... re-read the config. What's E01? And why do you have it hardset to 1000FDX? Cisco won't advertise the speed & duplex when you do that, you might find you're getting half-duplex at the far end with this...


H
Posted By jfrizzell on 04/01/2011 03:54 PM
Just received word from F5 support that HP NIC teams that consist of TLB and SLB are not supported by F5 LTM.  Also, it looks as if the Cisco switches are doing their job by blocking the ports connected to the second switch.  Although it seems a bit weird that this would occur with the LTMs in pass through mode, but I guess the Cisco switch sees it differently.   I will team the NICs using Network Fault Tolerance and this should solve my issue with access to the VIPs.

If anyone has any other suggestions or believes otherwise, please let me know.  Thank you to all that have helped with this issue thus far.

Ah... Not sure the F5 will care whether the HP NIC's are doing TLB/SLB or anything else. That's purely between the host and the switches. The F5 just sees an IP address, and (If they're on the same VLAN) a MAC address. Anything else (OK. MTU as well) is irrelevant.

On the port blocking. If you're re-sending traffic down one link fine an other, then the switch is doing it's job BECAUSE you're in pass-though mode. In this mode, the F5 re-sends the BPDU (SPanning Tree) packets back to the switch. So the switch knows that you've looped the traffic back on itself. Therefore it blocks one of the links, because otherwise you get broadcast storms as broadcast packets loop around & around...

I think you just have (Had? If it's now working) a problem with HP NIC teaming (Because it's not really good with multiple switches), and trying to aggregate links without using signalling. 

I'm still a bit suspicious about the early statement that you couldn't resolve addresses with the previous config... But you don't mention where your client and DNS servers were... So that may simply be lack of info.
 
H
 
Hamish,

Thank you for replying to this post.  The only Etherchannel configurations I have are between switches only.  The devices E-01/E-02 are Layer 3 Etherchannels to another network.   In the diagram provided, we have a  Layer 2 trunk between two Catalyst 3560E-48 switches.  We have no 802.3ad configurations between the F5 LMTs or NIC Teams.  It is just a single cable to each switch for redundancy purposes.

I am not a fan of HP Teaming.  I couldn't agree more with you about HP teaming and I have seen a number of issues with those causing loops as you described.  We do not have 802.3ad/LACP configured because we only have one NIC per switch and the 3560Es are not stackable.  This is why we are using one NIC to each switch with TLB.  What I did notice is that when we change the Load Balance Method on the teaming group to Destination IP, it works.  Really not sure why and to make that change on all server scares me without understanding why.  No other Load Balance Method works.  I have changed the 3560E load-balancing methods, but it had no effect and I have left it at src-dst-ip.

The LTMs connect to the switch ports on 37 & 38 for both switches and I have those set for autonegotiation of speed/duplex.

I have two DNS servers and one connects to SW1/SW2.  They are on the same VLAN as all the other servers and F5.  The DNS servers can resolve internal/external names including the VIPs.  It's just with the teamed NICs Team Type Selection of automatic/TLB and Load Balacing Method to Automatic it cannot reach the VIPs/LTM web management.  If I change the Load Balancing Method to Destination IP, I can access the VIPs/LTM web management.

Jeremy

Hamish wrote:
I think you just have (Had? If it's now working) a problem with HP NIC teaming (Because it's not really good with multiple switches), and trying to aggregate links without using signaling.


I agree and think that's what's going on here.. I'm a big fan of using the correct tool for the job.. It looks like you're trying to do more with less.. which most likely isn't your fault. Teaming those NICs properly across switches requires some price Switch hardware.. Or you could buy some old dusty Nortel Boxes and have your hand at some Split Multilink Trunking! oh noooo.. just kidding... they were the first to put it out on the market.. but never really got it working correctly ;)

So this is working with "Destination IP" on your teaming?, it sounds like it's the return traffic through the LTM which is not working from, which makes some sense.. Do you know what method it was using under automatic? that may shed some light..
The default method on the NIC teams is either TCP Connection or Destination MAC.  I say this because neither of these options work, but according to the HP NIC Teaming document it states on page 52 "Although in the current product automatic mode is identical to TCP Connection mode."
we're seeing a similar issue here. it's random and often traffic will flow one way but not the other and some virtual server ip's will be reachable but others will not. On the host the IP will have the correct MAC in the ARP table on the host but traffic will not pass. So far all but one have been corrected by changing the teaming type to NFT from automatic.  I was chalking it up to extensive use of Route Domains on our end but it sounds like a larger issue?
Hi Jfrizzell,
Can you please share the configuration file for your LTM boxes.

Regards,
I would like to report the same issue's and add some of my information to this discussion.

Setup: 2x ltm 1600 in active/standby mode.
Firmware's 9.xx to 11.xx

I can confirm that the symptoms disappear when using a server or workstation with a single NIC or changing the load balancing from automatic to network Fail-over only.

To add. i have this issue with the VIP's but also with the self ip's of the LTM's.
Also the mgmt interface is hooked up to a seperate flat(no routing) L2 network with its own ip range.
Even the management interface's behave in the way's described in the above posts when accessed on a machine with double NIC's in the management network.

On servers with multiple NICS connected to 1 switch with port-aggregation / (lacp trunks) this issue does not occur.
I went trough all the possible arp tables and i cannot find any mismatch in the IP -> arp.

Regards,
Ton
Opend up a support case @ f5.

Official statement:
In short: NO F5 does not support HP nic teaming.

There is no standard that defines how NIC teaming should work, therefor every implementation done by a vendor can be diffrent.
because of this proprietry nature of the HP nic teaming F5 also has no workaround.
Use 802.3ad, lacp. or nothing at all.

In my oppionion this is a poorly documented missing feature in F5 products.
and my guess is lots of customers will run into this issue.
Mmm.. Looking at the whole picture, that's probably fair. Much as I like bashing a vendor for not doing what I want, I'm not sure how F5 would support a feature from another vendor that was subject to arbitrary revision and change and can be implemented in so many different ways and called the same thing.

HP Teaming can encompass

LDAP (802.3ad) which IS supported.
Active/Standby (Which IS supported).

And several other ways. I think there's a couple of arbitrary pseudo load-balancing algorithms in there as well. They'd be the ones I'd steer clear of.

HOWEVER

if you limit your NIC teaming to LACP (802.3ad) I'm fairly confident the answer will come back that it is supported... Because 802.3ad is a STANDARD. Which teaming (Being a feature comprised of both stanards and proprietary configs) is not...

H

Your answer:

You must be logged in to reply. You can login here.