DevCentral Main Feed

To all my SysAdmin friends I say - Run your databases through BIG-IP and sleep better at night

Nojan Moshiri — Fri, 30 Jul 2010 04:36:17 GMT

When I was an “Internet Architect” (lofty title alert!) I used to hear this question fairly often in design meetings, whether to run the database (DB) through the load balancer or not. I would almost always come down on the side of “no there’s no point” because the DBs have their own high availability solutions, they don’t benefit from load balancing and there are usually no multi-master solutions. Also, load balancers are expensive and resources are finite on them.

Over the last few years a number of factors have changed, and today the answer is a solid maybe. There are a lot of compelling features and the crafty engineers that see the light may be able to solve some sticky architectural problems and even sleep better at night.

Change in viewpoint

Enter 2010, things have changed a lot and so has my viewpoint. More often now I’m finding that there are reasonable cases to be made for running the DB through the Application Delivery Controller (ADC). Resources are not as finite anymore, especially on BIG-IPs, and the added benefits include monitoring, flexibility, scaling and control. As an architect I always want more options and as a sydadmin I was stable solutions that let me sleep at night. The ADC has come of age and the benefits outweigh the main negative which is one more potential point of failure for a critical infrastructure component.

The changes that have made me change my mind is first, the resource issue on the ADC. Even from our BIG-IP 1600 series, our so-called “entry-level” point, our 10.2 release allows for passing of 1 Gigabit per second. On the SSL side, we’re talking about 5000 transactions per second of encrypted traffic. Many of the ADCs I’ve used in production spend a large amount of their time mostly idle, just serving front-end traffic and could easily handle the additional load of database connections. I’ve seen these boxes pushed to the limits and it doesn’t worry me nearly as much as it did even five years ago to run database connections through them for fear of overload.

But question still exists why bother?

Once we rule out the “hardware can’t handle it” argument, the second benefit is the ability to monitor the databases, built into our ADC. As Ryan Corder demonstrates in his entry Monitoring Open Source databases with BIG-IP, monitoring Postgres and MySQL is a snap with BIG-IP. This only makes me sleep better at night. I can setup replication to another local instance and create my own high-availability hot/standby cluster without all the overhead of a software clustered solution. Or, I can have the ability to instantly recognize outages and using iRules make intelligent traffic flow changes on the fly, without having to include my monitoring system. We all know how it works today, the monitoring system finds a problem, sends out a page to a system administrator (happy sys-admin day by the way guys and gals!) and meanwhile traffic is down until the problem can be resolved. How about this: the ADC finds the problem beginning with the very first request that has an issue and makes a decision to route traffic around the problem, and the sysadmin doesn’t have to run a fire-drill at that instant.

I’m already a long way towards sold on this now. But finally there’s the idea hinted at above, the flexibility of having the ADC in the way. This is the flexibility of making routing decisions based on layer-7 content. This is the flexibility of putting the databases where you need them and relying on the ADC to optimize TCP, or perhaps even to accelerate connections using BIG-IP WAN optimization. This is the flexibility of opening long-distance VMotion and having your database follow, all made possible by having an ADC in the architecture.

So, should I run my database through BIG-IP?

So, should you? It depends of course, if you’re a mom-and-pop shop with one site and no growth, probably not! But if you’re larger:

* Could you benefit from having more fine-grained control over the uptime and availability of your DB?

* Are you running MySQL or PostgreSQL? If you’re running Oracle, Sybase or MS-SQL, what kind of applications connect to your DBs?

* Is there a better connection manager solution available?

* Would the ADC conflict with your other high availability solution?

* Do you have a fairly complex architecture that could require multiple sites?

* Do you have an architecture that can change rapidly based on business needs?

Hopefully this will be another arrow in the quiver of the lofty Internet Architects ( :-) ) out there enabling them to successfully nail down another great infrastructure design.

Until later, I give all of the System Administrators out there the rest of the day off! May your pager be quiet and your systems remain up!

F5 Friday: Eavesdropping on Availability

Lori MacVittie — Thu, 29 Jul 2010 18:51:24 GMT

Eliminating the overhead associated with active health checks without sacrificing availability

One of the core benefits of cloud computing and application delivery (and primary purposes of load balancing) is availability. In the simplest of terms, achieving availability is accomplished by putting two or more servers (virtual or iron) behind a load balancing device. If one of the servers fails, the Load balancer directs users to the remaining server, ensuring the application being served from that server remains available.

The question then is this: how does the load balancer know when an application is not available? The answer is: health monitoring.

Every load balancer (and clustering solution) can do this at some level. It may be just an ICMP ping or a TCP three-way handshake or determining whether the HTTP and application response received are correct. It may be a combination of a variety of health monitoring options. Regardless of what the health check is doing, it’s getting done and an individual server may be taken out of rotation in the event that its health check response indicates a problem.

Now, interestingly enough there is more than one way to perform a health check. As you might have guessed the first way is to communicate out-of-band with the server and/or application. Every <user configured> time interval, the load balancer performs a check and then acts or doesn’t act upon the response. The advantage of this is that the load balancer can respond very quickly to problems provided the time interval is of sufficiently granular value. The disadvantage of this approach is that it takes up resources on the load balancer, the network, and the server. In a service-provider or cloud computing environment, the resources consumed by out-of-band health checks can be devastating to network performance and may well impact capacity of the server.

What else is there?

INBAND and PASSIVE MONITORING

While inband monitoring is relatively new, passive monitoring was pioneered by F5 many years ago. In fact, leveraging passive monitoring and inband monitoring together provides the means to more quickly address problems as they occur.

Inband monitoring was introduced in BIG-IP v10. Inband monitors can be used with either a Standard or a Performance (Layer 4) type virtual server, and as a bonus can also be used with active monitors. What inband monitoring does is basically eavesdrop on the conversation between a client and the server to determine availability. The monitor, upon an attempt by a client to connect to a pool member, behaves as follows:

If the pool member does not respond to a connection request after a user-specified number of tries within a user-specified time period, the monitor marks the pool member as down.
After the monitor has marked the pool member as down , and after a user-specified amount of time has passed, the monitor tries again to connect to the pool member (if so configured).

What inband monitoring does do – and does well – is eliminate all the extraneous traffic and connections consuming resources on servers and the network typically associated with active monitoring. But what it can’t do at this time is inspect or verify the correctness of the response. It’s operating strictly at the layer 4 (TCP). So if the server|application responds, the inband monitor thinks all is well. But we know that a response from a server does not mean that all is well; the content may not be what we expect. What we want is to mitigate the impact of monitoring on the network and servers but we don’t want to sacrifice application availability. That’s where passive monitoring comes in.

Passive monitoring is actually a technique that leverages network-side scripting (in our case F5 iRules) to inspect the content returned by an application and determine whether it is valid or not. If it is not valid, iRules affords the ability to mark the node down and/or resend the request to another (hopefully correctly working) application instance. Here’s a brief example that can mark the server down after three successive failures, otherwise attempts to “retry” the request:

   1: rule count_server_down {

   2:    when HTTP_REQUEST {

   3:    if { not [info exists orig_request]} {

   4:       set orig_request [HTTP::request]

   5:    }

   6: }

   7: when HTTP_RESPONSE {

   8:    if { [HTTP::status] >= 500 } {

   9:       set failures [session lookup dest_addr [LB::server addr]]

  10:       if { $failures >= 3 } {

  11:          LB::down

  12:       } else {

  13:          session add dest_addr [LB::server addr] [incr failures]

  14:          LB::detach

  15:          HTTP::retry $orig_request

  16:       }

  17:    }

  18: }

19:

Passive monitoring is real-time, it’s looking at real requests to determine actual availability and correctness of response. This is even more useful when you start considering how you might respond. The robust nature of iRules allows you to do some interesting manipulation of content and communication channel, so if you can think it up you can probably get it done with an iRule.

By combining inband with passive monitoring you end up with “inband passive monitoring”. This solution eliminates the overhead of active monitoring by eavesdropping on client-server conversations and ensures application availability by inspecting content.

For a great discussion of inband passive monitoring and a detailed scenario of how it might work in conjunction with a real application, check out Alan Murphy’s post on the subject, “BIG-IP v10: Passive Application Monitoring”.

BIG-IP v10: Passive Application Monitoring
All F5 Friday Entries on DevCentral
Inband monitoring [white paper]
Passive monitoring [white paper]

from tag BIG-IP

from tag monitoring

from tag availability

(more..)

Technorati Tags: MacVittie,F5,F5 Friday,monitoring,health checks,availability,BIG-IP,LTM,load balancing

DevCentral Weekly Podcast 142 – Enterprise Manager And Some Pizza

DevCentral Weekly Podcast — Thu, 29 Jul 2010 07:33:47 GMT

Welcome to the one hundred and fortieth edition of the DevCentral Weekly Roundtable Podcast! A weekly recap of the interesting things that have been going on within the DevCentral community.

Hosts: Colin Walker, George Watkins, Jeff Browning, Joe Pruitt, Scott Koon, and special guests Dan Kim, Bruce Butterfield, and Jim Ruddy from the Enterprise Manager team.

In this weeks podcast we had F5er’s Dan Kim and Bruce Butterfield from the Enterprise Manager team on to talk about their latest release. The covered some cool new features in the product as well as some sneak peeks of what to look out for down the road – you’ll have to listen to find out what they are! We ended things up with some details about our F5 Customer Summit next week in Chicago. Look out Chicago, here we come…

All of the links from this recording can be found with the dcpodcast142 tag at Delicious.com. You can also watch the video recording from our UStream.TV channel: DevCentral Podcast 142.

Get the Flash Player to see this player.

Technorati Tags: F5,DevCentral,Enterprise Manager,Management,Automation,Policy Control,Performance,TCO,Forecasting,Reporting,Dan Kim,Bruce Butterfield,Jim Ruddy,Colin Walker,George Watkins,Jeff Browning,Joe Pruitt,Scott Koon

A Storage (Capacity) Optimization Buying Spree!

Don MacVittie — Thu, 29 Jul 2010 02:54:10 GMT

Remember when Beanie Babies were free in Happy Meals, and tons of people ran out to buy the Happy Meals but only really wanted the Beanie Babies? Yeah, that’s what the storage compression/dedupe market is starting to look like these days. Lots of big names are out snatching up at-rest de-duplication and compression vendors to get the products onto their sales sheets, we’ll have to see if they wanted the real value of such an acquisition – the bright staff that brought these products to fruition – or they’re buying for the product and going to give or throw away the meat of the transaction. Yeah, that sentence is so pun laden that I think I’ll leave it like that. Except there is no actual meat in a Happy Meal, I’m pretty certain of that.

Today IBM announced that it is formally purchasing Storwize, a file compression tool designed to compress data on NAS devices. That leaves few enough players in the storage optimization space, and only one – Permabit – whose name I readily recognize. Since I wrote the blog about Dell picking up Ocarina, and this is happening while that blog is still being read pretty avidly, I figured I’d weigh in on this one also.

Storwize is a pretty smart purchase for IBM on the surface. The products support NAS at the protocol level – they claim “storage agnostic”, but personal experience in the space is that there’s no such thing… CIFs and NFS tend to require tweaks from vendor A to vendor B, meaning that to be “agnostic” you have to “write to the device”. An interesting conundrum. Regardless, they support CIFS and NFS, are stand-alone appliances that the vendors claim are simple to set up and require little or no downtime, and offer straight-up compression. Again, Storewize and IBM are both claiming zero performance impact, I cannot imagine how that is possible in a compression engine, but that’s their claim. The key here is that they work on everyone’s NAS devices. If IBM is smart, the products still will work on everyone’s devices in a year.

BENEFITS FOR YOU.

It would be sweet if you could get a 5x storage space improvement just by putting in one of these boxes, and if they are as easy to manage as claimed, the overhead is minimized while the benefit is maximized. The one concern that these boxes always evoke is recoverability of data. How in the world do you keep uptime if you lose the only box that can access the data on your NAS devices. And that is generally solved by putting in more than one device… Perhaps IBM can have a buy one get one free sale.

Some have wondered (aloud even) how IBM will manage its growing (in both size and complexity) storage offerings. I don’t think that’s a huge problem, to be honest. IBM has huge offerings of everything. From Tivoli and associated offerings to servers, they manage a complex solution set all of the time without problems. Presumably they’ll merge and end of life product lines as it makes sense, which would solve the complexity issue by itself, one would think.

AND THE RACE GOES ON.

Now that Dell, EMC, IBM, and NetApp all have compression/deduplication of one form or another, it remains to be seen if HP and HDS will get itchy fingers. With the rumors of HDS considering a Brocade purchase, that could get interesting quickly. Note that there are also rumors of HDS having something in their roadmap, but HDS is largely an enigma to US writers (I’ve dealt with them as a writer, they are tough to get solid information from), so I wouldn’t give those rumors much credence until you see what they’re doing. They might be OEMing or something, not developing. Though I find that unlikely. It will be interesting to see who, if anyone, will pick up Permabit.

For the record, F5 is a partner with nearly all of these vendors in one form or another, but doesn’t play in this space in any way. Our value-adds are Directory Virtualization and remote storage (So Called Capacity Optimized Transport) through our ARX and WOM products. Apologies for the repetitive disclaimer to regular readers, but that’s a fair statement for the individual that just happened on this blog.

ONLY REAL QUESTION LEFT: WILL YOU JUMP?

So now we’re getting to the fun part of this game. Very soon, disk optimization is going to be available to you if you use a big-name vendor. The interesting part to me is how many of you will use it. My guess? Lots. We’ve been fighting the disk space wars for years, and yesterday’s blog was all about how to survive in the short term. I think that resistance to data that can’t be read by any old machine is going to go away in the face of space shortages. I’ve been wrong before, but it’s a tough call for a CIO to tell the CEO “there’s this technology that could give us X times the space with little outlay and very little work, but we fear it”. Most smart CEOs would respond “the rest of us don’t understand IT, and thus fear it, but we work with you every day”.

So I’ll be seeing you around the water container, your hands trembling as you worry about what to do if that redundant pair of compression/dedupe boxes ever goes down, but you’ll be there, I’m pretty sure. Because contrary to the old maxim, disk ain’t so cheap.

Technorati Tags: Storage Optimization,Capacity Optimization,IBM,Storewize,HP,EMC,Dell,NetApp,HDS,Brocade,F5 Networks,Don MacVittie

WILS: Load Balancing and Ephemeral Port Exhaustion

Lori MacVittie — Wed, 28 Jul 2010 18:58:58 GMT

Understanding the relationship between SNAT and connection limitations in full proxy intermediaries.

If you’ve previously delved into the world of SNAT (which is becoming increasingly important in large-scale implementations, such as those in the service provider world) you remember that SNAT essentially provides an IP address from which a full-proxy intermediary can communicate with server-side resources and maintain control over the return routing path.

There is an interesting relationship between intermediaries that leverage two separate TCP stacks (such as full-proxies) and SNAT in terms of concurrent (open) connections that can be supported by any given “virtual” server (or virtual IP address, as they are often referred to in the industry). The number of ephemeral ports that can be used by any client IP address is 65535. Programmer types will recognize that as a natural limitation imposed by the use of an unsigned short integer (16 bits) in many programming languages.

Now, what that means is that for each SNAT address assigned to a virtual IP address, a theoretical total of 65535 connections can be open at any other single address at any given time. This is because in a full-proxy architecture the intermediary is acting as a client and while servers use well-known ports for communication, clients do not. They use ephemeral (temporary) ports, the value of which is communicated to the server in the source port field in the request. Each additional SNAT address available increases the total number of connections by some portion of that space. As you should never use ephemeral ports in the privileged range (port numbers under 1024 are traditionally reserved for firewall and other sanity checkers - see /etc/services on any Unix box) that number can be as many as 64512 available ports between the SNAT address and any other IP address. For example, if a server pool (virtual or iron) has 24 members and assuming the SNAT address is configured to use ephemeral ports in the range of 1024-65535, then a single SNAT address results in a total of 24 x 48k = 1,152k concurrent connections to the pool. If the SNAT is assigned to a virtual server that is targeting a single address (like another virtual server or another intermediate device) then the total connections is 1 x 48k = 48k connections.

Obviously this has a rather profound impact on scalability and capacity planning. If you only have one SNAT address available and you need the capabilities of a full-proxy (such as payload inspection inbound and out) you can only support a limited number of connections (and by extension, users). Some solutions provide the means by which these limitations can be mitigated, such as the ability to configure a SNAT pool (a set of dedicated IP addresses) from which SNAT addresses can be automatically pulled and used to automatically increase the number of available ephemeral ports.

Running out of ephemeral ports is known as “ephemeral port exhaustion” as you have exhausted the ports available from which a connection to the server resource can be made. In practice the number of ephemeral ports available for any given IP address can be limited by operating system implementations and is always much lower than the 65535 available per IP address. For example, the IANA official suggestion is that ephemeral ports use 49152 through 65535, which means a limitation of 16383 open connections per address. Any full-proxy intermediary that has adopted this suggestion would necessarily require more SNAT addresses to scale an application to more concurrent connections.

One of the advantages of a solution implementing a custom TCP/IP stack, then, is that they can ignore the suggestion on ephemeral port assignment typically imposed at the operating system or underlying software layer and increase the range to the full 65535 if desired. Another major advantage is making aggressive use of TIME-WAIT recycling. Normal TCP stacks hold on to the ephemeral port for seconds to minutes after a connection closes. This leads to odd bursting behavior. With proper use of TCP timestamps you can recycle that ephemeral port almost immediately.

Regardless, it is an important relationship to remember, especially if it appears that the Load balancer (intermediary) is suddenly the bottleneck when demand increases. It may be that you don’t have enough IP addresses and thus ports available to handle the load.

WILS: Write It Like Seth. Seth Godin always gets his point across with brevity and wit. WILS is an ATTEMPT TO BE concise about application delivery TOPICS AND just get straight to the point. NO DILLY DALLYING AROUND.

from tag SNAT

WILS: Client IP or Not Client IP, SNAT is the Question

from tag WILS

from tag capacity

(more..)

Technorati Tags: MacVittie,F5,load balancing,WILS,TCP,scalability,capacity,SNAT,IP addresses,virtualization

Here Comes Payback Time. Prepare for Storage Shortages.

Don MacVittie — Wed, 28 Jul 2010 05:48:28 GMT

The last couple of years have been painful, to say the least. Some call them unprecedented, financially, but I do believe that is pushing the descriptor a bit far, since there have been plenty of instances where business pretty much en-masse questioned the amount that IT returns for their investment and cut budgets, so the feel of this recession is not much different than what we’ve felt before, it’s just by necessity. The funny bit of this is that everyone seems to agree that IT spending still went up in 2009, just by a massively reduced amount. Since the pinch is definitely out there, one can only assume that a 1.6% (or so, depending upon your source) increase in spending was not enough to cover increases in maintenance costs and new purchases.

The impact on IT is pretty straight-forward, at least in my mind. Major IT projects were delayed or canceled based on tough funding decisions, and those projects ran the gamut from development to networking to outsourcing services. Some of these projects were not critical, and some were cut when the business they were going to support was curtailed, but some are “hidden gems” that will in the long run cost the business more than it is saving today.

But belt tightening went on across the entire organization, so IT is left to struggle with its portion of the pie, hoping that the shortfalls (necessary project wise) will be made up in the future. The only bright spot from a budgeting perspective is that new programs and products were cut before IT, since IT is corporate wide and viewed (mostly) strategically.

THE BIGGER THEY ARE, THE HARDER THE CUTS

One area that suffered heavily was storage purchases. Looking at the dollar amount and the growth rate, this is no surprise at all. While the growth rate strongly implies the need for more storage, it also strongly advises finding new ways to deal with storage growth. I’ve written about a few of them in my blog over the last few months, but a thorough examination of the ways available to you today to clean up storage usage is in order.

Don’t believe me? The Recent InformationWeek Analytics Research Report The State of Storage 2010 (shows a hefty 47% of respondents are struggling with the amount of storage demanded just by their mission critical applications. That’s a big number. This percentage topped out 2010 concerns for storage strategists, taking the number one spot away from 2009’s “Data Loss/Data Security” category (which came in at 44% this year).

While that doesn’t exactly leave a bleak picture for the near future – the implication being that 53% of us aren’t struggling as badly with storage resources – it is certainly not rosy.

IDEAS FOR SAVING ON STORAGE

There are a variety of tools out there to help you get through the crunch, and the technology and process list is growing. Some of them we’ve discussed before, some we haven’t, but let’s make a list. If I miss anything, feel free to add on in the comments. And I’ve linked all but a couple that are all over the news.

Tiered Storage. If you’re not there yet, go there. The cost of lower end storage is less even if you buy it from the same big name, and it holds less access-speed sensitive data just fine.
Compression. Consider it. Even though your data must be read by the compression engine, the amount of space you can recoup is huge. Not for everyone by a long shot though. There’s a short introductory piece here about it.
At-rest Deduplication. I list it separately because it’s the popular twin in the compression/dedupe category right now. Again, requires the dedupe engine rehydrate your data on read, but worth considering for the massive space savings.
Virtualization. This is one of the areas F5 plays in with ARX. If you abstract your disk, you can increase utilization on the back-end without disrupting users. Increased utilization means less outlay for new disk. IMO, NAS virtualization is much more reasonable/viable than SAN, simply because you can get your data back relatively painlessly if the virtualization engine fails. Coworker Renny Shen wrote a good overview of virtualization and storage this month that’s worth reviewing for background.
Thin Provisioning. I’ve warned about the potential for inadvertent abuse that thin provisioning harbors, but that doesn’t make it a useless technology, just one to use judiciously.
Remote Data Centers. If you have storage in many data centers and the storage is not being roughly equally utilized, time to start looking at ways to equalize data distribution. We play in this space also with our WOM product, since it speeds and secures replication and other data transfers across the WAN to a remote datacenter.
Cloud Storage. It is worth the time to investigate cloud storage, as I’ve chattered about frequently of late. Particularly if that storage is fronted by a cloud gateway.
Archival Storage. Remember the poor, lost, lonely archive? The place where outdated or underutilized data was supposed to go to die? Well it’s time to bring it back. Want to reduce storage costs? Reduce storage needs. I discussed one way to deal with this possibility a few weeks ago, but there are others. The point being that we need to start throwing stuff away. Since we can’t chuck it into the circular file, data that is unlikely to be needed again soon will have to go onto long-term storage and stay there forever.

IT’S A DUOCITY!

Much like Dilbert’s Top Secret Management tips, many of these ideas work best when used in tandem. Unlike Dilbert, none of these ideas is likely to get you fired. Put archival storage with Cloud storage and you get a place to stash data limited by only your operations budget. Put an ARX or other Directory Virtualization product in front of that cloud storage gateway and you have disk that is indistinguishable to the end user from local NAS devices, but is actually shoving stuff off of your expensive disk and into the cloud. Add tiered storage (automatically provided by ARX, maybe by other vendors), and your data is moving to the right place at the right time, but users don’t have to care. Utilize WOM to open WAN tunnels to a remote datacenter and optimize file transfers between the two and you’ve got resource leveling… It all adds up.

Hopefully you will soon be flush with new disk, but none of the above ideas would be precluded by adding new disk (though the extended portion of file information discussed in my Meyers’-Briggs blog might be too resource intensive to pursue if disk were plentiful). They are all additive, meaning new disk would simply last longer, which is certainly worthy of consideration. They vary in costs, some requiring little investment given your existing infrastructure, some requiring an investment that I would argue is minimal in comparison to buying new disk every year.

HOPE FOR THE BEST, PLAN FOR THE WORST

And should those rumbling about a “double-dip” recession be correct, it is certainly a good idea to have these tools in your stable before you are once again trying to do more with less. Here’s hoping they’re not right, but contingency planning dictates that, at least to some extent, we should act like they’re visionaries, just in case.

WHAT THE HECK ARE YOU STORING?

Fun random fact from that InformationWeek report? 58% of you are managing more than one terabyte of data and less than 100 terabytes. It boggles the mind. We are not accessing 100 terabytes of data in a year, so a ton of that is likely wasted space. Add in the 17% managing 100-500 Terabytes, and 75% of you have a whole lot of data. Ever wonder what it is? Ever ponder that there is stuff on your network no one has seen in a year or more? Creepy fun.

Technorati Tags: Virtualization,Storage Virtualization,IT Management,IT Budget,WAN Optimization,Cloud Storage Gateways,Cloud Storage,F5 Networks,Don MacVittie

CloudFucius Investigates: The Comeback Cloud

Pete Silva — Wed, 28 Jul 2010 03:06:58 GMT

Not that it was ever down and out, but the whole cloud computing concept has gotten off to a grand but challenging start. It was all the rage when first ‘conceived’ as the new way of hosting applications with the promise of cost savings, automation, flexible/dynamic architectures, fast and repeatable deployment and a pay-as-you-go model. The Coin Operated Cloud but with very little understanding of all the buttons, functions and risks. As IT started to comprehend the nuances of the cloud, then some very serious questions regarding it’s ability to protect and secure information came to light. This arrived at a time when the economic downturn could have vaulted the shared, less expensive infrastructure offerings to the top of any IT list since budgets were tight and resources scarce. It was the perfect storm of tight budgets and cost effective computing. However, the risk of data loss kept folks questioning the viability of putting sensitive data in the cloud. Plus, no consent on standards kept the mix of cloud offerings all over the place while limiting the customer’s ability to mix and match.

As cloud offerings matured and *some* security concerns addressed along with IT having a better grasp on risks and mitigation, the cloud is gradually becoming more attractive to enterprises. Like anything else that suddenly bursts on to the scene, the hype outweighs the reality and it takes time to fully understand and realize the benefits.

It seems that almost weekly a shiny new cloud computing survey is released and this week, Vanson Bourne (commissioned by Savvis) released a report that ties cloud computing to the economic recovery. Specifically, ‘68 percent of respondents said cloud computing will help their businesses recover from the recession.’ The survey found that 54% of respondents said cost cutting and more flexible IT provisioning is the biggest issue they face and they are optimistic with the prospect that cloud computing can cut costs. A significant number of IT decision makers, 96%, said they are as confident or more confident than they were last year that cloud computing is enterprise ready. Yankee Group also released a report that says cloud computing is on the cusp of broad enterprise adoption. They also indicate that the concept of cloud computing as a business enabler has jumped from 37% to 60% of respondents in just a year.

But can cloud computing save the economy?

That’s a stretch, according to David Linthicum in this article. The logic goes, if companies can save money with cloud computing and are more optimistic about the prospect of cloud computing, then they will start spending IT dollars for cloud deployments. This will, in turn, boost the cloud economy since providers will have to prepare and hire for the influx of customers. Since accelerated growth would occur, that would attract public and private investments in cloud computing technologies. The ‘catalyst,’ as Linthicum notes, is to get more investment dollars back into the technology industry, which should fuel the tech economy at least. Betting that cloud computing will turn the entire economy is a huge long-shot and unrealistic. The survey also revealed that even with all that confidence, security remains as the key adoption barrier. 52% of those who do not use cloud computing said ‘security of sensitive data’ as a top concern.

In a slightly related story, LimeWire Planning A Serious, Cloud-Based Comeback.

And one from Confucius: And remember, no matter where you go, there you are.

ps

The CloudFucius Series: Intro, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14

Resources:

Cloud Computing Will Fuel Economic Recovery: Survey

Will cloud computing save the economy?

Who's actually spending on cloud computing?

Yankee Group: Sky clears for cloud computing

Cloud Computing's Effect on the Hosting Industry

LimeWire Planning A Serious, Cloud-Based Comeback

Technorati Tags: F5, infrastructure 2.0, integration, cloud connect, Pete Silva, security, business, education, technology, application delivery, cloud, context-aware, web, internet, openstack

twitter: @psilvas

Extreme Automation. Dynamic Control with or without the Cloud.

Don MacVittie — Tue, 27 Jul 2010 03:45:54 GMT

In the data center of the future, you are going to need to be able to bring up new instances of an application, have them fully functional without any user intervention, and when they’re no longer needed they should clean up after themselves and quietly go away. Five years ago this was fantasy talk, two years ago it was coming to the fore, and today we can see clearly that such adaptable infrastructure is going to be mandatory for any installation/application that has a highly variable rate of throughput.

The drivers for this need for adaptability are varied, but the core ones that we have all heard an earful of are cloud – where you are charged for your usage, and leaving apps running when not needed is tantamount to throwing money away – and a highly virtualized environment where there are a lot of virtuals running per physical server and keeping an instance running that you do not need is tantamount to throwing CPU cycles away. Okay, the alliteration was good, but it really IS throwing CPU cycles away, nothing tantamount about it.

The problem is/has been that there are a lot of complex pieces to put together when making such a system work. It is difficult to say “spin up an instance, give it access to all the resources it needs, and make it play well with the other instances of the same application while directing traffic to it.”

IF YOU THINK ABOUT IT, THE JOKE IS ON US.

Yet all of this stuff exists – you provision servers and virtual servers and integrate them into your environment today – it just wasn’t designed with automation in mind. Which is kind of amusing if you think for a second about what computers are, at their heart. How we missed an opportunity to design for automation is a fun point to ponder.

Over the last couple of years, things have come a long way. VMWare has thin provisioning for disk, developers have gotten smarter about developing applications that automatically play nice with multiple instances of the same app running, and load balancers have APIs to help with automation. F5’s API is pretty thorough and allows you to do just about any automation task – indeed, I myself helped develop a complete automation system in Java with iControl that ran from Linux shell apps. You can check out some similar functionality in my iC2I series of Tech Tips from 2008. Here’s the one that shows how to create a Virtual IP from code with a minimum of information available iC2I: Creating a Virtual Simplified. The process isn’t simple because the architecture and assumptions – like the existence of a pool of servers to service the Virtual IP, monitors to check health, etc. – aren’t simple, but since you only create this code once, and this is the basis of a Java implementation to manage the Virtual IP creation process, it’s not too overly burdensome either. The flip side of this one (and I believe another installment in the iC2I series) is removing a node from a pool – deprovisioning from the Virtual IP perspective. I haven’t looked into the VCenter functionality to stop an instance, but likely it is there also.

When I wrote the iC2I series, TMSH did not yet exist. Now that it does, much of the functionality that was done in that series with Java could be done with command line scripts, though automation and coordination is admittedly easier to do from a Java application that can be run anywhere, it is another option to make the datacenter more dynamic.

This is the part where I normally say “and you can do this stuff with other vendors too…” You’ll note that this time I’m not. You can. It’s not so easy.

IS THIS NOT CLOUDISH?

At this point in this blog, many of you will question how close we are to cloud computing without a ton of excess overhead. Lori and I were discussing that very topic and came to the conclusion that while this is data center automation, it is a long way from cloud computing. Suffice it to say that you can manage to bring things up and down, but the Business Process Automation portion – self-service provisioning, checking the health of the overall Virtual IP before harvesting a little-used instance, projecting needed usage into the future (to keep from harvesting a virtual machine that is likely to be needed in five minutes even if it is idle now) is all missing.

That doesn’t stop this functionality from being extremely useful in the data center of today or tomorrow though. The ability to spin up new instances and work them into the overall architecture at a whim (remember that by being behind a BIG-IP under a Virtual IP, you don’t have to worry about firewall or other infrastructure issues, and through VMWare VCenter you can set up disk and other per-instance tasks) is huge and has implications for your ability to sleep peacefully at night.

Using iSessions you can even distribute load to a completely separate data center (or cloud provider) and make the instances appear to the BIG-IP (and all connections running through it) to be coming from the current location. That allows you to automate not just the data center, but also shift load to a backup location that might normally only come online during downtime. Multiple data centers, one connection point, automation… Now we’re getting somewhere, no?

The picture at the right could be any BIG-IP arrangement (or most load balancers for that matter), except that “pool” could be in this data center, in the cloud, in a datacenter on the other side of the globe… It really doesn’t matter with iSessions, as long as you can build a reliable Internet connection to the location and there is a BIG-IP there also.

THE FUTURE, IT IS SO BRIGHT.

The point is that we’ve arrived at true data center automation and I have seen it in action. It takes a time investment to get things going at first, but once they’re up, you can even hand out scripts (or point at a web interface) for IT application owners that allow them to add virtual machines on-demand by running a command line or clicking on a web page.

Once again, we see the power of the future coming to fruition, and the future appears to be a very cool place. Moving forward, perhaps the other pieces of cloud will fall as nicely into place, but this level of dynamism allows you to offer up the apps your business customers want, when they want them, where they want them, for as long as they want them… Without the overhead of spinning up machines yourself, or leaving dead instances running on your servers. And that doesn’t even touch on VMotion…

Technorati Tags: Automation,iC2I,Virtualization,VMWare,iSessions,iControl,F5 Networks,Don MacVittie



Related Articles and Blogs

iC2I: Introduction

iC2I: Dumping a Node

iC2I: Dumping iRules

iC2I: Creating a Virtual Server

iC2I: Creating Virtuals Simplified

iC2I: What Virtuals Rely on This iRule?

iC2I: What Pools Rely on This Server?

iC2I: What Virtuals Rely on This Server?

Introducing Long Distance VMotion With VMWare

The World Doesn’t Care About APIs

Lori MacVittie — Mon, 26 Jul 2010 19:15:24 GMT

Bottles, birds, and packets: how the message is exchanged is less important than what the message is as long as it gets there.

I heard it said the other day, regarding the OpenStack announcement, that “the world does not care about APIs.”

Unpossible! How could the world not care about APIs? After all, it is APIs that make the Web (2.0) go around. It is APIs that drive the automation of infrastructure from static toward dynamic. It is APIs that drive self-service and thin-provisioning of compute and storage in the cloud. It is APIs that make cross-environment integration of SaaS possible. In general, without APIs we’d be very unconnected, un-integrated, un-collaborative, and in many cases, uninformed.

Now, it could be said that the world doesn’t care about APIs until they’re highly adopted, but unlike the chicken and the egg question (which may very well have been answered, in case you weren’t paying attention), it is still questionable whether the success of sites like Facebook and Twitter and the continued growth of SaaS darlings like Salesforce.com are dependent upon exactly that: their API.

The API is the new CLI in the network. The API is the web’s version of EAI (Enterprise Application Integration), without which we wouldn’t have interesting interactions between our favorite sites and applications. The API is the cloud’s version of the ATM (Automated Teller Machine) through which services are provisioned with just a few keystrokes and a valid credit card. The API is the means by which interoperability of cloud computing will be enabled because to do otherwise is to create the mother-of-all hub-and-spoke integration points. As Jen Harvey ( co-founder of Voxilate, developer of voice-related mobile applications) pointed out, an API makes it possible to develop a user-interface that essentially obscures the underlying implementation. It’s the user-interface the users care about, and if you don’t have to change it as your application takes advantage of different clouds or services or technologies, you ensure that productivity and user-adoption – two frequently cited negative impacts of changes to applications in any organization – are not impacted at all. It’s a game-changer, to be sure.

So the API is the world in technology today, how could we not care about it?

WE DO. WE JUST CARE MORE ABOUT the MODEL

Maybe the point is that we shouldn’t because the API today is just a URI and URIs are nearly interchangeable.

Face it, if interoperability between anything were simply about the API then we’d have already solved this puppy and put it to sleep. Permanently. But it’s not about the API per se, it is, as William Vambenepe is wont to say, “it’s the model, stupid” [Edited per William's comment below, 7-27-2010]

Unfortunately, when most people stand up and cheer “the API” what they’re really cheering is “the model.” They aren’t making the distinction between the interface and the data (or meta-data) exchanged. It’s what's inside the message that enables interoperability because it is through the message, the model, that we are able to exchange the information necessary to do whatever it is the API call is supposed to do. Without a meaningful, shared model the API is really not all that important.

The API is how, not what, and unfortunately even if everyone agreed on how, we’d still have to worry about what and that, as anyone who has every worked with EAI systems can tell you, is the really, really, super hard part of integration. And it is integration that we’re really looking for when we talk about cloud and interoperability or portability or mobility, because what we want is to be able to share data (configuration, architectures, virtual machines, hypervisors, applications) across multiple programmatic systems in a meaningful way.

COMMODITIZATION REALLY MEANS NORMALIZATION

Here’s the rub: having an API is important, but the actual API itself is not nearly as important as what it’s used to exchange.

APIs, at least those on the web and taking advantage of HTTP, are little more than URIs. Doesn’t matter if it’s REST or SOAP, the end-point is still just a URI. The URI is often somewhat self-descriptive and in the case of true REST (which doesn’t really exist) it would be nearly completely self-documenting but it’s still just a URI. That means it is nearly a trivial exercise to map “/start/myresource” to “/myresource/start”. But when the data, the model, is expressed as the payload of that API call, then things get … ugly. Is one using JSON? Or is it XML? Is that XML OVF? Schemaless? Bob’s Homegrown Format? Does it use common descriptors? Is a load balancer in cloud a described as an application delivery controller in cloud b? Is the description of a filter required in cloud a using iptables semantics or some obscure format the developer made up on the fly because it made sense to her?

Mapping the data, the model, isn’t a trivial exercise. In fact without a common semantic model it would require not the traditional one-chicken sacrifice but probably a whole flock in order to get it working, and you’d essentially be locked-in for all the same reasons you end up locked-in today: it costs too much and takes too much effort to change. When pundits and experts talk about commoditization of cloud computing, and they do often, it’s not an attempt to minimize the importance of the model, of the infrastructure, but rather it’s a necessary step toward providing services in a consistent manner across implementations; across clouds. By defining core cloud computing services in a consistent manner and describing them in similar terms, advanced services can then be added atop those services in a like manner without impacting negatively the ability to migrate between implementations. If the underlying model is consistent, commoditized if you will, this process becomes much easier for everyone involved.

Consider HTTP headers. There is a common set, a standard set of headers used to describe core functions and capabilities. They have a common model and use consistent semantics through which the name-value pairs are described. Then there’s custom headers; headers that follow the same model but which are peculiar to the service being invoked. In a cloud model these are the differentiated value-added cloud services (VACS) Randy Bias mentions in his post regarding the announcement of OpenStack and the ensuing cries of “it will be the standard! it will save the world!”. The most important aspect of custom HTTP headers that we must keep in any cloud API or stack is that if they aren’t supported, they do not negatively impact the ability to invoke the service. They are ignored by applications which do not support them. Only through commoditization and a common model can this come to fruition.

Having an API is important. It’s what makes integration of applications, infrastructure, and ultimately clouds possible. But it isn’t the definition of that interface across disparate implementations of similar technology that will make or break intercloud. What will make or break intercloud is the definition of a consistent semantic model for core services and components that can be used to describe the technologies and policies and meta-data necessary to enable interoperability.

Related Posts

Cloud, Standards, and Pants

Infrastructure 2.0: Squishy Name for a Squishy Concept

Despite Good Intentions PaaS Interoperability Still Only Skin Deep

from tag model

F5 and the Cloud

Cloud Computing's Other Achilles' Heel: Software Licensing

from tag API

Hindsight is Always Twenty-Twenty

Are You Scrubbing the Twitter Stream on Your Web Site?

Infrastructure Integration: Metadata versus API

Are admins developers too?

REST API Developers Between a Rock and a Hard Place

Cloud interoperability must dig deeper than the virtualization layer

The API Is the New CLI

WILS: Automation versus Orchestration

Magic Virtualization-Fairy Dust and the New Network

Next-Generation Management of Data Centers Should be Modeled on Social Networking

(more..)


Technorati Tags: MacVittie,F5,infrastructure 2.0,cloud computing,API,model,OpenStack,Randy Bias,William Vambenepe,Jen Harvey,Web 2.0,integration,standards,intercloud

Dell Buys Ocarina Networks. Dedupe For All?

Don MacVittie — Mon, 26 Jul 2010 04:28:23 GMT

Storage at rest de-duplication has been a growing point of interest for most IT staffs over the last year or so, just because de-duplication allows you to purchase less hardware over time, and if that hardware is a big old storage array sucking a ton of power and costing a not-insignificant amount to install and maintain, well, it’s appealing.

Most of the recent buzz has been about primary storage de-duplication, but that is merely a case of where the market is. Backup de-duplication has existed for a good long while, and secondary storage de-duplication is not new. Only recently have people decided that at-rest de-dupe was stable enough to give it a go on their primary storage – where all the most important and/or active information is kept. I don’t think I’d call it a “movement” yet, but it does seem that the market’s resistance to anything that obfuscates data storage is eroding at a rapid rate due to the cost of the hardware (and attendant maintenance) to keep up with storage growth.

WHO AND WHAT?

Ocarina Networks is one of a handful of vendors that was dealing with de-duplication abstracted from the tier. They didn’t care what tier you were de-duplicating, only that they were very good at it. They’ve been working hard to get your attention, and failing that to get the attention of resellers/OEMs. This move shows that they managed to receive that attention. And a good thing too, since this market is going to be over-run by big names providing de-duplication directly. Even though I think that de-duplication falls into the category of technologies you want outside of your storage arrays so that you can apply it across vendors and tiers, we’ve all seen that similar functionality, if offered by the big names, ceases to exist as a stand-alone market, just through market attrition.

Dell is a provider of a variety of storage systems, and while you don’t think of them when you think HP/EMC/NetApp/IBM/HDS (when did The Big Three become The Big Five?), they certainly move a lot of disk. From entry level NAS devices to high-end iSCSI, they’re out there with their own products, and reselling the likes of EMC for things they don’t directly offer of their own (or OEM, but these days even IBM OEMs some storage boxes). At a minimum, Dell is a force in the storage market, I think of them more as “the next generation” because they’re poised to take advantage of the move to IP storage, while not invested in FC in any great depth.

IS THE WHOLE GREATER THAN THE SUM OF THE PARTS?

The merger of these two companies makes for some interesting possibilities. Much like when Dell bought EqualLogic, it doesn’t seem like a perfect fit, but the potential for a merged future that betters Dell’s place in the business market exists. It is no longer just massive companies with millions of servers that are suffering from storage expense overload, lots of smaller organizations are feeling the pinch of a never-ending stream of unstructured data. From the perspective that most of this unstructured data is driving the business, that’s good. From the perspective of the costs to store this data, that’s bad. Dell should find good resonance with its customer base, and hopefully enough of a market to make this purchase worthwhile (for the record, I have no idea if EqualLogic has worked out for Dell as a valuable purchase, but I’ve been a fan of EqualLogic since the early days, and presume that they kept the momentum going under the aegis of Dell).

ALWAYS WITH THE QUESTIONS.

The question I – and presumably many others – have is “will Dell keep this open to all vendors?” They are in a unique position to do so, they resell other vendors’ gear, they sell their own, their focus is not on totally dominating the storage market with the newest, snazziest, shiniest boxes, but rather on giving their customer base an array of options that suit their needs within the price bounds of the market segment they’re selling to (I know that you can buy storage from them for $3000, and that some storage from them is more than $50,000 – and that’s all aimed at the SMB market). They have the relationships to keep Ocarina free of the fetters of single-vendorism, and Ocarina has contracts that would make it one of the top choices for de-dupe. The only question is what Dell’s management was thinking when they signed the contract. It would be a valid road for them to pursue to attempt to fold Ocarina into their existing offerings and focus it primarily or even solely on Dell storage products… If they wanted to be a storage vendor. My gut says probably not, and I hope that this time my gut is right. They have a lot more business than just storage, so another valid path for them to take is to offer the Ocarina product for any certified storage. IMO that would be the best option, they can then sell it to their existing and future customers without constraint.

Dave Raffo over at SearchStorage has a pretty good listing of the relationships this purchase impacts and what the relationships entail. It will be interesting to see how Dell handles each of them.

And in that second scenario, we’d all win. If you don’t need de-duplication, you can buy from whomever, if you do need de-duplication, you can buy Ocarina devices from Dell, get their world-class support, and get your storage… From whomever. In my opinion that would be a win for everyone.

THE FUN LITTLE DISCLAIMER PARTS

Note: Dell is a business partner of F5 (as are all of the other vendors mentioned). While I have not talked to anyone at Dell or Ocarina leading up to to writing this blog, I thought it only fair to mention our relationship.

And F5 currently has no horse in this race, we don’t make storage, our de-dupe (via WOM and EDGE Gateway) is on-the-fly and complimentary to that done by Ocarina, and we don’t sell servers. This blog simply reflects my interest in the changing storage market space and not any kind of official F5 position on the topic. If we even have one.

Technorati Tags: Storage Deduplication,Ocarina Networks,Dell,Dell EqualLogic,HP,EMC,NetApp,IBM,HDS,F5 Networks,F5 WOM,F5 EDGE Gateway,Don MacVittie



Related Articles and Blogs

Dell-Ocarina deal will alter landscape of primary storage deduplication

Data dedupe technology helps curb virtual server sprawl

Expanding Role of Data Deduplication

The Reality of Primary Storage Deduplication

The Battle of Economy of Scale versus Control and Flexibility

Lori MacVittie — Sun, 25 Jul 2010 20:53:00 GMT

When strategies are formed it quickly becomes obvious that cloud computing is more about balance than anything else.

At a time when you’d think cloud computing would be the primary “go to” strategy for managing scale and rapid growth multiple well-known and demanding organizations are building their own data centers instead.

With all the hype around cloud being faster, cheaper, and more efficient these folks must be crazy, right?

Not at all. In fact, these moves illustrate the growing friction between the economy of scale offered by cloud computing and the control and flexibility that is part and parcel of owning one’s own data center.

In April Twitter announced plans to build a data center of its own. On Wednesday it provided additional details on the Twitter Engineering blog.

“Later this year, Twitter is moving our technical operations infrastructure into a new, custom-built data center in the Salt Lake City area,” wrote Twitter’s Jean-Paul Cozzatti, who said having dedicated data centers will provide more capacity to accommodate growth of 300,000 new users per day. “Keeping pace with these users and their Twitter activity presents some unique and complex engineering challenges. Importantly, having our own data center will give us the flexibility to more quickly make adjustments as our infrastructure needs change.”

-- Data Center Knowledge, “Twitter Picks Utah for New Data Center”

Twitter isn’t the only Web 2.0 savvy organization moving to their own data center. Facebook earlier this year announced it, too, was also investing in building out its own data center.

BUT CLOUD AUTO-SCALES and STUFF!

It’s not all about scalability. I know that sounds nearly heretical, but it’s not. And it’s not a new mantra, either. Scalability is certainly a factor in why one would choose cloud computing over a localized deployment, but also important are control and flexibility.

Another consideration is the ability to customize your data center infrastructure to provide more granular control of operations. “That control gives us a ton of flexibility, and we can build new things without having to wait for our partner,” said Heiliger [Jonathan Heiliger, Facebook’s VP of Technical Operations]

-- Data Center Knowledge, “Data Centers: For When The Cloud is Not Enough”

If I’ve said it once I’ve said it a thousand times: control is a huge factor in the decision making process and something that isn’t effectively offered by today’s public cloud computing offerings. Remember the Information week analytics Cloud computing survey in 2009?

Even though security remains concern number one, control and configurability are on the top of the list, as well. The issue of control has almost always gone hand in hand with cloud adoption inhibitors, but it always takes a back seat to the more glamorous and scary “security” issue. These are not minor stumbling blocks in many cases, and the inability to rapidly adapt an infrastructure to meet growth and scale and make architectural changes, if necessary, are paramount to success. If cloud computing cannot provide the agility necessary to meet these challenges then it is logical to assume that organizations will either (a) stay in the local data center or (b) move to a local data center from the cloud when it becomes obvious the environment is inhibiting forward momentum.

Current adoption patterns indicate that this is not an anomaly, but will instead likely become the norm for organizations. Applications that are initially deployed “in the cloud” will, upon becoming a critical business application or growing beyond the meager means of control and flexibility offered by the cloud, will migrate to the data center, where control and agility are provided by the simple fact that the organization can change at will any piece of the infrastructure – from its physical implementation to its logical organization – at will. This is evident in the percentage of organizations using cloud for “dev and test” but not for production. Clearly the economy of scale and rapidity of deployment makes the cloud a perfect environment for development and testing but not necessarily production.

ECONOMY of SCALE MAY be INHIBITING SCALE

The irony is that the economy of scale offered by cloud may well be biting cloud in the proverbial derrière as it becomes the inhibitor to effective scale by limiting or making extremely difficult the architectural changes necessary for an application to scale in a cloud environment.

At some point scalability can become not about the application but about its infrastructure and the way in which that infrastructure interacts. It can become about the network and its components and how applications end up interacting with and through that infrastructure. In a cloud computing environment it is rarely the case that a customer can impact that infrastructure and, when it can, it is then limited by other factors such as underlying virtualization technology and the physical server infrastructure on which the application is ultimately deployed. If the answer to a scalability obstacle is more bandwidth and higher throughput, you can’t really add another NIC to a server in the cloud. That’s not your call. But it is if you’re in the data center, and it is virtualization – not cloud - that ultimately provides the agility to make such a change and rapidly propagate that change across the application deployment.

It isn’t always about costs. Well, okay, it is about cost but in IT it’s about cost as it relates to performance, or flexibility, or other operational functionality required to successfully meet data center and business goals. When spending less on infrastructure results in higher operational costs, the organization really hasn’t saved money at all. Savvy CIO and CTOs understand that it’s not a battle, but a balancing act. It’s not about achieving the highest economy of scale, but the best economy of scale given the specific operational and business needs.

Related Posts

Fear and Loathing of Cloud Computing

from tag control

What is a Strategic Point of Control Anyway?

I Have a Dream

The Three Reasons Hybrid Clouds Will Dominate

The Infrastructure 2.0 Trifecta

Control, choice, and cost: The Conflict in the Cloud

Do you control your application network stack? You should.

The Wisdom of Clouds: In Cloud Computing, a Good Network Gives You Control...

from tag agility

F5 and the Cloud

Infrastructure 2.0: Aligning the network with the business (and the rest of IT)

Cloud Computing: It's the destination, not the journey that is important

State Street Puts Agility in The Fast Lane - Virtualization

Knowing is Half the Battle

(more..)



Technorati Tags: MacVittie,F5,cloud computing,virtualization,scalability,control,architecture,agility,infrastructure

IT and Data: If Not Me, Then Who? If Not Now, Then When?

Don MacVittie — Fri, 23 Jul 2010 00:59:54 GMT

YES, IT IS ABOUT THE BUSINESS…

Over the years data archiving and backup technology have waxed and waned, growing closer and further away from each other as the needs of the enterprise, new technologies, and the external economic and regulatory environment have changed. There have been clear indications that the need for reliable backup and archiving of data was a growing regulatory requirement, yet it did nothing for the business, and that meant it was a hard-to-sell expense.

BUT LET’S NOT GENERALIZE TOO MUCH.

Out in the TwitterSphere there are some thinkers starting to expound that we need more business analysts and less techs, and even a few saying that we should stop being technologists all together in IT. One has even suggested that we stop calling it IT and call it something more aligned with business. That way lies danger for all things infrastructure or IT focused. Back in the late 90s I was at a place where the Network Engineers had to sell an upgrade to a network segment. One that should have been upgraded long before their proposal. As an Enterprise Architect, I was involved in looking over what they did, reviewing the presentation, and helping them out. They had to fight tooth and nail to upgrade a segment that was years out of date, showing projections of usage and predicting a segment meltdown. Mind you that network segments, backups, lots of that stuff is not going anywhere… It is all part of the overhead that is IT, whether your IT is in the cloud or in your data center on physical servers and everywhere in between. It is also all technical. The usage on a network segment, which backup/replication/archive system to use, these things are decision points that require technical knowledge of the systems involved.

REPLICATION AND ARCHIVE – COOPETITION?

But I digress. Back to the topic at hand, which is your data. When courts started slapping enterprises with penalties for “lost” email archives, and requirements for data retention in some industries started to go up, many started looking at long term archival again, but replication was just heating up, and we’re multi-tasking, but not that multi-tasking, these things were similar and happening at the same time. Replication could be sold to the business as disaster recovery, archival storage is risk management. Replication won. But with replication came a slew of “for long term storage” caveats – getting the most out of technology for the business, even when the business is resistant. Or perhaps to overcome the business’ resistance.

Where does that leave us? Well, for many people, and against the advice of many others, replication is a key to both our short and long term data retention strategies… We need that replication to a remote site to go through so that we have short term DR capability, and so that we have long term archival capability. Sometimes the archiving is done off the replica, sometimes it is the replica, perhaps with some snapshotting. I won’t tell you this is just a bad idea, you know it, and if you’re using a replica as your archive, we’ll just assume it is due to the business environment you are operating in.

IT’S GONNA GET WORSE.

We in IT are facing even more hurdles to infrastructure upgrades, and while in the nirvana of the future, your infrastructure needs may go down, the reality of today is that they’re going up. Those replicas, backups, and archives are going to be more important as the regulatory situation changes yet again in the coming year, and the chances of you implementing something with no add to the business value of IT go down. One of the things that has been increasing over time, and cloud/SaaS increase is the burden on your Internet connection(s). The amount of data sent off site is going up, and these days the vast majority of us use our Internet connections to send nearly all data off-site. Trucks and tapes aren’t more secure, they’re not more reliable, and they’re certainly not faster, so Internet bits and bytes won. That means you need to get the most out of that pipe though, and for most organizations, the most isn’t at all accurate in today’s usage scenario.

WAN OPTIMIZATION HELPS

That’s why the resurgence of WAN Optimization. We need to get the data to its destination as quickly as possible, as reliably as possible, and as inexpensively as possible. All while increasing the amount of data being transferred. With high performance WAN Optimization you can put off making a never-ending investment in more bandwidth. Which adds up quickly if you have to pay by the month for increased bandwidth, considering that T3 to OC3 prices are a huge multiple – around 5x the monthly fees – anything that helps you defer the need to perform an upgrade that will cost you thousands of dollars – or more – per month is worth checking out. And that’s without the equipment and service costs.

And with WAN Optimization, you’re benefitting all applications that run through it, not just making replication more reliable and secure, which makes it easier to sell to the business. And there’s always that chance that by installing WAN Optimization you can reduce your remote datacenter bandwidth needs, which could result in an immediate cost savings. Not that I expect this to be typical, but the bandwidth reduction WAN Optimization offers makes it possible, particularly if you’re barely into the next size of pipe.

Buying more bandwidth is always an option, but like so many things in IT, it is a high-cost, every month expense, so exploring options that can optimize your connection and reduce or defer upgrades – yes, like our WOM product, but I’m talking more generally – should be on your checklist before you sign that contract. After all, if not you, then who? If not now, then when?

INTERNET RESEARCH LESSONS

Funny bit about this title, I’ve known of this phrase since I was a kid, and researching it now, it seems to have no actual progenitor. Many believe it is a modification of the famous quote by Hillel the Elder, but it gets a lot of attribution, including a rapper and Gorbachev – both of which came to prominence after I learned of the phrase. So you can claim it if you want!

Technorati Tags: Data Replication,WAN Optimization,Bandwidth,Backup,Data Archiving,F5 WOM,F5 Networks,Don MacVittie



Related Articles and Blogs

WAN Optimization: Getting the Most Bang For the Buck

How Much is a T1 Line? (dated info on t1 and larger, well documented)

Zombies! This Years’ Vampires Are Zombies!

F5 Friday: Beyond the VPN to VAN

Lori MacVittie — Thu, 22 Jul 2010 19:23:16 GMT

Web 2.0 and cloud computing have naturally pushed all things toward application-centric views, why not the VPN?

When SSL VPNs were first introduced they were a welcome alternative to the traditional IPSEC VPN because they reduced the complexity involved with providing robust, secure remote access to corporate resources for externally located employees.

Early on SSL VPNs were fairly simple – allowing access to just about everything on the corporate network to authenticated users. It soon became apparent this was not acceptable for several reasons, most prominently standing out the risk of infection by remote employees who might have been using personal technology to work from home. While most organizations have no issue with any employee working a few extra hours at home, those few extra hours of productivity can be easily offset by the need to clean up after a virus or bot entering the corporate network from an unsecured, non-validated remote source. This was especially true as one of the selling points for SSL VPN was (and still is) that it could be used from any endpoint. The “clientless” nature of SSL VPN made it possible to use a public kiosk to log-in to corporate resources via an SSL VPN without fear that the ability to do so would be “left behind.” I’m not really all that sure this option was ever widely used, but it was an option.

Then SSL VPNs got more intelligent. They were able to provide endpoint security and policies such that an “endpoint”, whether employee or corporate owned, had to meet certain criteria – including being “clean” – before it was allowed access to any corporate resource. This went hand in hand with the implementation of graded authentication, which determined access rights and authorization levels based on context: location, device, method of access, etc… That’s where we sat for a number of years. There were updates and upgrades and additions to functionality but nothing major about the solution changed.

Until recently. See, the advent of cloud computing and the increasing number of folks who would like to “work from home” if not as a matter of course then as a benefit occasionally has been driving all manner of solutions toward a more application-centric approach and a more normalized view of access to those applications. As more and more applications have become “webified” it’s made less sense over time to focus on securing remote access to the corporate network and more sense to focus on access to corporate applications – wherever they might be deployed.

THE NEXT GENERATION of ACCESS CONTROL

That change in focus has led to what should be the next step in the evolution of remote access – from SSL VPN to secure access management, to managing application access by policy across all users regardless of where they might be located.

Similarly, it shouldn’t matter whether corporate applications are “in the cloud” or “in the data center”. A consistent method of managing access to applications across all deployment locations and all users reduces the complexity inherent in managing both sides of the equation.

We might even call this a Virtual Application Network (VAN) instead of a Virtual Private Network (VPN) because what I’m suggesting is that we create a “network” of applications that is secured by a combination of transport layer security (SSL) and controlled by context-based access management at the application layer. Whether a user is on the corporate LAN or dialed-in from some remote location that has yet to see deployment of broadband access shouldn’t matter. The pre-access validation that the accessing system is “clean” is just as important today when the system is local as if it were remote; viruses and bots and malware don’t make the distinction between them, why should you?

By centralizing application access across users and locations, such secure access methodologies can be used to extend control over applications that may be deployed in a cloud computing environment as well. Part of F5’s position on cloud computing is that many of the solutions that will be required to make cloud-deployed applications viable is that the control that exists today over locally deployed applications must be extended somehow to those remote applications as a means to normalize management and security as well as controlling the costs of leveraging what is supposed to be a reduced cost environment.

That’s part of the promise of F5’s BIG-IP Access Policy Manager (APM). It’s the next step in secure remote access that combines years of SSL VPN (FirePass) experience with our inherent application-aware delivery infrastructure. It provides the means by which access to corporate applications can be normalized across users and application environments without compromising on security and control. And it’s context-aware because it’s integrated into F5’s core enabling technology platform, TMOS, upon which almost all other application delivery functionality is based and deployed.

I highly encourage a quick read of George Watkin’s latest blog on the topic, Securing the Corporate Intranet with Access Policy Manager, in which he details the solution and some good reasons behind why you’d want to do such a thing (in case I’m not convincing enough for you). You may also enjoy a dive into a solution presented in a previous F5 Friday, “F5 Friday: Never Outsource Control”, that describes an architectural approach to extending normalized control of application access to the cloud.

Related Posts

All F5 Friday Entries on DevCentral

The Other Hybrid Cloud Architecture

The Three Reasons Hybrid Clouds Will Dominate

from tag VPN

The Virtual Public-Private Cloud Connection

from tag APM

I Find Your Lack of Win Disturbing

The days of IP-based management are numbered

from tag SSL

Turning the Pushdo Bot Into the Push-oh-no-you-don’t Bot

Virtual Network Infrastructure: Virtually Good Enough?

WILS: SSL TPS versus HTTP TPS over SSL

F5 Friday: Secure, Scalable and Fast VMware View Deployment

Hardware Acceleration Critical Component for Cost-Conscious Data Centers

WILS: Why Does Load Balancing Improve Application Performance?

SANS Top 25 Epic Fail: CWE-319

Google SPDY Protocol Would Require Mass Change in Infrastructure

Network Security Does Not Imply Application or Database Security

(more..)


Technorati Tags: MacVittie,F5,F5 Friday,APM,SSL,VPN,SSL VPN,remote access,cloud computing,Firepass,application access

DevCentral Weekly Podcast 141 - #CameraFail

DevCentral Weekly Podcast — Thu, 22 Jul 2010 06:17:29 GMT

   Welcome to the one hundred and fortieth edition of the DevCentral Weekly Roundtable Podcast! A weekly recap of the interesting things that have been going on within the DevCentral community.

Hosts: Colin Walker, George Watkins, Jason Rahm, Jeff Browning, Joe Pruitt, and Scott Koon.
In this weeks podcast we discussed Jeff and Colin’s trip to Japan where they threw down some #camerafail, the Gestalt IT Tech Field Day at F5 last week (where we recorded sessions on F5, VMWare Orchestration, API’s in the Woods, Security, Storage Optimization, and the Delegate Roundtable), iRules breaking the world, HTTP Version in redirects, Coloring some Bits, Securing the Corporate Intranet with APM, and OpenSearch on DevCentral.

All of the links from this recording can be found with the dcpodcast141 tag at Delicious.com. You can also watch the video recording from our UStream.TV channel: DevCentral Podcast 141.

Get the Flash Player to see this player.

Technorati Tags: F5,DevCentral,Japan,VMWare,Orchestration,APIs,Security,Storage Optimization,iRules,HTTP,Intranet,APM,OpenSearch,Colin Walker,George Watkins,Jason Rahm,Jeff Browning,Joe Pruitt,Scott Koon

Colin And Jeff Discuss Their Trip To Japan

DevCentral TV — Thu, 22 Jul 2010 03:58:59 GMT

Joe sat down with Jeff and Colin and took a few minutes to discuss their recent trip to visit our team in Tokyo, Japan.

Listen in and hear about a recap of the meetings they were involved with – including an awesome event with the DevCentral User Group.

You will also get a taste of what they dined on while they were there.

Technorati Tags: F5,DevCentral,Tokyo,Japan,User Group,Jeff Browning,Colin Walker,Joe Pruitt

F5 Application Designer: Health Based Load Balancing

Julian Balog — Thu, 22 Jul 2010 03:22:52 GMT

With the release of the F5 Management Pack v2.1.2 (available for download here) the F5 Application Designer extensibility framework has been updated to support various load-balancing scenarios, based on health and performance monitoring conditions of the applications monitored to System Center Operations Manager (SCOM).

1. The F5 Application Designer Extensions: a Quick Overview

The F5 Application Designer Extensions are application specific management packs built on top of the core F5 Management Pack. They provide holistic monitoring and auto-configuration support for various server platforms and application environments fronted by F5 devices (such as IIS 7, SharePoint 2007/2010, VMM Server 2008/R2). The traditional way of monitoring all these applications in System Center Operations Manager usually would be mostly scoped to the individual platform itself rather than a unified health monitoring view of the whole application environment.

Here’s where the F5 Application Designers come into play and wire the applications together into a unified health monitoring diagram, as “Application Instances”, enabling a more comprehensive view of the applications’ health in System Center Operations Manager.

With the F5 Application Designer extensions, the F5 LTM Pool Members are paired with their related application components (such as Windows servers, Web servers, Virtual servers, SharePoint servers, etc) and the resulting object hierarchies are wired together as “Application Instances” grouped within an “Application Instance Group”, which ultimately is paired with the related F5 LTM Virtual Server, making up the “Overall Application” monitoring object. Such an application can be for example the entire deployment of an e-commerce web-farm, monitored within System Center Operations Manager.

The health of the App Instance by default represents the worst health of any of it's member components. In turn, the overall Application's health depends on at least one of the App Instances being healthy as well as the health of the underlying LTM Virtual Server.

More information on F5 Application Designer Extensions related topics can be found here:

Application Designer Library: http://devcentral.f5.com/wiki/default.aspx/MgmtPack/ApplicationDesigner.html

Holistic Monitoring and Auto Configuration using the F5 Management Pack and IIS 7: http://devcentral.f5.com/weblogs/jhendrickson/archive/2010/04/23/holistic-monitoring-and-auto-configuration-using-the-f5-management-pack.aspx

2. Health Based Load Balancing - Overview

In this article (which is part of a series on similar topics), I'll talk about the F5 Application Designer Extensions and their use in providing intelligent monitoring decisions within the System Center Operations Manager, in various data-center-specific scenarios. In particular: the health-based load balancing, where F5 LTM Pool Members get enabled/disabled depending on the health of various applications monitored.

Description: Enable / Disable LTM Pool Member based on application specific health-monitor state change

Use Case:

Trigger an Error health-state for an IIS 7 specific monitor (IIS 7 Application Pool) by stopping the Default Application Pool on the IIS Server hosted on LTM Pool Member 1

See the related (hosting) App Instance health state turn red / Error

See the related LTM Pool Member (1) health state turn red / Error

Load balancing follows LTM Pool Member 2

Resume the Success health-state for the IIS 7 monitor (IIS 7 Application Pool) by starting the Default Application Pool on the IIS Server hosted on LTM Pool Member 1

See the related (hosting) App Instance health state turn green / Success

See the related LTM Pool Member (1) health state turn green / Success

Load balancing follows LTM Pool Members 1 and 2.

3. Prerequisites

Install the F5 Management Pack (http://devcentral.f5.com/mpack)

Windows Server 2008 Internet Information Services 7 management pack (MS System Center catalog)

System Center Virtual Machine Manager 2008 R2 management pack (MS System Center catalog)

Windows Server 2008 Operating System (Discovery) management pack (MS System Center catalog)

F5 Application Designer Library extension packs (http://devcentral.f5.com/mpack):

Core App Designer: Common Library

IIS App Designer: Microsoft Internet Information Server 2008

4. Preparation

Discover F5 device (with at least 1 LTM Virtual Server and 2 LTM Pool Members)

Discover IIS 7 web servers (expected to run on the LTM Pool Members discovered above)

Discover overall application deployment (F5 + IIS 7) (see details here):

Look for the topic on “discovering an existing application” (here)

5. Scenario

Trigger an Error health-state for an IIS 7 specific monitor (IIS 7 Application Pool) by stopping the Default Application Pool on the IIS Server hosted on LTM Pool Member 1

See the related (hosting) App Instance health state turn red / Error

See the related LTM Pool Member (1) health state turn red / Error

Load balancing follows LTM Pool Member 2

Resume the Success health-state for the IIS 7 monitor (IIS 7 Application Pool) by starting the Default Application Pool on the IIS Server hosted on LTM Pool Member 1

See the related (hosting) App Instance health state turn green / Success

See the related LTM Pool Member (1) health state turn green / Success

Load balancing follows LTM Pool Members 1 and 2.

6. Resources

Video tutorials: http://www.vimeo.com/13548421 (Health Based Load Balancing Scenario - Overview)
http://www.vimeo.com/13523437 (Health Based Load Balancing Scenario)

PowerPoint presentation:

health-based-load-balancing-scenario.pptx

F5 Application Designer: Load Balancing Scenarios

Julian Balog — Thu, 22 Jul 2010 01:55:38 GMT

With the release of the F5 Management Pack v2.1.2 (available for download here) the F5 Application Designer extensibility framework has been updated to support various load-balancing scenarios, based on health and performance monitoring conditions of the applications monitored to System Center Operations Manager (SCOM).

1. The F5 Application Designer Extensions: a Quick Overview

The F5 Application Designer Extensions are application specific management packs built on top of the core F5 Management Pack. They provide holistic monitoring and auto-configuration support for various server platforms and application environments fronted by F5 devices (such as IIS 7, SharePoint 2007/2010, VMM Server 2008/R2). The traditional way of monitoring all these applications in System Center Operations Manager usually would be mostly scoped to the individual platform itself rather than a unified health monitoring view of the whole application environment.

Here’s where the F5 Application Designers come into play and wire the applications together into a unified health monitoring diagram, as “Application Instances”, enabling a more comprehensive view of the applications’ health in System Center Operations Manager.

With the F5 Application Designer extensions, the F5 LTM Pool Members are paired with their related application components (such as Windows servers, Web servers, Virtual servers, SharePoint servers, etc) and the resulting object hierarchies are wired together as “Application Instances” grouped within an “Application Instance Group”, which ultimately is paired with the related F5 LTM Virtual Server, making up the “Overall Application” monitoring object. Such an application can be for example the entire deployment of an e-commerce web-farm, monitored within System Center Operations Manager.

The health of the App Instance by default represents the worst health of any of it's member components. In turn, the overall Application's health depends on at least one of the App Instances being healthy as well as the health of the underlying LTM Virtual Server.

More information on F5 Application Designer Extensions related topics can be found here:

Application Designer Library http://devcentral.f5.com/wiki/default.aspx/MgmtPack/ApplicationDesigner.html

Holistic Monitoring and Auto Configuration using the F5 Management Pack and IIS 7 http://devcentral.f5.com/weblogs/jhendrickson/archive/2010/04/23/holistic-monitoring-and-auto-configuration-using-the-f5-management-pack.aspx

2. Load Balancing Scenarios

With this article we initiate a series of presentations centered around various Data Center Orchestration scenarios, where the F5 App Designer extensions not only help with the overall health monitoring of a complex application environment, but also provide automatic intelligent decisions based on certain health and performance monitor triggers.

Here's a brief overview on some of these scenarios (use cases).

2.1. Health Based Load Balancing (Scenario #1)

Description: Enable / Disable LTM Pool Member based on application specific health-monitor state change

Use Case:

Trigger an Error health-state for an IIS 7 specific monitor (IIS 7 Application Pool) by stopping the Default Application Pool on the IIS Server hosted on LTM Pool Member 1

See the related (hosting) App Instance health state turn red / Error

See the related LTM Pool Member (1) health state turn red / Error

Load balancing follows LTM Pool Member 2

Resume the Success health-state for the IIS 7 monitor (IIS 7 Application Pool) by starting the Default Application Pool on the IIS Server hosted on LTM Pool Member 1

See the related (hosting) App Instance health state turn green / Success

See the related LTM Pool Member (1) health state turn green / Success

Load balancing follows LTM Pool Members 1 and 2.

2.2. Performance Threshold Rule Based Load Balancing (Scenario #2)

Description: Enable / Disable LTM Pool Member based on application specific threshold rules

Use Case:

Create a unit monitor based on an IIS 7 specific rule (ASP.NET\Worker Process Restarts Performance Rule), targeting the ASP.NET Application Pool object

Create the ASP.NET worker process recycle condition based on the number of requests made

The unit monitor created above will mark the health of the ASP.NET Application Pool object's state as red / Error

See the related (hosting) App Instance health state turn red / Error

See the related LTM Pool Member (1) health state turn red / Error

Load balancing follows LTM Pool Member 2

Resume the Success health-state for the unit monitor by lowering the number of requests and having the ASP.NET worker process recycle no more

See the related (hosting) App Instance health state turn green / Success

See the related LTM Pool Member (1) health state turn green / Success

Load balancing follows LTM Pool Members 1 and 2.

2.3. Performance / System Metrics (Provisioning) Based Load Balancing (Scenario #3)

Description: Update LTM Pool Member load-balancing ratios based on system-performance / provisioning

Use Case:

Set the load-balancing method for the targeted LTM Pool to “Ratio (member)”

Start by setting the Ratio on both LTM Pool Members to 1

Provision LTM Pool Member 1 (virtual machine) with 2GB of RAM

Provision LTM Pool Member 2 (virtual machine) with 4GB of RAM

See how the load balancing ratios on LTM Pool Member 1 and LTM Pool Member 2 get updated (automatically) to 33 and 67 respectively, following the capacity (total memory) of the LTM Pool Members

Details on configuring and running these use cases will follow in subsequent articles, so stay tuned.

Get Out Your Crayons. We Need to Color Us Some Bits

Lori MacVittie — Wed, 21 Jul 2010 19:10:56 GMT

Those eight bits in the IP header aren’t doing much of anything these days, perhaps it’s time to put them to work

Back in the early days of bandwidth management, when quality of service and prioritization of traffic were on everyone’s minds because we were stuck with low throughput connectivity, there was a brief discussion about the use of IP’s TOS (Type of Service) bits as a means to meet specific application performance needs. I say brief because, well, it never really got anywhere. See, even though the creators of the IP specification had looked into the future and provided a technical solution to prioritization of traffic they couldn’t have looked into the future and seen the organizational roadblocks to leveraging such a simple but effective method of managing traffic.

The biggest problem was that you could ensure that TOS bits were honored within the organization, but once those packets passed through the organization’s boundary of control, i.e. onto the Internet, there was no guarantee or requirement that any other organization honor those bits. And because packets flow through many, many different routers and switches along its very long yet ironically very short travels from datacenter to client, if just one fails to honor the bits then the packets go gray and prioritization is lost.

Hence it was that bandwidth management moved up the stack, with queuing and rate shaping at the transport and application layers of the stack becoming the norm and the “coloring” of TOS bits fell into disuse, like childhood crayons set aside in favor of cool gel pens and clicky-mechanical pencils.

THE NEED STILL EXISTS

Interestingly enough, the need for prioritization and bandwidth management still exists and, in many cases, it could become of paramount importance to the successful implementation of a mature cloud.

It is easy to forget that when you look under the covers of a cloud computing environment that there still exist physical network connections that comprise that environment. Despite the magnitude of virtualization in use at all layers that abstracts the entire infrastructure from its physical implementations, that network is still there. It’s massive, it’s huge, and it’s a spider’s web of connectivity.

I am not the only one thinking about this, I’m fairly certain. An InformationWeek Analytics survey last year included a concern rarely seen in cloud computing surveys: “speed to activate new services/expand capacity.” And folks are apparently somewhat concerned about this. Less so than other “problem” areas of cloud but enough concerned that it made the chart.

The problem is that because everyone is still actually sharing a physical network connection all that traffic gets mixed up on the wire. At the physical layer, every packet is the same regardless of what payload it’s carrying. But the reality is that some packets are more “important” in the sense that some will be extremely time sensitive. A request to provision a service that’s under increasing load is more important than many application requests but both look the same to the routers and switches that get those packets from point A to point B within a cloud computing environment.

As cloud computing providers and enterprises get better at automating the provisioning and elastic scalability processes, the packets that make up requests for such operational tasks will become increasingly time sensitive and important to ensuring operational efficiency and success.

The problem, as once reared its ugly head before, is that all that traffic – customer and operational – is running over HTTP. REST, SOAP, whatever. The APIs that make it possible to provide “compute as a service” are almost unilaterally implemented using a combination of HTTP and REST or SOAP-based architectural principles. And so are the applications running in the cloud. Everything is running over HTTP and thus, as with bandwidth management challenges in the past, it becomes increasingly difficult to distinguish traffic without inspecting its payload.

And payload inspection always adds latency. It may be only microseconds but it’s still latency. And if every router has to do it, well, microseconds eventually add up to seconds. And when we’re talking about ensuring the order of operations in a provisioning process, those seconds can make a big difference.

BACK to BEGINNING

Sometimes the solution really is to go back to the beginning, to our roots, to the network.

It could be that the solution lies in out-of-band management networks. If not physically at least logically separated network channels that are by default prioritized and therefore will never get stuck in a traffic jam trying to get to location “C” because the application at location “A” is physically on the same network and heavily oversubscribed at the moment an important management request is trying to get to “C”. This would, however, add another layer of complexity to the management of not just the physical but the logical network. Complexity that is almost always translated into higher costs, which of course gets passed on to the customer.

It could be that DCSP (DiffServ or Differentiated Services Code Point (DSCP)) – what eventually became of TOS because it was basically unused - could become the solution precisely because a single provider “owns” the entire network. Because a provider has control over all the components comprising the underlying network infrastructure it could enforce the honoring of DSCP bits across its network and thus prioritize traffic at the IP layer.

DiffServ is concerned with classifying packets as they enter the local network. This classification then applies to Flow of traffic where a Flow is defined by 5 elements; Source IP address, Destination IP, Source port, Destination port and the transport protocol. A flow that has been classified or marked can then be acted upon by other QoS mechanisms. Multiple flows can therefore be dealt with in a multitude of ways depending on the requirements of each flow. Packets are first Classified according to their current DSCP. Then they are separated into queues where one queue may be routed via a marking mechanism and another queue may be examined more closely.

-- Quality of Service overview

If a provider segments by port “cloud management” traffic from “normal” traffic, this solution would be fairly easy to implement. If not, well, then we’re going to need something else. It’s the concept behind DiffServ and TOS that’s important to leverage in such a solution – the recognition that some traffic must be prioritized to ensure delivery in a timely fashion.

EIGHT BITS

What seems clear, to me at least, is that eventually we’re going to run into scenarios in which we need something akin to Operations Performance Management (OPM) to ensure that management and control messages are delivered in a timely fashion.

This is especially true as cloud computing matures and we start to see the dynamism inherent in infrastructure 2.0 components put into broader use. Real-time enforcement of security and delivery policies must be real-time. They can’t be near-time, they must happen now. When we start relying on existing open standards like HTTP and messaging hubs and event-driven networking architectures we have to remember we get the good and the bad from those existing standards and implementations. We get the ease of use and integration, the flexibility, and the ubiquity of support, but we also get the problems that have plagued quality of service implementations forever: when everything is delivered via HTTP then everything looks like HTTP. Differentiation is nice, but we need to have a way to do that that doesn’t impede performance in a cloud computing environment.

In this case, it seems wise to look down the stack and return to a perhaps less sophisticated but absolutely more elegant and simple means of distinguishing not only traffic but precedence and terms of service. DSCP or TOS or whatever we might decide to slap into those 8 bits in the IP header (perhaps there’s room for a new specification and use of those bits?) would be infinitely more scalable and easily supported in a cloud computing environment for distinguishing between the small subset of internal “traffic types” that need to be managed.

Related Posts

The Rise of the Out-of-Band Management Network

from tag ethernet

That’s Awesome! And By Awesome I Mean Stupid

from tag management

Why IT Needs to Take Control of Public Cloud Computing

When (Micro)Seconds Matter

Beware the Availability Rat Hole in the Cloud

Virtual Server Sprawl: FUD or FACT?

Infrastructure 2.0: The Diseconomy of Scale Virus

Next-Generation Management of Data Centers Should be Modeled on Social Networking

How to instrument your Java EE applications for a virtualized environment

Managing Virtual Infrastructure Requires an Application Centric Approach

And the Killer App for Private Cloud Computing Is...

‘Drowsy’ Networking

As Deep as a Puddle

(more..)


Technorati Tags: MacVittie,F5,performance,management,TOS,ethernet,network,operations performance management,OPM,DSCP,DiffServ,cloud computing

Zombies! This Year’s Vampires Are Zombies!

Don MacVittie — Wed, 21 Jul 2010 16:50:17 GMT

A couple of weeks ago, one of the people I follow on our home Twitter account (focused on role-playing and miniature wargames) tweeted “Zombies! This Year’s Vampires Are Zombies! Oh Please Let This Year’s Vampires Be Zombies!!” as a joke aimed at those in the entertainment industry that were riding the fad wave of Vampire films/books/games. Seriously, the show that did the penitent Vampire in the 80s was good, everything portraying a “good-guy” Vampire after that was lame follow-on.

That’s how I feel about high-tech sometimes. “Cloud is This Year’s SOA! Oh Please Let Cloud Be This Year’s SOA! All we have to do is change our marketing material!” seems to be the refrain. The benefits to you the user are often glossed over with the demand that you do something now before it’s too late (eg: Before the Zombies get you!). Security and storage are in the throes of this phenomenon too… I’ll just say “primary storage deduplication” and leave it at that.

And then someone with their head on straight reminds me that we’re still fighting the fires of 2000, let alone solving the problems of tomorrow… In this case it was Howard Marks’ short-but-sweet blog about data loss.

In our home network, Lori and I use copies and RAID to keep our backups, there is not a ton of timestamping and versioning except – as is the case in Howard’s blogs – the external facing stuff. I know many (too many) small to medium businesses and even a few departments in larger organizations that use the same type of methodology. Replication will help in the event of a hardware failure, but can cause you heartburn if the database is systematically destroyed through queries. While Howard is a bit sharp on snapshots, they can be a lifesaver, though they’re not a panacea either.

And the statistics on the quality of backups and the number of people who even bother to validate them are horrendous. Why bother if you’re not checking the quality?

Look at the old problems, the ones we’ve been fighting for years, and ask yourself a couple of questions…

Have things changed technologically that make a different way better?

Is the problem we were working on/ignoring/pondering four or ten years ago resolved?

Regardless of the future of the infrastructure in your IT department, IT is the guardian and steward of both data and systems. Don’t lose sight of that in the rush to deploy shiny new solutions. Of course every vendor believes they have solutions to help you do things better - have I mentioned WOM or ARX lately? But the place to start is an evaluation of where you are, what infrastructure/systems/needs you are weak in, and only then go looking for a solution to them. Using WOM to back up to a remote datacenter is a good solution – if you need it – as is using ARX with a Cloud Storage Gateway – again, only if you need it. But what you do need is assurances that no matter what the nightmare unleashed upon your IT department – be it trojan, file integrity issues, or selective DB deletes, that you have a route back to working systems with consistent data. If you can’t guarantee that today, I’d start figuring out how to do so… Quickly.

Technorati Tags: replication,RAID,data loss,F5 WOM,F5 ARX,F5 Networks,Don MacVittie



Related Articles and Blogs

D’oh, I should have Made a Backup

Data Loss Statistics

Avoiding an IT Crisis

Searching DevCentral Just Got Easier

Joe Pruitt — Wed, 21 Jul 2010 05:21:19 GMT

I recently received an internal iRule email and one of our folks created a search provider for FireFox to search DevCentral. Lori quickly responded and asked if we could get this posted to DevCentral. Why not if it will help the community so I took a look. Then it occurred to me that a while back I created a search provider definition based on the OpenSearch specification. For some reason, on our last site refresh, the links in our website were removed so the browser didn’t natively pick them up.   I fixed that so now you can add DevCentral as a native search target in your browser of choice.

Here’s a little background on OpenSearch, how I implemented it on DevCentral, and how to set it up in your browser.

OpenSearch

OpenSearch is a format that can be used to describe a search engine so that it can be accessed and used by search client applications such as web browsers. It’s basically just an XML file that you put on your webserver and by adding a hidden tag in your application pages, a browser is able to automatically access the search pages on your site.

Creating the Search Engine

The first step is to create the search engine definition file. In this case, I called that file OpenSearch.xml. The format for that file is defined here. For DevCentral, you can view the definition directly at OpenSearch.xml.

1: <OpenSearchDescription>

2: <ShortName>DevCentral Search</ShortName>

3: <Description>Search F5's DevCentral Developer Community</Description>

4: <Url type="text/html" method="get"

5: template="http://devcentral.f5.com/Default.aspx?..q={searchTerms}"/>

6: <Image width="16" height="16">http://devcentral.f5.com/favicon.ico</Image>

7: <InputEncoding>UTF-8</InputEncoding>

8: <SearchForm>http://devcentral.f5.com/</SearchForm>

9: </OpenSearchDescription>

It’s pretty self explanatory. The only thing you need to make sure you include in is the {searchTerm} template value into your search url.

Exposing it to Browsers

Once you have the definition setup, you need to broadcast it to the browsers accessing your site. This can be done with a hidden link tag in the header of your HTML pages. By adding a “link” element of type “application/opensearchdescription+xml”, the browser will look at the “href” attribute for the location of the search provider.

1: <head>

2: ..

3: <link rel="search" type="application/opensearchdescription+xml"

4: title="DevCentral Search" href="http://devcentral.f5.com/opensearch.xml" />

5: </head>

That’s it. Now site search is ready to be configured in the browsers. For each of the following steps, click on the DevCentral homepage before starting.

Installing in FireFox

Click on the drop down on the left side of the search control in the browsers address bar. You will see a Add “DevCentral Search” menu item.

Select this item and you will now have “DevCentral Search” as your default search provider. This can be changed back by selecting another drop down from the menu.

Now you should see the F5 logo in your search control and by typing a search term in it, it will take you to the DevCentral search results for that query.

At this point the search provider is available to you on any website you visit, not just when you are visiting DevCentral.

Installing in Internet Explorer

Installation for Internet Explorer is almost as easy as FireFox. The search control dropdown on the top right of the browser will have a “Add Search Providers” menu item.

Select that and you will then see “DevCentral Search” as an option. By selecting that menu item, you will be presented with a “Add Search Provider” dialog box. Click the “Add” button.

At this point, you now have Internet Explorer configured to browse DevCentral anytime you please. If you didn’t make it your default search provider in the Add dialog box, you’ll have to select it from the menu.

Now by typing a search term, you will be taking to the DevCentral search just like you were with the previous Firefox installation.

Installing in Google Chrome

Google Chrome behaves a bit different as it doesn’t have a dedicated search control, but relies on the address box to do both navigation and search. The search provider settings can be found by right clicking in the address control and you will find an “Edit search engines…” menu item.

By selecting that menu item, you will be presented with a dialog box containing the currently configured default engines, and others that Google thinks you might like.

Look for “DevCentral Search” at the bottom of the list. Highlight it and click the “Make Default” button on the right side of the dialog box.

Now, just enter your search term in the address control and Chrome will take you to the DevCentral search results page.

Enjoy!

-Joe

Technorati Tags: F5,DevCentral,OpenSearch,Internet Explorer,Firefox,Chrome,Search,Joe Pruitt

Let’s Face It: PaaS is Just SOA for Platforms Without the Baggage

Lori MacVittie — Tue, 20 Jul 2010 19:36:27 GMT

At some point in the past few years SOA apparently became a four-letter word (as opposed to just a TLA that leaves a bad taste in your mouth) or folks are simply unwilling – or unable – to recognize the parallels between SOA and cloud computing .

This is mildly amusing given the heavy emphasis of services in all things now under the “cloud computing” moniker. Simeon Simeonov was compelled to pen an article for GigaOM on the evolution/migration of cloud computing toward PaaS after an experience playing around with some data from CrunchBase. He came to the conclusion that if only there were REST-based web services (note the use of the term “web services” here for later in the discussion) for both MongoDB and CrunchBase his life would have been a whole lot easier.

For an application developer, as opposed to an infrastructure developer, all these vestiges of decades-old operating system architecture add little value. In fact, they cause deployment and operational headaches—lots of them. If I had taken almost any other approach to the problem using the tools I’m familiar with I would have performed HTTP operations against the REST-based web services interface for CrunchBase and then used HTTP to send the data to MongoDB. My code would have never operated against a file or any other OS-level construct directly.

[…]

Most assume that server virtualization as we know it today is a fundamental enabler of the cloud, but it is only a crutch we need until cloud-based application platforms mature to the point where applications are built and deployed without any reference to current notions of servers and operating systems.

-- Simeon Simeonov “The next reincarnation of cloud computing”

Now I’m certainly not going to disagree with Simeon on his point that REST-based web services for data sources would make life a whole lot easier for a whole lot of people. I’m not even going to disagree with his assertion that PaaS is where cloud is headed. What needs to be pointed out is what he (and a lot of other people) are describing is essentially SOA minus the standards baggage. You’ve got the notion of abstraction in the maturation of platforms removing the need for developers to reference servers or operating systems (and thus files). You’ve got ubiquity in a standards-based transport protocol (HTTP) through which such services are consumed. You’ve got everything except the standards baggage. You know them, the real four-letter words of SOA: SOAP, WSDL, WSIL and, of course, the stars of the “we hate SOA show”, WS-everything.

But the underlying principles that were the foundation and the vision of SOA – abstraction of interface from implementation, standards-based communication channels, discrete chunks of reusable logic – are all present in Simeon’s description. If they are not spelled out they are certainly implied by his frustration with a required interaction with file system constructs, desiring instead some higher level abstracted interface through which the underlying implementation is obscured from view.

CLOUDS AREN’T CALLED “as a SERVICE” for NOTHING

Whether we’re talking about compute, storage, platform, or infrastructure as a service the operative word is service.

It’s a services-based model, a service-oriented model. It’s a service-oriented architecture that’s merely moved down the stack a bit, into the underlying and foundational technologies upon which applications are built. Instead of building business services we’re talking about building developer services – messaging services, data services, provisioning services. Services, services, and more services.

Move down the stack again and when we talk about devops and automation or cloud and orchestration we’re talking about leveraging services – whether RESTful or SOAPy – to codify operational and datacenter level processes as a means to shift the burden of managing infrastructure from people to technology. Infrastructure services that can be provisioned on-demand, that can be managed on-demand, that can apply policies on-demand. PaaS is no different. It’s about leveraging services instead of libraries or adapters or connectors. It’s about platforms – data, application, messaging – as a service.

And here’s where I’ll diverge from agreeing with Simeon, because it shouldn’t matter to PaaS how the underlying infrastructure is provisioned or managed, either. I agree that virtualization isn’t necessary to build a highly scalable, elastic and on-demand cloud computing environment. But whether that data services is running on bare-metal, or on a physical server supported by an operating system, or on a virtual server should not be the concern of the platform services. Whether elastic scalability of a RabbitMQ service is enabled via virtualization or not is irrelevant. It is exactly that level of abstraction that makes it possible to innovate at the next layer, for PaaS offerings to focus on platform services and not the underlying infrastructure, for developers to focus on application services and not the underlying platforms. Thus his musings on the migration of IaaS into PaaS are ignoring that for most people, “cloud” is essentially a step pyramid, with each “level” in that pyramid being founded upon a firm underlying layer that exposes itself as services.

SOA IS ALIVE and LIVING UNDER an ASSUMED NAME for ITS OWN PROTECTION

If we return to the early days of SOA you’ll find this is exactly the same prophetic message offered by proponents riding high on the “game changing” technology of that time.

SOA promised agility through abstraction, reuse through a services-oriented approach to composition, and relieving developers of the need to be concerned with how and where a services was implemented so they could focus instead on innovating new solutions. That’s the same thing that all the *aaS are trying to provide – and with many of the same promises. The “cloud” plays into the paradigm by introducing elastic scalability, multi-tenancy, and the notion of self-service for provisioning that brings the financial incentives to the table.

The only thing missing from the “as a service” paradigm is a plethora of standards and the bad taste they left in many a developer’s mouth. And it is that facet of SOA that is likely the impetus for refusing to say the “S” word in close proximity to cloud and *aaS. The conflict, the disagreement, the confusion, the difficulties, the lack of interoperability that nearly destroyed the interoperability designed in the first place – all the negatives associated with SOA come to the fore upon hearing that TLA instead of its underlying concepts and architectural premises. Premises which, if you look around hard enough, you’ll find still very much in use and successfully doing exactly what it promised to do.

Simeon himself does not appear to disagree with the SOA-aaS connection. In a Twitter conversation he said, “I still have scars from the early #SOA days. Shouldn't we start with something simpler for PaaS?”

To which I would now say “but we are”. After all it wasn’t – and isn’t - SOA that was so darn complex, it was its myriad complex and often competing standards. A rose by any other name, and all that. We can refuse to use the acronym, but that doesn’t change the fact that the core principles we’re applying (successfully, I might add) are, in fact, service-oriented.

Related Posts

from tag IaaS

Cloud Today is Just Capacity On-Demand

Cloud computing is not Burger King. You can’t have it your way. Yet.

from tag PaaS

Despite Good Intentions PaaS Interoperability Still Only Skin Deep

Is PaaS Just Outsourced Application Server Platforms?

from tag SOA

Amazon Elastic Load Balancing Only Simple On the Outside

The End of 3-Teared Architectures

Redefining SOA

Have a can of Duh! It’s on me

Forklifts, Rip and Replace, and Other IT Fairy Tales

The Cloud Metastructure Hubub

Virtualization Gone Wild: Infrastructure as virtual appliances

Data as a Service Could Drastically Impact Success of SQL Injection Attacks

(more..)



Technorati Tags: MacVittie,F5,SOA,services,IaaS,PaaS,standards,infrastructure,Simeon Simeonov,virtualization

CloudFucius Asks: Will Open Source Open Doors for Cloud Computing?

Pete Silva — Tue, 20 Jul 2010 06:02:22 GMT

There has been a lot of press already about OpenStack’s announcement yesterday about their new open source cloud computing software. OpenStack says that the goal is, ‘to allow any organization to create and offer cloud computing capabilities using open source software running on standard hardware.’ The software is intended to to allow companies to automatically create and manage large deployments of virtual private servers and remove the concern of vendor lock-in since the software will allow customers to span multiple cloud providers. Customers and service providers alike can use their own physical hardware to create large cloud environments, public or private, across the globe. It is also positioned to give customers more choice in how they want their specific cloud environment designed and deployed. Almost 30 companies are participating with the folks at Rackspace and NASA (Nebula cloud computing platform) leading the charge.

Certainly, there are several attractive pieces to this, including the notion of cloud-standards, but will it finally open the flood gates for mass adoption of Cloud deployments? Maybe not for the enterprise, at least initially. Openstack honestly admits, ‘OpenStack is probably not something that the average business would consider deploying themselves yet. The big news for end customers is the potential for a halo effect of providers adopting an open and standard cloud: easy migration, cloud-bursting, better security audits, and a large ecosystem of compatible tools and services that work across cloud providers.’ This means that Openstack is really aimed at *very* technical enterprises (very large with lots of resources) and service providers. Thus, the play for the enterprise does not exist (yet) here, *except* for management layer players who could leverage it to build something they could sell to enterprises to “make it easy” for them. (thanks Lori!)

In addition, as Ted Julian of the Yankee Group points out in this story, security is still the great unknown since there doesn’t seem to be a security vendor on the list of Openstack participants. I’m sure that list will grow over time, especially with the press that it’s getting, and the ever present cloud security concerns will eventually be addressed. This project is in the very early stages and will continue to evolve as folks pick up the code, test it and decide how it might work for them. Maybe it’ll also help push along and enable the whole Inter-Cloud notion.

And one from Confucius: The cautious seldom err.

ps

The CloudFucius Series: Intro, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13

Resources:

The Recipe for Clouds Goes Open-Source

Open Stack Launches

NASA and Rackspace part the clouds with open source project

Rackspace's Risky Open Cloud Bet

Rackspace Launches Open-Source Cloud Platform Called OpenStack

NASA drops Ubuntu's Koala food for (real) open source

NASA and Rackspace open source cloud fluffer

All for One Cloud and One Cloud for All

Rackspace, NASA launch OpenStack: Can it prevent cloud lock-in?

OpenStack is great, but Clouds need security. Meet the Clean Cloud.

OpenStack Wiki

Openstack Code

Technorati Tags: F5, infrastructure 2.0, integration, cloud connect, Pete Silva, security, business, education, technology, application delivery, cloud, context-aware, web, internet, openstack

twitter: @psilvas

Many Stops is Good for Vacation, Not for WAN Opt

Don MacVittie — Tue, 20 Jul 2010 04:21:59 GMT

Anyone who has children and travels by car will tell you that there is no substitute for the mandatory array of bathroom breaks that must be taken by those children. One of the many reasons I prefer to travel at night when driving long distances is that children who are asleep are not asking to pull into the next rest stop for yet another restroom break. And I was one of those children. My father once told me I had the smallest bladder on the planet… Right before my mother made him stop at a gas station for me.

Another favorite is the “tourist trap” stop, where someone in the car with authority decides to stop, even though everyone over the age of 12 knows that the stop will be wasted on sites like “The World’s only three horned steer!” or “The ultimate hedge maze!” These may be things I’ve seen. May be.

When driving, these stops slow your drive to your destination and if someone was waiting for you, ultimately make you late, but since you are on a discrete vacation, this is not terribly bad. You get less time on-site, but in the end return home and get into your old routine.

This is completely not true with your data.

STOP! GO! STOP! WAIT!

Every stop on the network path that your data makes is a small disaster for your overall network throughput. The closer your WAN is to capacity, the greater the impact of each stop.

Those of us who have been hanging in the networking space for too long call it latency, and treat it as a bullet point on a brochure, but it’s a real beast that consumes resources, sometimes with horrendous effects on your network throughput.

The problem is basically the same one as a long car ride. You’re clipping along, someone sees a rest stop (in this case a network device of any kind – Application Acceleration, WAN Optimization, Encryption, whatever), and the packet stops off. The greater the latency, the longer the stop. And then it goes on until it sees the next one. Each stop may be small, but they’re stops that slow down the rate of the packet across the network/WAN/MAN/airwaves…

And because most network protocols require a response, and most network protocols are timed, that way lies doom. Imagine if there are 5000 cars behind you, and you have to stop at every restroom you pass, and they all have to follow you through the rest room. But there are 5000 cars coming the other way that have to wait for your cars to get off the road before they can take off. And if you’re too late they send a separate car to go check on you, and then a copy of you gets sent… Okay, maybe the car analogy is breaking down here… Because networks are more complex – there is one you, and you’re going to get to your destination. If a packet doesn’t make it to the destination in a timely manner, a request is sent to go get it, and then the packet comes again.

WHAT I WAS REALLY TRYING TO SAY WAS…

The more congested your network, the more problems latency causes, just because the chance of slowing things down too much grows (it’s not that simple, but close enough for this blog), and once you reach a certain level of packets not arriving in a timely manner, requests and retransmits start filling the rest of the over-burdened pipe and causing even more delays.

What can you do about it? First, make sure it is or might be a problem for you. Congestion and latency are measurable these days, and it should be pretty straightforward to determine if you have a device that’s gumming up the works. Figuring out where latency is introduced into a connection can be harder but is not insurmountable.

LESS STOPS, MORE GOODNESS

Also, combine functionality where it makes sense. Many years ago I came out against all-in-one security boxes because they offer a single point of failure for a whole array of security on your network, but if you already have a high latency link, they might just ease your pain by eliminating the time it takes to parse packets (among other things) multiple times. Several of our solutions act in much the same manner – utilizing TMOS on BIG-IP means that any of our solutions sold as a module - WOM, ASM, etc. can act on the packet in conjunction with load balancing requests. Other vendors do similar things by combining functionality (of course I believe not as well as F5, that’s why I work here), and each time you don’t have to touch the packet/envelope/etc, is one less tiny bit of latency you don’t have to worry about.

With multiple cores, SSL offload, and outrageously huge memory capacities, and Saas/Cloud making WAN communications a growing portion of our overall data communications, we have again turned the corner where the server doesn’t have to be the bottleneck, the network might well be. So it’s time to consider latency as more than a check-box again.

NOTE: There are many sources of latency, I’m merely drawing your attention to one that is completely within your control.

After all, the key difference between vacation and the network is that the packet doesn’t care if it stops or not, so don’t make it if you don’t have to.

Technorati Tags: Network Latency,WAN Optimization,Vacation,F5 WOM,F5 BIG-IP,F5 ASM,F5 Networks,Don MacVittie



Related Articles and Blogs

Latency Definitions on Wikipedia

Latency and IP

Verizon Business WW Network Latency Chart

How to Test Network and Internet Latency in MS-Windows (very basic)

Researchers Crack Network Latency Nut With New Algorithm

HTTP Now Serving … Everything

Lori MacVittie — Mon, 19 Jul 2010 19:06:15 GMT

You can’t assume anything about an application’s performance and delivery needs based on the fact that it rides on HTTP.

I read an interesting article during my daily perusal of most of the Internet (I’ve had to cut back because the Internet is growing faster than my ability to consume) on “Virtual Micro Networks.”

The VMN concept goes well beyond Virtual Local Area Networks (VLANs). Like VLANs or any other network, VMNs transport data from source to destination. But VMNs extend beyond transport to consider security, location, users, and applications. VMNs address:

Where is the information? The answer to this question used to be a physical server or storage device but application switching and server/storage virtualization makes this more dynamic and complex.

[…]

VMNs also must be aware of traffic type. For example, voice, video, and storage traffic is extremely latency-sensitive while HTTP traffic is not. Additionally, some network traffic may contain confidential information that should be encrypted or even blocked.

What are the specific characteristics of the information? Network-based applications may be made up of numerous services that come together at the user browser. How they get there isn’t always straightforward, thus the rise of vendors like Citrix NetScaler and F5 Networks. This is also where security comes into play as certain traffic may be especially sensitive, suspicious, or susceptible. [emphasis added]

What’s driving creation of Virtual Micro Networks

Okay, so first things first: the author really is using another term to describe what we’ve been calling for some time an application delivery network. That was cool in and of itself; not the emergence of yet another TLA but that the concept is apparently out there and rising. But what was even more interesting was the conversation this started on Twitter. If you don’t follow @csoandy (that’s the Twitternym of Andy Ellis of Akamai Networks) you might want to start. Andy pointed out that the statement “HTTP traffic is not latency-sensitive” is a bit too broad and went on to point out that it really depends on what you’re delivering. Live video, after all, is sensitive to latency no matter what protocol is transporting it.

Andy put it well in an off-line conversation when he said, “There's also the myth that HTTP isn't for low latency apps. HTTP lets you take advantage of optimizations done in TCP and HTTP to accelerate delivery.” A great point and very true. All the “built-in acceleration and optimization” of an application delivery controller’s TCP stack is free for HTTP because after all, HTTP rides on TCP. But ironically this is also where things get a bit wonky.

The reality is that the application data is sensitive, not the protocol. But because the data HTTP was initially designed to transport was not considered to be latency sensitive, you almost have to look at HTTP as though it is, which is why the broad statement bothered Andy in the first place. We wouldn’t say something like “TCP” or “UDP” is not sensitive to latency because these are transport layer protocols. We need to know about the data that’s being transported. Similarly, we can’t (anymore) say “HTTP isn’t sensitive to latency” because HTTP is the de facto transport protocol of web applications. As I remarked to Andy, the move to deliver everything via HTTP changes things significantly. “Things” being the entire realm of optimization, acceleration, and application delivery.

Context is Everything

As the initial blog post that started this conversation pointed out, and which nothing Andy and I discussed really changed, is that our nearly complete reliance on HTTP as the de facto transport protocol for everything means that the infrastructure really needs to be aware of the context in which requests and responses are handled.

When an HTTP GET and its associated response might in one case be a simple binary image and in another case it might be the initiation of a live video stream, well, the infrastructure better be able to not only recognize the difference but handle them differently. HTTP doesn’t change regardless, but the delivery needs of the data do change.

This is the “application aware” mantra we (as in the entire application delivery industry) have been chanting for years. And now it’s becoming an imperative because HTTP no longer implies text and images, it implies nothing. The infrastructure responsible for delivering (securing, optimizing, accelerating, load balancing) the access to that application data cannot assume anything; not session length, not content length, not content type, not optimal network conditions. The policies that might ensure a secure, fast, and available web application are almost certainly not the same policies that will provide the same assurance for video, or audio, or even lengthy binary data. The policies that govern the delivery of a user-focused application are not the same ones that should govern the delivery of integration-driven applications like Web 2.0 and cloud computing APIs. These are different data types, different use cases, different needs.

Yet they are all delivered via the same protocol: HTTP.

Dynamic Infrastructure Needed

What makes this all even more complicated (yes, it does get worse as a matter of fact) is that not only is the same protocol used to deliver different types of data but in many cases it may be delivered to the same user in the same session.

A user might move from an article or blog to a video back to text all the while a stream of Twitter and Facebook updates are updating a gadget in that application. And the same infrastructure has to handle all of it. Simultaneously.

Wheeeeeeeeeee!

That HTTP can be extended (and has been, and will continue to be) to include broad advanced capabilities has been a blessing and a curse, for as we deliver more and more differenter content over the same protocol the infrastructure must be able to ascertain dynamically what type of data is being delivered and apply the appropriate actions dynamically. And it has to incorporate user information, as well, because applications highly sensitive to latency need special care and feeding when delivered over a congested, bandwidth constrained network as opposed to delivery via a high-speed, low latency LAN. The application delivery network, from user to application and back, must be context-aware and able to “turn on a dime” as it were, and adjust delivery policies based on conditions at the time of the request and subsequent responses. It’s got to by dynamic.

Consider the comparison offered by Andy regarding video served via traditional protocols and HTTP:

Consider a live stream; say, Hope for Haiti. A user opens a browser, and has a small embedded video, with a button to expand to full screen. With most streaming protocols, to get a higher resolution stream, your player needs to either:

a) start grabbing a second, high res stream in the background, and guess when to splice them over. (now consider if the stream is too fat, and you need to downgrade)

b) pause (drop existing stream) and grab a new stream, exposing buffering to a user.

c) signal somehow to the streaming server that it should splice in new content (we built this. it's *hard* to get right. And you have to do it differently for each protocol).

With HTTP, instead what you see is:

a) Browser player is grabbing short (usually 2) second chunks of live streaming content. When it detects that it is fuller screen, and, inferring available bandwidth by how long it takes to download a chunk, ask for a higher resolution chunk for the next available piece.

Quite the difference, isn’t it? But underlying that simplicity is the ubiquity of HTTP and a highly dynamic, flexible infrastructure capable of adapting to the sensitivities specific not only to the protocol, to the data, but to the type of data being delivered.

So it turns that both Andy and I are both right, it just depends on how you’re looking at it. It isn’t that HTTP is sensitive to latency, it isn’t. But the data being delivered over HTTP most certainly is. But it is confusing to discuss HTTP in broad, general terms because you can’t assume anymore that what’s being delivered is text and images. We don’t talk in terms of TCP when we talk web applications, so maybe it’s time to stop generalizing about “HTTP” and start focusing on applications and data, about the content, because that’s where the real challenges surrounding performance and security are hiding.

Related Posts

What’s driving creation of Virtual Micro Networks

HTTP: The de facto application transport protocol of the Web

from tag context-aware

Top-to-Bottom is the New End-to-End

Call Me Crazy but Application-Awareness Should Be About the Application

Cool Hand Cloud

When Everything is a Threat Nothing is a Threat

The IP Address - Identity Disconnect

What if users could specify their own SLAs?

WARNING: Security Device Enclosed

The End of DNS As We Know It

Denied!

Intercloud: The Evolution of Global Application Delivery

Picard and Dathon at El-Adrel

Infrastructure 2.0: As a matter of fact that isn't what it means

(more..)


Technorati Tags: MacVittie,f5,Akamai,Andy Ellis,HTTP,context-aware,application-aware,application delivery network,dynamic control plane,infrastructure 2.0,application delivery

Securing the Corporate Intranet with Access Policy Manager

George Watkins — Mon, 19 Jul 2010 07:44:11 GMT

Fire doors are used to minimize damage to a structure during a fire. In the event of a fire, a central monitoring system will trigger the release of the doors isolating flames and smoke at the epicenter of the disaster while protecting adjacent sectors. While most of our office buildings are built in this fashion, corporate IT environments are largely built like multi-story stick-built mansions with no segregation mechanisms. Someone playing with fire in the tool shed can bring the entire house to the foundation. Does your infrastructure have any such safety net?

When engineers hear ‘VPN’ or access management, they think of a device that sits on the edge between the corporate intranet and the Internet. All too often, internal corporate traffic is allowed to bypass these barriers and access corporate resources directly. While there may be authentication mechanisms protecting these services, it is almost impossible to secure them all without placing some barrier between them and the users. Enter APM.

APM is well suited as a demarcation point not only for access from the Internet, but also from the corporate user space. In any organization, users are inherently your largest security risk. Whether a user is accessing the intranet from a non-corporate sponsored machine from their house or a workstation in their cubicle, they should be expected to adhere to the same security standards.

In a “flat” security model, all corporate users can see any resource (even if it is just a login page) whether or not they meet the authorization requirements. Is there really any reason why John in product development should ever be able to see the corporate payroll system? Even if he can’t access the information, he can still see it. That means if John’s workstation is compromised, an attacker may be able to use his access to execute an exploit against this system thereby gaining access to precious corporate information. This may be a far-fetched example, but such things have happened and in many cases they’ve made major news headlines. This is not the kind of press any large corporation wants.

In terms of physical security, we can use APM to ensure that every packet that is exchanged between our user subnets and the secure corporate space is encrypted. This protects any insecure data from being compromised by an eaves dropping attack and does so at a fraction of the cost of fiber.

If properly implemented, Access Policy Manager should not present any hurdles to the end user. In fact, with single sign-on (SSO) credential mapping features, APM should actually improve user experience. This means when Joan in human resources logs into her Edge Client in the morning and receives her session for the day, APM will cache her credentials. She won’t have to type her credentials every time she access the payroll system because APM will enter them for her.

The end result of securing your corporate resources with APM is increased security for your environment and improved accessibility for your users. While nothing can protect against every potential security breach, APM can go a long way in ensuring that a single attack will not bring your organization to its knees.

A funny Thing Happened on the Way to the Branch Office…

Don MacVittie — Mon, 19 Jul 2010 04:04:59 GMT

Where you are going has a huge impact upon the mode of transportation that you choose to use. If you are going to the neighbor’s house, you tend to walk. If on your way to town or downtown, you tend to take a motorized vehicle. Out for a leisurely trip around the subdivision? You are likely going to ride a bike. Going to another continent, probably a plane, maybe a ship… You get the idea.

Several years ago (while working for Network Computing) I reviewed WAN Optimization products, with an eye to file transfer acceleration. Interestingly, F5 decided not to participate in that review (I won’t conjecture why, they weren’t alone). At the time, all WAN Optimization was treated pretty much equally. Tacit Networks (later bought by Packeteer, who was subsequently purchased by Blue Coat) was pretty zeroed in on remote office access and acceleration, but all of the other vendors I looked at were, at the time, treating all WAN Optimization greatly the same.

These days that’s far from the truth, with different feature sets of WAN Optimization products aimed at DC-to-DC or DC-to-Cloud versus DC-to-remote office. Talking over the differences and how they influence product development with one of our uber-smart PD people spawned some intriguing points that make these product sets completely different, though they may well use the same set (or overlapping sets) of functionality.

Back in the day, Tacit networks had approached the problem by going very Microsoft heavy. They tied their file acceleration to Microsoft technology, along with several other “add-on” functionality bits. It made their product unique, but overall the product didn’t fit well into our review because we were looking at something more acceleration focused and they were more general remote office focused (you could accelerate a Windows-shared printer in the DC, for example). These days the industry has a more firm grasp on what it takes to serve a remote office versus a remote DC, and the devices share a decent amount of functionality while still targeting different market needs.

The thing that got this blog started was pretty simple, but I had not paused to consider. Remote office devices pretty much require fail-to-wire functionality because there is no other route to get out of most remote offices, and there is often no IT staff at the remote office to make the routing changes. DC-to-DC devices however do not require fail-to-wire because there are alternate routes out of the building and plenty of staff to jump if issues come up. This issue becomes less clear if you consider the inclusion of security functionality in WAN Optimization devices. Do you fail to wire on a device that is supposed to be providing edge security? And if not, what are the business implications of rejecting all connections if something goes wrong with the box. This conundrum pretty much guarantees that if the application/connection is important enough to put a WAN Optimization device onto, you should probably get a redundant pair right at the outset.

Never easy for the IT person, is it? There have been some great changes – like integrated security and all-in-one QoS – that do make your life easier, but they increase the workload… At a minimum on initial configuration and likely off-and-on over the life of the product. But if it keeps you from buying more bandwidth, secures the connection without burden on servers, and handles QoS issues by balancing how much bandwidth a given protocol/app can use up… A few hours setting it up and tweaking configs is reasonable, no?

No doubt there will be more to come on similar topics, I have a ton of notes from these conversations, and there’s some cool stuff hiding in there.

Technorati Tags: WAN Optimization,F5 WOM,Microsoft,F5 Networks,Remote office connectivity,Don MacVittie



Related Articles and Blogs

F5 Steers WAN Optimization to Virtual, Cloud Environments

WAN Optimization 101: Know Your Options

Like a Matrushka, WAN Optimization is Nested

Stop Repeating Yourself: Deduping WAN Opt Style

Defeating Attacks Easier Than Detecting Them

Lori MacVittie — Sun, 18 Jul 2010 19:15:41 GMT

Defeating modern attacks – even distributed ones – isn’t the problem. The problem is detecting them in the first place.

Last week researchers claimed they’ve discovered a way to exploit a basic security flaw that’s used in software that’s in high use by Web 2.0 applications to essentially support if not single-sign on then the next best thing – a single source of online identity.

The prevalence of OAuth and OpenID across the Web 2.0 application realm could potentially be impacted (and not in a good way) if the flaw were to be exploited. Apparently a similar flaw was used in the past to successfully exploit Microsoft’s Xbox 360. So the technique is possible and has been proven “in the wild.”

The attacks are thought to be so difficult because they require very precise measurements. They crack passwords by measuring the time it takes for a computer to respond to a login request. On some login systems, the computer will check password characters one at a time, and kick back a "login failed" message as soon as it spots a bad character in the password. This means a computer returns a completely bad login attempt a tiny bit faster than a login where the first character in the password is correct.

[…]

But Internet developers have long assumed that there are too many other factors -- called network jitter -- that slow down or speed up response times and make it almost impossible to get the kind of precise results, where nanoseconds make a difference, required for a successful timing attack.

Those assumptions are wrong, according to Lawson, founder of the security consultancy Root Labs. He and Nelson tested attacks over the Internet, local-area networks and in cloud computing environments and found they were able to crack passwords in all the environments by using algorithms to weed out the network jitter.

Researchers: Password crack could affect millions

ComputerWorld, July 2010

Actually, after reading the first few paragraphs I’m surprised that this flaw wasn’t exploited a lot sooner than it was. The ability to measure fairly accurately the components that make up web application performance is not something that’s unknown, after all. So the claim that an algorithm can correctly “weed out” network latency is not at all surprising.

But what if the performance was randomized by, say, an intermediary interjecting additional delays into the response? You can’t accurately account for something that’s randomly added (or not added, as the case may be) and as long as you seeded the random generation with something that cannot be derived from the context of the session there are few algorithms that could figure out what the random generation seed might be. That’s important because random number generation often isn’t and it can often be predicted based on knowing what was used to seed the generator. So we could defeat such an attack by simply injecting random amounts of delay into the response.

Or, because the attack depends on an observable difference in timing, simply normalizing response times for the login process would also defeat this attack. This is the solution pointed out in another article on the discovery, “OAuth and OpenID Vulnerable to Timing Attack”, in which it is reported developers of impacted libraries indicate that just six lines of code will solve this problem by normalizing response times. This, of course, illustrates a separate problem, which is the reliance on external sources to address security risks that millions may be vulnerable to now because while it’s a simple resolution, it may takes days, weeks, or more before it is available.

This particular attack would indeed be disastrous were it to be exploited given the reliance on these libraries by so many popular web sites. And though the solutions are fairly easy to implement, that isn’t the real problem. The real problem is how difficult such attacks are becoming to detect, especially in the face of the risk incurred by remaining vulnerable while solutions are developed and distributed.

DEFEATING is EASY. DETECTING is HARD.

The trick here, and what makes many modern attacks so dangerous is that it’s really really hard to detect them in the first place.

Any attack that could be distributed across multiple clients – clients smart enough to synchronize and communicate with one another – becomes difficult to detect, especially in a load balanced (elastic) environment in which those requests are likely spread across multiple application instances. The variability in where attacks are coming from makes it very difficult to see an attack occurring in real-time because no single stream exhibits the behavior most security-focused infrastructure watches for.

What’s needed to detect such an attack is to be more concerned with what is being targeted rather than by whom or from where. While you want to keep track of that data the trigger for such brute-force attacks is the target, not the client activity. Attackers are getting smart, they know that repeated attempts at X or Y will be detected and that more than likely they will find their client blacklisted for a period of time (if not permanently) so they’ve come up with alternative methods that “hide” and try to appear like normal requests and responses instead. In fact it could be postulated that it is not repeated attempts to login from a single location that are the problem today, but rather the attempt to repeatedly login from multiple locations across time that’s the problem. So what you have to look for is not necessarily (or only) repeated requests but also at repeated attempts to access specific resources, like a login page.

But a login page is going to see a lot of use so it’s not just the login page you need to be concerned with, but the credentials, as well. In any brute force account level attack there are going to be multiple requests to try to access a resource using the same credentials. That’s the trigger. It requires more context than your traditional connection or request based security triggers because you’re not just looking at IP address, or resource, you’re looking deeper and trying to infer from a combination of data and behavior what’s going on.

THIS SITUATION will BECOME the STATUS QUO

As we move forward into a new networking frontier, as applications and users are decoupled from IP addresses and distribution of both clients and applications across cloud computing environments becomes more common, the detection of attacks is going to get even more difficult without collaboration.

The new network, the integrated and collaborative network, the dynamic network, is going to become a requirement. Network and application network and security infrastructure needs a new way of combating modern threats and their attack patterns. That new way is going to require context and the sharing of that context across all strategic points of control at which such attacks might be mitigated.

This becomes important because, as pointed out earlier, many web applications rely upon third-party solutions for functionality. Open source or not, it still takes time for developers to implement a solution and subsequently for organizations to incorporate and redeploy the “patched” application. That leaves users of such applications vulnerable to exploitation or identity theft in the interim. Security infrastructure must be able to detect such attacks in order to protect users and corporate resources (infrastructure and applications and data) from exploitation while they wait for such solutions. When is more important than where when it comes to addressing newly discovered vulnerabilities, but when is highly dependent upon the ability of infrastructure to detect an attack in the first place.

We’re going to need infrastructure that is context-aware.

Related Posts

from tag brute-force

Stop brute force listing of HTTP OPTIONS with network-side scripting

from tag Twitter

Are You Scrubbing the Twitter Stream on Your Web Site?

Twitter Account Lockouts Continue to Plague Users

from tag context-aware

The IP Address - Identity Disconnect

Infrastructure 2.0: As a matter of fact that isn't what it means

When Everything is a Threat Nothing is a Threat

Denied!

Cool Hand Cloud

Call Me Crazy but Application-Awareness Should Be About the Application

WARNING: Security Device Enclosed

Picard and Dathon at El-Adrel

(more..)



Technorati Tags: MacVittie,F5,security,web application security,OAuth,OpenID,context-aware,Web 2.0,Digg,Twitter,brute-force,timing attack,web application firewall,infrastructure 2.0

Cloud Storage Gateways, stairway to (thin provisioning) heaven?

Don MacVittie — Fri, 16 Jul 2010 01:15:04 GMT
With thanks to Led Zeppelin for some great lyrics.

There's a sign on the wall
But she wants to be sure
'Cause you know sometimes words have
Two meanings

Since cloud computing has a bit of an identity crisis, and cloud storage is just starting to realize one itself, it should be no surprise to anyone that “cloud storage gateway” has more than one meaning. While they are all a single market, implementation and deployment details make them very distinct products. In such a young market, differentiation is easy, even if selling your differentiation as a plus is not. Some vendors are already attempting to turn product differentiation into market segmentation – the upcoming Cirtas product, for example, is referred to by them as a “Cloud Storage Controller” because they believe that better defines their product, though they acknowledge that the market term “gateway” has taken on.

When she gets there she knows
If the stores are all closed
With a word she can get what she came for

Okay, so you don’t quite have that power, but all of these products do offer you a significant bonus in terms of cloud storage in the form of thin provisioning. For several years now you have had the capability to tell your server it had more storage than was actually dedicated to it, and if it ran over what was actually dedicated, more was allocated from a pool. The problem with this model is that you have to be certain you have enough storage to cover the worst reasonable case – what percentage of your servers might request extra storage over the weekend, and how much might they request. Weekend, month, year… Whatever your timeframe for buying new storage. The point of over-provisioning is that you’ll likely be oversubscribed, but you’re taking the risk that the oversubscription will never come due at the same time with one large calamitous bang. I wrote about this scenario and how virtualization has made the risk worse here.

Yes, there are two paths you can go by
But in the long run
There's still time to change
The road you're on

Enter Cloud Storage Gateways. First a bit about cloud providers… They scale up to as much as you need (as long as your payments are covering it anyway), and down as your usage goes down. I won’t say all of them fit this pattern because there are a bewildering number of players looking to make a name in this space, and believe it or not, F5 doesn’t pay me to ponder cloud storage or cloud storage gateways, they merely allow me to chat about it, but I’m not taking weeks researching all of this, more like hours. The big players do indeed scale up and down, only billing you for actual usage though.

Now that we have covered that for any who didn’t know, the cloud storage gateway handles the intricacies of dealing with various cloud storage providers, and most cache locally and encrypt on the way out.

Starting to see the silver lining yet? They give you thin provisioning limited only by how much money you’re willing to risk. The current model gives you thin provisioning limited by either how much you’re willing to pay to guarantee you have enough disk for the worst case, or the amount of risk you’re willing to take on. Cloud storage gateways navigate that mine-laden sea for you and guarantee that your servers will stay up as long as you’re willing to foot the bill.

Of course that doesn’t eliminate planning for you, but it does allow you to move the choke point up and down much more easily. You can eliminate the risk without a significant cash outlay, if that is your desire, as long as you know what it will cost you if your servers all start requesting more and more storage.

And it makes me wonder

(if you just went “Ooohhh Yeahhh-hah” in your head, take a moment to laugh at yourself, it’s healthy)

The biggest risk in cloud storage gateways is the one I mentioned in a previous blog. If they are scooped up by cloud storage vendors that suddenly remembered the lingua franca of enterprise IT storage is not SOA, they will surely limit your options on the back-end. One of the strengths of these products is that you can point one at two completely different cloud storage vendors and remind each that with a flick of a switch, you can be on the other one, so 48 hour response times are not acceptable. That benefit would almost certainly disappear if a cloud storage vendor bought up your cloud storage gateway vendor. Otherwise, the risk is not any larger than any other cloud solution.

Your head is humming and it won't go
In case you don't know
The piper's calling you to join him

Cloud everything is the buzz du-jour, and cloud storage with a gateway to make it appear on your network as a NAS is a good idea for tier three, and these vendors are all (I’ve only spoken to three, so “all” is a bit of a stretch) saying they’re getting traction on tier two also, which makes sense for a lot of tier two data. Either way, it’s coming your way, and you should consider if it has a space in your datacenter. The idea of truly thin provisioning is a huge one that even further removes you from the limitation of monstrous disk arrays.

And if thin provisioning with no subscription worries is on your list of things that would help you sleep at night, I suggest you go out and try…

Bu-uying a stairway… To (thin provisioning) heaven

And I won’t even get into what these solutions coupled with the automated tiering of ARX can do for you. That’s for another blog.

Technorati Tags: Cloud Storage Gateways,Cloud Storage,Thin Provisioning,Cirtas,F5 Networks,Don MacVittie



Related Articles and Blogs

Stairway to Heaven Lyrics

Thin Provisioning Plus VMs - Armageddon in a Virtual Box

Cloud Storage Gateways: Short Term Win, but Long Term…?

More Cloud Storage Gateways Come Out

Show Me The Gateway – Talking Storage to the Cloud

F5 Friday: An On-Demand Turing Test

Lori MacVittie — Thu, 15 Jul 2010 19:08:30 GMT

Detecting bots requires more than a simple USER_AGENT check today…

Anyone who’s taken an artificial intelligence class in college or grad school knows all about the Turing Test. If you aren’t familiar with the concept, it was a “test proposed by Alan Turing in his 1950 paper Computing Machinery and Intelligence, which opens with the words: "I propose to consider the question, 'Can machines think?'" Traditional Turing Tests always involve three players, and the goal is to fool a human interviewer such that the interviewer cannot determine which of the two players is human and which is a computer. There are variations on this theme, but they are almost always focused on “fooling” an interviewer regarding some aspect of the machine that it is attempting to imitate.

Common understanding has it that the purpose of the Turing Test is not specifically to determine whether a computer is able to fool an interrogator into believing that it is a human, but rather whether a computer could imitate a human.^[44] While there is some dispute whether this interpretation was intended by Turing — Sterrett believes that it was^[43] and thus conflates the second version with this one, while others, such as Traiger, do not^[41] — this has nevertheless led to what can be viewed as the "standard interpretation." In this version, player A is a computer and player B a person of either gender. The role of the interrogator is not to determine which is male and which is female, but which is a computer and which is a human.^{[45]

-- Wikipedia, Turing Test}

Over the past decade, as the web has grown more connected and intelligent, so too have the bots that crawl its voluminous pages attempting to index the web and make it possible for search engines like Google and Bing to be useful. Simultaneously have come the evil bots, the scripts, the automated attempts at exploiting vulnerabilities and finding holes in software that enable malicious miscreants to access data and systems to which they are not authorized. While a web application firewall and secure software development lifecycle practices can detect an attempted exploit, neither are necessarily very good at determining whether the request is coming from a bot (machine) or a real user.

Given the very real threat posed by bots, it’s becoming increasingly important for organizations to detect and prevent these automated digital rodents from having access to web applications, especially business-critical applications. The trick is, however, to determine which requests are coming from bots and which ones are coming from real users. It’s a trick not only because this determination is difficult to make with a high degree of confidence in the result, but because it needs to be determined on-demand, in real-time.

What organizations need is a sort of “on-demand Turing test” that can sort out the bots from the not bots.

HUMAN INTERACTION HEURISTICS

In a recent release of F5’s web application firewall, ASM (Application Security Manager), there was a very neat little “trick” that essentially provides just what we’ve described: an on-demand Turing test. This capability of ASM attempts to detect whether or not the client side is human by monitoring client-side interaction events. The assumption is, of course, that a bot will not be interacting with the page but a human must interact – clicking a mouse, typing – in order to interact with the application.

The intention behind this feature is to allow administrators to configure policies that act upon the determination. This is particularly helpful in preventing web scraping and other automated attack tools that make use of scripts and bots to attack sites.

How does it work?

Because ASM is deployed as part of a unified application delivery platform, it is able to take advantage of the systems’ underlying foundational technologies. In this case, TMOS – a full proxy that provides the capability to intercept, inspect, and transform application data. It is application aware, and can interact in real-time with the messages being exchanged between the client and the server. In much the same way as we leverage this capability to inject the Gomez Javascript code necessary to monitor any application for which BIG-IP provides application delivery services, we can inject Javascript into clients that provides the information necessary to determine whether the client is a human being or an automated system.

Based on the results of the determination, the administrator can take action. Rejecting the connection or redirecting to a honey-pot are common actions to take, but ASM like all TMOS-enabled solutions offered as part of the BIG-IP family of solutions the limits are really based on organizational need. iRules (network-side scripting) can be employed to provide additional custom and unique actions, logging, etc…or a pre-defined set of actions can be easily applied. This also provides some measure of future-proofing against modifications by miscreants to bots and automated scripts. Evolving the heuristics used to determine human from bot can be updated without requiring upgrades to the system, requiring modification of injected scripts. the ability to leverage iRules further allows organizations to develop their own methods that may or may not be based on ASM’s included functionality.

This is not a panacea. There is no web application security solution that is. But it is more than what is currently available in most web application security solutions today, and it’s something that isn’t generally included in the application itself, for a variety of reasons. It’s certainly not listed in the OWASP top ten methods for securing an application. There are legitimate uses for bots, after all, so the use of such a deterministic system to unilaterally deny or permit requests must be carefully balanced across the typical usage patterns and needs of the organization.

Regardless, it is another tool in the information security toolbox that can be applied to better secure applications against “users” of malicious intent, and it’s certainly moving web application security in the right direction by being as context-aware as possible when attempting to respond to a request for a resource.

Related Posts

All F5 Friday Entries on DevCentral

from tag bot

Turning the Pushdo Bot Into the Push-oh-no-you-don’t Bot

from tag javascript

Facepalm: Google Wave Choice of XMPP Not the Death of HTTP

Why Flash can't win the Web application war

Why it's so hard to secure JavaScript

A Billion More Laughs: The JavaScript hack that acts like an XML attack

Working around client-side limitations on custom HTTP headers

from tag security

Telecommute your way to a greener bottom line

JSON versus XML: Your Choice Matters More Than You Think

WILS: How can a load balancer keep a single server site available?

3 reasons you need a WAF even if your code is (you think) secure

An Unhackable Server is Still Vulnerable

Security is not a luxury item

(more..)


Technorati Tags: MacVittie,F5,F5 Friday,web application security,Turing,bot,security,Web 2.0,javascript,application delivery

F5 Tech Field Day – Delegate Roundtable

DevCentral TV — Thu, 15 Jul 2010 14:02:39 GMT

Web Magazine Gestalt IT brings together innovative IT product vendors and independent thought leaders who have immense influence on the ways that products and companies are perceived and understood by the general public.

F5 Networks hosted a session for the Tech Field Day on Thursday, July 15th.

In the final session, the delegates have a round table discussing their thoughts on the presentations this afternoon at F5 Networks.

Technorati Tags: F5,DevCentral,Gestalt IT,TechFieldDay

F5 Tech Field Day – Storage Optimization And A Practical Tiering Demo

DevCentral TV — Thu, 15 Jul 2010 13:59:39 GMT

Web Magazine Gestalt IT brings together innovative IT product vendors and independent thought leaders who have immense influence on the ways that products and companies are perceived and understood by the general public.

F5 Networks hosted a session for the Tech Field Day on Thursday, July 15th.

In this session, F5’s Rick Gillett and Bob Blair talk about Storage Optimization and give a Practical Tiering Demo.

Technorati Tags: F5,DevCentral,Gestalt IT,TechFieldDay,Rick Gillett,Bob Blair,Storage,Optimization,Tiering

F5 Tech Field Day – Security: Compliance And Access Control

DevCentral TV — Thu, 15 Jul 2010 13:57:09 GMT

Web Magazine Gestalt IT brings together innovative IT product vendors and independent thought leaders who have immense influence on the ways that products and companies are perceived and understood by the general public.

F5 Networks hosted a session for the Tech Field Day on Thursday, July 15th.

In this session, F5's Nogan Moshiri discusses Security with regards to compliance and access control.

Technorati Tags: F5,DevCentral,TechFieldDay,Nojan Moshiri,Security,Compliance,Access Control

F5 Tech Field Day - Tales From The Woods With iControl and iRules

DevCentral TV — Thu, 15 Jul 2010 13:53:29 GMT

Web Magazine Gestalt IT brings together innovative IT product vendors and independent thought leaders who have immense influence on the ways that products and companies are perceived and understood by the general public.

F5 Networks hosted a session for the Tech Field Day on Thursday, July 15th.

In this session, F5’s Joe Pruitt takes the delegates on a walk through the woods with iControl and iRules in action.

Technorati Tags: F5,DevCentral,Gestalt IT,TechFieldDay,iControl,iRules,Joe Pruitt

F5 Tech Field Day – Orchestration Of VMWare VMotion Over Distance

DevCentral TV — Thu, 15 Jul 2010 13:50:09 GMT

Web Magazine Gestalt IT brings together innovative IT product vendors and independent thought leaders who have immense influence on the ways that products and companies are perceived and understood by the general public.

F5 Networks hosted a session for the Tech Field Day on Thursday, July 15th.

In this session, F5's Erick Hammersmark gives a live demo of Orchestration of VMWare illustrating non-disruptive VMotion over Distance.

Technorati Tags: F5,DevCentral,Gestalt IT,TechFieldDay,VMWare,Orchestration,vMotion,Erick Hammersmark

F5 Tech Field Day – Intro To F5 As A Strategic Point Of Control

DevCentral TV — Thu, 15 Jul 2010 13:44:36 GMT

Web Magazine Gestalt IT brings together innovative IT product vendors and independent thought leaders who have immense influence on the ways that products and companies are perceived and understood by the general public.

F5 Networks hosted a session for the Tech Field Day on Thursday, July 15th.

In this session, F5’s Dan Matte and Kirby Wadsworth welcome the bloggers and give an overview of F5 as a Strategic Point of Control.

Technorati Tags: F5,DevCentral,Gestalt IT,TechFieldDay,Dan Matte,Kirby Wadsworth

Cloud is Defined, Right?

Don MacVittie — Thu, 15 Jul 2010 04:32:59 GMT

At the end of the odd but intriguing movie Existenz, one of the primary characters looks at the other after killing a bunch of people and says “We’re still in the game, right?” With the implication that you the viewer really don’t know if they’re still in the Virtual Reality game they were playing. Sometimes, Cloud feels like that. I can just go “We’re still in the cloud, right?”

Here we are, it is 2010, the pundits have been hailing cloud for years, and yet there is still a vast gulf of understanding of what is the cloud, exactly out there. Recently I was involved in a Twitter conversation with Mike Fratto (of Network Computing), Andy Ellis (of Akamai), Lori (of F5, as am I), with occasional input from Dustin Amrhein (of IBM), Greg Knieriemen (of Chi Corp and the InfoSmack podcast), Vanessa Alvarez (of Frost and Sullivan), and Tom Petrocelli (formerly of IP.com) where it became painfully clear that it is indeed not at all settled. Not even amongst such an august group of individuals.

The thing is, that nearly two years ago, Lori gave “The Last Cloud Definition You’ll Ever Need”, and laid out what Cloud was and was not. It’s a good definition, but it is one that many vendors do not want to acknowledge for a variety of reasons, the most prevalent of which is marketing of their own products.

In this conversation, Andy was simply asking “is a Content Delivery Network (CDN) not cloud?” now you might be quick to say “marketing! He works for Akamai!”, but my experience is that he’s deeper than that, and not really a marketer… He IS the CSO of Akamai after all, not exactly a marketing position. Also, his question made sense to me. A CDN delivers content across a wide geography on demand. A CDN is billed very much like all cloud services are billed, and many have APIs to manipulate what’s out there, how it’s delivered, etc.

It didn’t make sense to Lori, and she was continuing the conversation with a different set of individuals in that wonderful “Twitter fracture effect” that causes conversations to veer off not just in topic but participants also, they generally seemed to be in agreement with her assessment. Mike was pretty much in agreement with her also.

I spend a lot of time talking cloud with Lori because we’re together essentially all of the time, and she’s largely focused in that space right now. I don’t see it as too clear. If Cloud is or includes Infrastructure-As-A-Service (IaaS), and CDN was the original IaaS product, then it seems to me that it is worth exploring whether it is indeed not cloud just by virtue of being what it is.

In the definition I linked to above, CDN is excluded by the simple fact that it is not an application delivery mechanism, but a content delivery mechanism. But that definition would preclude cloud storage from the definition of “what is cloud” also. And I think that cloud storage meets all of the numbered points in the definition, though due to the difference between storing data and delivering applications, point number four is only adhered to by some cloud storage providers.

The NIST Notional Definition of Cloud Computing (MS-Word Doc) is commonly referenced by Cloud aficionados, and also seems to limit “Cloud” to application services, which would imply that there is no “Cloud Storage”, see above for my views on that notion.

The problem is that you just can’t say (as Brenda Michelson of Elemental Links said while I was writing this blog) “The tubes formerly known as web”. She was joking, even used the #snark tag, but we do need a definition that goes beyond application delivery, or at least makes cloud storage fit into the application delivery paradigm (heh. I said paradigm in a blog. I’m so 90s) that is currently en-vogue.

I’m trying to get a handle around the issue, because if you read the SNIA Cloud Storage documentation, it doesn’t clear the issue up, it ignores it and says “all these things are “cloud storage”. Nice, but how does that fit into the current definitions of cloud? And that doesn’t even touch Microsoft offering MS-SQL Server on Azure, which is technically SaaS, but smacks of cloud because of whom the customer is…

I think that is where we come to the key to defining cloud as it exists today. It does leave some vendors out, but only those aiming at end users, and I’d argue those are cloud-delivered applications.

So how about this for a new definition…

“IT services designed to interact with and/or take the place of core enterprise IT hardware or software infrastructure”.

It’s short, and no doubt I’ll get feedback that will help refine it, but it does encompass the core – this would be inclusive of Databases as services, it would be inclusive of what we all think of traditionally as “cloud” – server allocation – it would include cloud storage, it would rule out applications with end user interfaces that happen to be hosted in cloud or cloud like environments because those have never been considered “infrastructure”. It would include CDN, but if you’re going to put cloud storage in the bucket, you’re going to have to accept that CDN is a service that meets the above definition.

It fits, it captures the spirit of cloud computing, and it leaves out those IaaS vendors that have been a serious bone of contention for everyone else. Some IaaS vendors can claim “Hosted in the cloud”, and that would make everyone happy, just don’t claim to be “the cloud” if your target user is an end user and not an administrator.

Feedback, conversations, commentary, and even flame are all welcome. Visits from the men in white coats will not be appreciated.

And thanks to all those listed. My blog topic for today was not nearly as interesting as this one.

Technorati Tags: Cloud,Cloud Infrastructure,Cloud Computing,Cloud Storage,F5 Networks,IBM,Akamai,Network Computing Magazine,Elemental Links,Chi Corp,Frost and Sullivan,Azure,Microsoft,Don MacVittie



Related Articles and Blogs

Wikipedia Cloud Computing Entry

What Cloud Computing Really Means

Twenty One Experts Define Cloud Computing

Cloud Computing – The Last Definition You’ll Ever Need

DevCentral Weekly Podcast 140 – Hostname Blackberryization

DevCentral Weekly Podcast — Thu, 15 Jul 2010 03:23:52 GMT

   Welcome to the one hundred and fortieth edition of the DevCentral Weekly Roundtable Podcast! A weekly recap of the interesting things that have been going on within the DevCentral community.

Hosts: George Watkins, Jason Rahm, Joe Pruitt, and Scott Koon.
In this weeks podcast we discussed Authenticating At signs (@), Thoughts on Hostnames, Blackberry and Smartphones in the corporate environment, The benefits of desktop virtualization, and finished it up with a preview of Today’s Gestalt IT Tech Field Day at F5.

All of the links from this recording can be found with the dcpodcast140 tag at Delicious.com. You can also watch the video recording from our UStream.TV channel: DevCentral Podcast 140.

Get the Flash Player to see this player.

Technorati Tags: F5,DevCentral,Authentication,Hostname,Blackberry,Smartphone,Virtualization,TechFieldDay,George Watkins,Jason Rahm,Joe Pruitt,Scott Koon

Booth Babes Don’t Wear Glasses

Lori MacVittie — Wed, 14 Jul 2010 19:16:18 GMT

Gidget or gadgets, that’s not really what the debate is all about anyway

An axiomatic truth of technology today is that women in technology are, few and far between. The recent debate over booth babes slides naturally into the question “how can we encourage young women to enter science, technology, engineering and mathematics (STEM) fields”? After all, the argument goes, what young woman would willingly enter a field where she’ll be assumed to be ignorant unless she proves otherwise? Where she’ll be admired not for her Masters degree but for her mastery of makeup?

STEM, and technology in particular, has an unclear career path that tends to put off young women and, unsurprisingly, young men, too. Young women, according to research, aren’t thinking about the difficulties that exist being a woman in a traditionally male-dominated field – and they do exist, there’s no denying that. They’re thinking “what the heck would I do with a degree in [insert STEM field here]?” [Scientific American, “U.S. Students say “yah for science””] Young women aren’t avoiding STEM because technology vendors at technology shows are hiring “booth babes.” In fact looking closer at the debate that’s risen regarding booth babes it really has nothing to do with them and everything to do with men and attitudes.

It isn’t the existence of booth babes that causes the treatment cited by Denise Dubie in a recent post on this very debate over the practice, that is the result of a field that’s (1) still young compared to other STEM fields, (2) dominated by men, and (3) the fact that women in general haven’t been out en force for all that long. It’s been less than 100 years since the right of women to vote was recognized, and less than 40 since we’ve really been accepted into the work force. Most of us are first and maybe second generation “working women”, professional women. We’re still paving the path for our daughters and their daughters. It’s no surprise that men in general, then, haven’t had a whole lot of time yet to adjust.

A recent post by Network World blogger Michael Morris caused a bit of discussion on that site as well as chatter across social media network Twitter. He held a contest immediately following Cisco Live to identify the hottest booth babe at the show.

[…]

The one thing I do note at these shows, being in a position required to approach and speak to attendees and being female, is that I am in the definitely in the minority there. And often when I approach male attendees, they seem a bit shocked that I am talking technology with them. And when sitting in on sessions and looking around the room, I am always elated to find another female in attendance. But I have found that some of those women work for the vendor doing the presentation and aren’t IT professionals attending the show. Still I have to admit in the 10+ years of going to shows, the number of women in attendance has increased, based on my informal opinion. [emphasis added]

The ‘booth babe’ debate

Denise Dubie, NetworkWorld

It would likely be difficult to find a woman in technology that hasn’t had this same experience. The shock, the surprise, the change in tone and demeanor that comes from male counterparts upon realizing that the woman they’re talking to knows what she’s talking about. In some cases, she knows a heck of a lot more than they do. But that “shock and awe” isn’t restricted to trade shows, and it’s really got nothing to do with booth babes. It’s the preconceived notions many men carry along with them to trade shows. Notice that these opinions didn’t originate at a trade show replete with lusty booth babes, they were carried along. That shock and awe isn’t peculiar to trade show environments, it is everywhere. On the phone, on a webinar, in a lab. In environments where there are no booth babes. Hence, they are not really part of the “problem” at all.

SWAG versus SKIRTS

Go ahead, take the booth babe out of the picture. The assumptions and pre-conceived notions held by many in IT will still be there. The front line in any booth is assumed to be staffed by both male and female “talent” or employees that are not necessarily very technical. Period. Whether they’re male or female, that “screening line” isn’t going to be able to describe in detail, for example, the anatomy of an SSL handshake. Go ahead, ask. Ask a man, ask a woman. Doesn’t matter. Front line booth staff are almost unilaterally there to screen and direct. They exist to welcome you to the booth, scan your badge, and to help you get to the right person if you’re interested in a discussion. They are there because they are outgoing and friendly and enjoy engaging with people, for which those of us who are hiding “in the booth” and are not outgoing are grateful.

Now many folks point out that using booth babes generates a lot of useless leads. Mostly true. But the same can be said for interesting swag, as well. Folks who visit a booth to snatch up the latest geegaw or enter a drawing to win an iPad aren’t exactly solid leads in the first place, so whether they’re attracted by a nice set of legs or a shiny toy isn’t really all that relevant now, is it? It’s not as if the guy who stopped by for a shiny, sparkly ball for his kids is any more interested in your product than the one who was attracted by a beautiful set of … legs.

It is more than expected that women in technology want the respect they deserve based on their knowledge and expertise, not their choice in clothing or shoes. But how do you know who they are? And for that matter, how do you which men are the ones that have the knowledge you’re looking for? You can’t assume (or shouldn’t) that any given man at a technology show is any more technically astute than another because there are a variety of folks who attend those shows: from the IT manager who may not have 1337 technical skillz to the router jockey that does. Despite the attempts of Hollywood to the contrary, you can’t tell who’s is or isn’t a geek by looking at them. You simply can’t judge the competency and knowledge of an IT worker from their outward appearance or their title or even their degree.

On the other hand, you can be relatively sure that a booth babe is just that: a booth babe. Their proximity to the booth and uniform of short skirts, tight tops, and heavily glossed lips is hard to miss. I am not likely to be mistaken for a booth babe and conversely they are unlikely to be mistaken for someone with a clue. I will, however, almost certainly run into someone who is very surprised that I know a thing or two. Why? Because I’m a woman. In technology. And as much as I’d like to at times, I don’t carry a 2x4 wrapped in copies of my master’s degree in computer science, so there’s no obvious clue other than the fact that I’m not dressed like a booth babe.

The debate over “booth babes” isn’t really about booth babes. It’s not even really about the inability to distinguish the geeks from the rest of the crowd. It’s about the reaction of men to a technically competent woman, no matter what she’s wearing. It’s the surprise and the shock in their expression and voice. It’s about the assumption that no woman is technically competent – at a trade show or on a conference call. That’s at the heart of this debate and others like it, and whether booth babes are present or not is unlikely to have an impact on those assumptions.

NOTE: It’s been mentioned by “those in the know” that the event that started this whole brouhaha, CiscoLive, will be “booth babe” free in the future.

Related Posts

from tag women

Computer Engineer Barbie: We Need Details not Dolls

Confessions of a tech blogging grandmother

I've Concluded That You Guys Don't Think I'm an Idiot - A Reflection on the Sexes and the Field : Girl Developer

from tag technology

64 Things Every Geek Should Know - LaptopLogic.com

from tag F5

The Virtual Public-Private Cloud Connection

Madness? THIS. IS. SOA!

That Whole Concept is Broken

Following Google’s Lead on Security? Don’t Forget to Encrypt Cookies

Is PaaS Just Outsourced Application Server Platforms?

I CAN HAS DEFINISHUN of SoftADC and vADC?

Magic Virtualization-Fairy Dust and the New Network

Despite Good Intentions PaaS Interoperability Still Only Skin Deep

(more..)



Technorati Tags: MacVittie,F5,women,technology,booth babes,Denise Dubie,conferences

Risk Averse or Cutting Edge? Both at Once.

Don MacVittie — Wed, 14 Jul 2010 04:03:54 GMT

A couple of articles that have come out in the last week or so got me to pondering the current state of IT Management mentality, and it is somewhat amusing, if you think it through. We’re risk-averse where all evidence indicates we don’t need to be, and we’re cutting edge, risk aware but not averse, where all evidence says we should be treading with caution. The curious thing to me is that all of this is happening at once, and we’re okay with it.

The first article that brought this to my attention was published on The Register about Enterprise Storage, and talks about an enterprise storage guy who is discussing how his organization will only use a few top- of-the-line storage vendors, and how up-and-coming solutions have almost no chance of getting in the door of his enterprise.

The second article comes compliments of Network Computing and discusses a failure of a cloud provider, and is another IT person talking about how their cloud-based provider failed them and failed to resolve their issues in a timely manner.

EASILY AMUSED

Am I the only one amused by these two articles, taken in tandem? I mean storage is commodity stuff these days. I’ve written before about the joys of RAID and the highly capable software add-ons available in tier two storage vendors. The industry has been around long enough that true differentiators for the vast majority of storage environments are service and support. These days the new features that high-end, tier one vendors offer are all aimed at the high end of the storage spectrum. Most enterprises don’t need flash drives for a long-term cache, for example.

The cloud? It is immature, ill-defined (as a market), poorly documented, and lacks standards. What vendor you choose, or even if you jump in at all, is of tremendous importance to the success of your IT endeavors and possibly to the success of your enterprise.

We’re at an odd point in the technological life-cycle where the hype factor for cloud is driving some to try things that when looked at through the prism of realistic capabilities, it just isn’t ready for, while the storage industry has long been one where enterprises are told they might lose critical data if they use anything but the most feature packed, expensive vendors…

WHAT COLOR IS YOUR RISK ASSESSMENT?

But in the end you have the organization to look out for. Put the inexpensive disk in and give it a test run, right next to the cloud solution. Seriously. If you’re willing to test-drive cloud solutions, then you should be willing to test drive less expensive storage solutions too. If you’re not doing either, then at least you are consistent.

The one caveat I would offer in the storage space is to make sure the functionality that you do need is in your chosen tier-two vendor’s offerings. If you rely upon replication for DR, buying storage that doesn’t include replication is kind of self-defeating. But given the state of the storage market, I’ll speculate that most enterprises are well-served by the functionality in less expensive solutions.

And check performance of course. Put high-performance gear in where it is needed, and not where it isn’t. Some enterprises need the incremental speed boosts of higher-end storage arrays, but some don’t, and in some cases the lower-cost solution performs as well or better than the higher-end gear.

NOPE. NOT ABOUT US.

Now some of you will immediately jump to the conclusion that this could be generalized to include F5’s markets. I say no, because F5 has provided VM solutions for dev, test, and low-volume implementations and physical solutions for your other needs. Our other products are cutting-edge high-quality (like WOM and EDGE Gateway), or technically tier two storage (ARX and Data Manager). So I really am only talking about the storage market, where even the “secondary storage” arrays of top-line vendors are more expensive than the “primary storage” lines of other quality vendors.

For the cloud? Well, pick a vendor and try them out. Don’t go crazy and move your entire infrastructure, stay loosely coupled, determine what they can and can’t do, and like every other tool in your toolbox, use them for their strengths. Or find a new vendor if your experiences are like the ones in the Network Computing article.

In other words, become more risk-averse on the cutting edge technology, and less on the stuff that’s been around since the 80s.

Technorati Tags: Cloud,Storage,F5 Networks,Don MacVittie



Related Articles and Blogs

Despite a Claim of five nines, a 48 hour outage blew the curve

The Enterprise Storage Buyer: Why I Stick With the Big Guys

Startup with all-star backers aims to disrupt the storage market (Start-up? I covered them more than three years ago…)

Isilon Scale-out Storage Selected by NAVTEQ

Cloud Computing Journal on Sys-Con

CloudFucius Tunes into Radio KCloud

Pete Silva — Wed, 14 Jul 2010 02:59:00 GMT

Set the dial and rip it off – all the hits from the 70s, 80s, 90s and beyond – you’re listening to the K-Cloud. We got The Puffy & Fluffy Show to get you going in the morning, Cumulous takes you through midday with lunchtime legion, Mist and Haze get you home with 5 o’clock funnies and drive-time traffic while Vapor billows overnight for all you insomniacs. K-Cloud; Radio Everywhere.

I came across this article which discusses Radio’s analogue to digital transition and it’s slow but eventual move to cloud computing. How ‘Embracing cloud computing requires a complete rethinking of the design, operation and planning of a station’s data center.’ Industries like utilities, technology, insurance, government and others are already using the cloud while the broadcast community is just starting it’s exploration, according to Tom Vernon, a long-time contributor to Radio World.

Like many of you, I grew up listening to the radio (music, I’m not that old) and still have a bunch of hole-punched record albums for being the 94th caller. I listened to WHJY (94-HJY) in Providence and still remember the day in 1981 when it switched from JOY, a soft, classical station to Album-Orientated Rock. Yes, I loved the hair-metal, arena rock, new wave, pop and most what they now call classic rock. It’s weird remembering ‘Emotional Rescue’ and ‘Love Rollercoaster’ playing on the radio as Top 40 hits and now they are considered ‘classics.’   Um, what am I then?!?

That article prompted me to explore the industries that have not embraced the cloud, and why. Risk adverse industries immediately come to mind, like financial and health care. There have been somewhat contradictory stories and surveys recently indicating both that, they are hesitant to adopt the cloud and ready to embrace the cloud. A survey by LogLogic says that 60% of the financial services sector felt that cloud computing was not a priority or they were risk-averse to cloud computing. This is generally an industry that historically has been an early adopter of new technologies. The survey indicates that they will be spending IT dollars on ‘essential’ needs and that security questions and data governance concerns is what’s holding them back from cloud adoption.

About a week later, results from a survey done by The Securities Industry and Financial Markets Association (SIFMA) and IBM reports that there is now a strong interest in cloud computing after a couple years of reluctance. The delay was due to the cost of implementing new technologies and the lack of talent needed to mange those systems. Security is not the barrier that it once was since their cloud strategies include security ramifications. They better understand the security risks and calculate that into their deployment models. This InformationWeek.com story says that the financial services industry is indeed interested in cloud computing, as long as it’s a Private Cloud. The one’s behind the corporate firewall, not Public floaters. And that security was not the real issue, regulations and compliance with international border laws were the real holdback.

In the healthcare sector, according to yet another survey, Accenture says that 73% said they are planning cloud movements while nearly one-third already have deployed cloud environments. This story also says that ‘healthcare firms are beginning to realize that cloud providers actually may offer more robust security than is available in-house.’

Is there a contradiction? Maybe. More, I think it shows natural human behavior and progression when facing fears. If you don’t understand something and there is a significant risk involved we’ll generally say, ‘no thanks’ to preserve our safety and security. As the dilemma is better understood and some of the fears are either addressed or accounted for, the threat level is reduced and progress can be made. This time around, while there are still concerns, we are more likely to give it a try since we know what to expect. A risk assessment exercise gives us the tools to manage the fears. Maybe the threat is high but the potential of it occurring is low or the risk is medium but we now know how to handle it. It’s almost like jumping out of a plane. If you’ve never done it, that first 3000ft tethered leap can be freighting – jumping at that height, hoping a huge piece of fabric will hold and glide you to a safe landing on the ground. But once you’ve been through training, practiced it a few times, understand how to deploy your backup ‘chute and realize the odds are in your favor, then it’s not so daunting. This may be what’s happening with risk averse industries and cloud computing. Initially, the concerns, lack of understanding, lack of visibility, lack of maturity, lack of control, lack of security mechanisms and their overall fear kept these entities away, even with the lure of flexibility and potential cost savings. Now that there is a better understanding of what types of security solutions a cloud provider can and cannot offer along with the knowledge of how to address specific security concerns, it’s not so scary any more.

Incidentally, I had initially used KCLD and WCLD for my cloud stations until I realized that they were already taken by real radio stations out of Minnesota and Mississippi.

And one from Confucius: Everything has its beauty but not everyone sees it.

ps

The CloudFucius Series: Intro, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12

Related:

Radio’s Future May Be in the Clouds

The Cloud Fear Factor in Financial Services

Financial services firms hesitant to adopt cloud computing

Financial Services Like The Cloud, Provided It's Private

Risk-Averse Verticals Embrace Cloud Computing

Healthcare Taking Computing To The Cloud

Wall Street Firms Set to Increase IT Spend through 2011 on Transformation Initiatives

The State of Cloud Computing Security

Risk is not a Synonym for “Lack of Security”

Technorati Tags: F5, infrastructure 2.0, integration, cloud connect, Pete Silva, security, business, education, technology, application delivery, cloud, context-aware, web, internet

twitter: @psilvas

A Taste of Blackberries

Jason Rahm — Wed, 14 Jul 2010 02:36:46 GMT

No, not the kiddie lit favorite by Doris Buchanan Smith, I mean the smart phones. I was not a member of the smart phone club when I started at F5. In fact, my first week on the job was at our international sales conference and I remember watching Jeff and Joe scroll, click, and type like the wind and Jeff leaned over and said, “They’re addicting, you’ll see.” I got my first Blackberry, the Curve 8310, a week later. I liked having all the keys for typing, as the limited texts I’d written with the basic phones was a chore, but the buttons were pretty small for my stubby fingers, and given that I h
ave the dexterity of one wearing mittens, well, let’s just say it took some getting used to. I wasn’t immediately impressed overall, and couldn’t for the life of my figure out whey this was so appealing. However, after figuring out the email usage and installing google maps, facebook, and twitter, I was well on my way to becoming an addict.
Two things drove my nuts about my 8310. One was the camera. The pinhole for the sensor was not covered at level with the phone with glass, so it was near impossible to keep the lens clean. The second was the trackball. It was constantly getting stuck, and after a few falls, would occasionally pop out. This came to a head at Tech Ed last month when it fell out at night, in a gravel driveway. I crawled around in the hot sticky mess that is the South in the summer, but couldn’t find it. So I learned a few of the trackball-less navigation techniques that allowed me to at least make phone calls and read emails. This problem led to a short experience with the Curve 8320. Not really an evolution here, as both models I assume were released concurrently, but I found it odd that in this line at least, you had to choose between GPS and wi-fi. I was a little shocked while out driving and needing to find a place I hadn’t been that google maps couldn’t zero in on my location, only to find I had no GPS receiver. The wi-fi was nice, though, as it’s a little quicker while at home than using the 3G network. I’m sure it was a buggy unit, but the 8320 froze constantly and often hard shutdown, requiring about 10 minutes to return to normal. Not good while on phone calls.

Fast forward to the Bold 9700. I received this bad boy yesterday after waving the white flag on figuring out the 8320 instability issues. While I’m not crazy about the change in the feel of the buttons, I’m excited to see the trackpad instead of the trackball. Also, the camera lens is covered at-level with glass. Woo Hoo! Both issues addressed. I guess I’m easy, right? Another nice feature is the grip pad on the back of the phone that prevents sliding (and fidget-induced phone spinning). I’m still getting used the interface, but overall, I’m impressed with the advances. I’m a little annoying by all the preview software that I can’t use, but that’s not just a Blackberry problem.

Technorati Tags: F5 DevCentral,Blackberry,Blackberry Curve,Blackberry Bold,Jason Rahm

Out, Damn’d Bot! Out, I Say!

Lori MacVittie — Tue, 13 Jul 2010 18:53:04 GMT

Exorcising your digital demons

Most people are familiar with Shakespeare’s The Tragedy of Macbeth. Of particularly common usage is the famous line uttered repeatedly by Lady Macbeth, “Out, damn’d spot! Out, I say” as she tries to wash imaginary bloodstains from her hands, wracked with the guilt of the many murders of innocent men, women, and children she and her husband have committed.

It might be no surprise to find a similar situation in the datacenter, late at night. With the background of humming servers and cozily blinking lights shedding a soft glow upon the floor, you might hear some of your infosecurity staff roaming the racks and crying out “Out, damn’d bot! Out I say!” as they try to exorcise digital demons from their applications and infrastructure.

Because once those bots get in, they tend to take up a permanent residence. Getting rid of them is harder than you’d think because like Lady Macbeth’s imaginary bloodstains, they just keep coming back – until you address the source.

A RECURRING NIGHTMARE

One of the ways in which a bot can end up in your datacenter wreaking havoc and driving your infosec and ops teams insane is through web application vulnerabilities. These vulnerabilities in both the underlying language and server software as well as the web application itself, are generally exploited through XSS (Cross-site scripting). While we tend to associate XSS with attempts to corrupt a data source and subsequently use it as distribution channel for malware, a second use of XSS is to enable the means by which a bot can be loaded onto an internal resource. From there the bot can spread, perform DoS attacks on the local network, be used as a SPAM bot, or join in a larger bot network as part of a DDoS.

The uses of such deposited bots is myriad and always malevolent.

Getting rid of one is easy. It’s keeping it gone that’s the problem. If it entered via a web application it is imperative that the vulnerability be found and patched. And it can’t wait for days, because the bot that likely exploited that vulnerability and managed to deploy the bot inside your network is probably coming back. In a few hours. You can remove the one that’s there now, but unless the hole is closed, it’ll be back – sooner rather than later.

According to an ARS Technica report, you may already know this pain as “almost all Fortune 500 companies show Zeus botnet activity”:

Up to 88% of Fortune 500 companies may have been affected by the Zeus trojan, according to research by RSA's FraudAction Anti-Trojan division, part of EMC.

The trojan installs keystroke loggers to steal login credentials to banking, social networking, and e-mail accounts.

This is one of those cases in which when is more important than where the vulnerability is patched initially. Yes, you want to close the hole in the application, but the reality is that it takes far longer to accomplish that then it will take for attackers to redeposit their bot. According to WhiteHat Security’s Fall 2009 Website Security Statistics Report a website is 66 percent likely to be vulnerable to an XSS exploit, and it will take an average of 67 days for that vulnerability to be patched.

That’s 67 days in which the ops and infosec guys will be battling with the bot left behind – cleaning it up and waiting for it to show up again. Washing their hands, repeatedly, day and night, in the hopes that the stain will go away.

WHAT CAN YOU DO?

The best answer to keeping staff sane is to employ a security solution that’s just a bit more agile than the patching process. There are several options, after all, that can be implemented nearly immediately to plug the hole and prevent the re-infection of the datacenter while the development team is implementing a more permanent solution.

Virtual patching provides a nearly automated means of plugging vulnerabilities by combining vulnerability assessment services with a web application firewall to virtually patch, in the network, a vulnerability while providing information necessary to development teams to resolve the issue permanently.

Network-side scripting can provide the means by which a vulnerability can be almost immediately addressed manually. This option provides a platform on which vulnerabilities that may be newly discovered and have not yet been identified by vulnerability assessment solutions can be addressed.

Web application firewall can be manually instructed to discover and implement policies to prevent exploitation of vulnerabilities. Such policies generally target specific pages and parameters and allow operations to target protection at specific points of potential exploitation.

All three options can be used together or individually. All three options can be used as permanent solutions or temporary stop-gap measures. All three options are just that, options, that make it possible to address vulnerabilities immediately rather than forcing the organization to continually battle the results of a discovered vulnerability until developers can address it themselves.

More important than how is when, and as important as when is that the organization have in place a strategy that includes a tactical solution for protection of exploited vulnerabilities between discovery and resolution. It’s not enough to have a strategy that says “we find vulnerability. we fix. ugh.” That’s a prehistoric attitude that is inflexible and inherently dangerous given the rapid evolution of botnets over the past decade and will definitely not be acceptable in the next one.

John Pescatore, vice president and distinguished analyst, speaking about the enterprise threat landscape Tuesday at the Gartner Security and

Risk Management Summit, said rapid changes are taking place in the way IT organizations deliver services, largely via virtualization and cloud computing , and the way businesses consume them, which create infrastructure breakage points ripe for exploit.

"Ninety percent of attacks are exploiting vulnerabilities we already knew about, by missing patches, deciding not to patch, or uses of technology in which we made the decision to deploy without putting security controls on it," Pescatore said. "Less than 1% are zero-day attacks; the other 99% are exploited configurations and unpatched machines that the simplest vulnerability scan would've found."

And since attackers have had so much success using bots to take advantage of these vulnerabilities, Pescatore said that trend is likely to continue through 2012, when the ongoing cat-and-mouse game between attackers and enterprise defenders leads to the next threat delivery breakthrough.

With IT and security resources always at a premium, Pescatore said enterprises must prioritize their defensive efforts simply by asking themselves what an attacker would want to get from them; in most cases it's personally identifiable information (PII) in the form of credit card and Social Security numbers, or sensitive intellectual property. [emphasis added]

Gartner: Enterprises must learn to detect botnet threats

PATCH FIRST, ASK QUESTIONS LATER

It’s going to get harder and harder to exorcise those digital demons. That means the best strategy is to be proactive; to ensure the proper protections are in place before you go live. To provide developers with the training and tools to ensure that known vulnerabilities are addressed, and that the infrastructure or application is sanitizing inbound data and scrubbing outbound data.

Protecting the organization is no longer about which group or which solution should or shouldn’t be responsible for web application security. When web application vulnerabilities took legs and began to be a problem for the entire datacenter and the business, it became more about enabling solutions and less about who or how or what.

Patch the vulnerability first, in whatever way can make that happen immediately. Then you can discuss and debate about what’s the proper long-term solution for your organization. After all, no one would suggest that the little Dutch boy who stuck his finger in the dike should stand there forever, holding back the flood of water. But no one would suggest he was wrong to do it, either.

Related Posts

Network Security Does Not Imply Application or Database Security

Are You Scrubbing the Twitter Stream on Your Web Site?

Turning the Pushdo Bot Into the Push-oh-no-you-don’t Bot

The Application Delivery Spell Book: Detect Invisible (Application) Stalkers

from tag vulnerability

When Is More Important Than Where in Web Application Security

I Can Has UR .htaccess File

An Unhackable Server is Still Vulnerable

Using Resource Obfuscation to Reduce Risk of Mass SQL Injection

Infosecwriters.com

New TCP vulnerability about trust, not technology

from tag security

Compliance in the Cloud

If Your Users See an HTTP Error Code You're Doing It Wrong

The Real Meaning of Cloud Security Revealed

What's the difference between a web application and a blog?

WILS: How can a load balancer keep a single server site available?

Robert Half Technology :: Top 10 Tech Investments: Information Security Leads List, CIO Survey Shows

(more..)



Technorati Tags: MacVittie,F5,web application security,web application firewall,network-side scripting,vulnerabilities,virtual patching,WhiteHat Security,Gartner,Arstechnica,zeus bot

Thoughts On Hostname Nomenclature

George Watkins — Mon, 12 Jul 2010 08:24:00 GMT

There are few topics as controversial in the world of systems administration as hostname nomenclature. I'm not talking about the merits of using Disney characters versus the names of stars. I'm discussing how to structure a host's name in such a way that it requires a minimal amount of effort to decipher its purpose. I am going to present a naming convention that has worked very well for me in the past in both Windows and *nix environments. This nomenclature should reflect four key pieces of information about our host: location, environment, purpose, and a unique identifier. This hostname will be 15 characters in length with each field consisting of 3 letters or numbers separated by hyphens: loc-env-pur-001.

This is not just an issue for the systems administrator, this is also a matter of department policy. It is easy to follow a naming convention when there are one or two administrators in agreement, but when there are 50 in your organization it gets a little more tricky. Left to their own vices, the installers will adhere to whatever names they find convenient. If a hostname convention isn't actively reviewed by upper management and required to go through an approval process, it will just turn out to be a mess. There needn't be hours of meetings on this topic, but a basic thumbs up or down from a director can make everyone's life easier in the long run.
Location

A hostname should reflect the physical location of a piece of hardware. This element is often omitted altogether in smaller environments that have a single office. It may seem redundant to include this if you only operate one location, but will be a pain to go back and correct when you open another office. Do yourself a favor by adding this from the beginning. You'll be a lot happier when your manager brings you the good news about the new office you are opening in London.

As a rule of thumb, I use the nearest airport's three letter code. This is easy to stick to and minimizes arguing as to how some localities should be abbreviated. Seattle-Tacoma International has a three letter abbreviation of 'SEA', so the first three letters of a machine name located in Seattle would be 'sea'. If we opened another datacenter in Portland, those hostnames would start with 'pdx'.
Environment

We use the environment field to indicate whether this database server is the MySQL server in corporate, development, test, etc. By placing the environment in the hostname we can reduce mistakes where we "thought we were doing that in test." We've all done it and it can turn a normal morning into a firestorm.

Some of the more common environments I have encountered in my tenure have been: corporate, production (also seen it called delivery), development, test, stage, stress, and operations. With that in mind, all of my production hardware in Portland will start with 'pdx-prd'. As the name starts to take shape, you should notice how easy it is to grep a zone file and give your boss a list of all the test machines you have at the satellite development office in Dublin.
Purpose

Whether the machine will be used to host a web server, an e-mail relay, or a corporate database, it needs to be displayed in the hostname. I want to know what the machine is doing when I look at my Remote Desktop toolbar or my *nix prompt. I don't want to have to look at the process list, then search through the service's configuration to figure out what it is doing.

The list of possible purposes or services is too long to list here, but we can take a common service to demonstrate my point. We are getting ready to roll out some web servers at our new production datacenter in Boston. Our closest airport in Logan International and Director Tom says we're using 'prd' and 'web' for production web servers. Our new web servers are all going to start with the same prefix, 'bos-prd-web'. Done deal, there's not too much to argue about here.
Unique identifier

This is self-explanatory, but we need a way to differentiate all of our web servers in Boston. I do this by using a base 10 integer padded with zeros. This works well unless we have more than 999 of a particular host type in one environment at a single location. The easy fix for this would be to substitute a hexadecimal integer in place of the decimal integer. This will allow you to have up to 4095 machines in that particular server class.

A My 40th development SQL server at our main campus in Seattle will be named 'sea-dev-sql-040'.
The pressing questions

So now comes the question: "but we have all that information already broken out by subdomain, why would I put it in the hostname?" The default number of searchable domains on a Linux host is 6. This can be changed to a different value in resolv.h, but is an unacceptable expectation for the casual user. On the flip side, as an administrator I don't want to spend all day typing the fully qualified domain names of my servers. Searching one or a couple domains makes life a lot easier.

As for the 15 character maximum length, this is a limitation of NetBIOS. NetBios usage ceased with the releases of Vista, Server 2008, and all subsequent releases of Windows. While this is becoming less of a limitation with newer operating systems, I still like to provide backward compatibility for those users who may not be using the most current version.
Conclusion

This is by no means the definitive guide to hostname nomenclature. This is merely my personal opinion. There are few things that irritate me more than walking into a new position where someone has named everything from switches to web clusters after Star Wars characters. Don't get me wrong, I like Star Wars as much as the next guy, but please don't name a database server 'Chewie'.

I hope you found this informative and will take some of it to heart while architecting your infrastructure. Future systems administrators will thank you.

Use the Force Luke. (Zzzaap)

Don MacVittie — Mon, 12 Jul 2010 04:50:58 GMT

In Star Wars Episode IV: A New Hope (the first Star Wars movie for those more curmudgeonly than I), Luke is training with a light saber…

Ben Kenobi: Remember, a Jedi can feel the Force flowing through him.
Luke Skywalker: You mean it controls your actions?
Ben Kenobi: Partially, but it also obeys your commands.

VIRTUALIZATION AND THE FORCE

Virtualization is much like that – it limits your options, in effect controlling your actions, while also helping you to do your job more effectively. The problem is that, like the dark side, the temptation is always there to think of virtualization as the answer to every problem. It’s insidious, and at this point in time, it’s telling. It always intrigues me to have discussions about virtualization, because you can quickly tell the other person’s level of involvement by the approach they take to the topic. “Everything Virtual” is the hew and cry of those who aren’t responsible for the systems in question, while cautious optimism is the approach of those who have to live with – and work on – the systems that would be virtualized.

I was recently on a call with another vendor, and they said they did not offer a virtualized solution because they wanted to be able to back up any performance claims they made without the added complexity of having to state hardware requirements to meet their performance numbers, and needed the ability to guarantee a certain level of throughput, something they just plain could not do if they had no control over the hardware that their system was being run on. Indeed, they couldn’t even control how many VMs were sharing the hardware they were given, or how many shared the NIC that their network device required to do its job. I pointed out that our motivation was first to get a copy of an ADC into the hands of developers, and second to meet the needs of customers with less throughput than our boxes are typically rated for. Their response was cool detachment that said louder than if they’d shouted it “yes, but that’s not the only way it will be used”.

DIFFERENT REASONS, DIFFERENT USES

Which is true, but only superficially. Most enterprises want a supported configuration that they can call someone and get answers straight-away, so, for example, even if virtual Application Delivery Controller (vADC) throughput grows beyond what their existing product can handle, they are highly likely to put another vADC in rather than have to tell someone on the other end of the phone that they’re running four gigabits through a single 1 Gig NIC and can’t figure out what’s wrong with performance.

But the pressure will be there to virtualize your entire infrastructure, which is, at this point in time, not feasible. That pressure is okay, those are people who want the best for the organization, just keep reality in the spotlight and all will go well. We’ve come a long way, but server virtualization is still ongoing in many organizations and VDI is a distant possibility in most. So size your other infrastructure to what you need, and don’t get too wrapped up in the virtual/not virtual conversation beyond “can we service these needs reasonably with a virtual X”? If yes, then it’s an option, if not then physical is your solution.

DON’T LET IT BECOME A RELIGIOUS DISCUSSION

Like so many things in high-tech – cell phones, operating systems, programming languages, DBMS’s all spring to mind - It’s much less about the rhetoric and much more about what solves your problem the best. Putting in a virtual infrastructure device that is going to generate calls for you in the middle of the night because it is overloaded is not the best answer to any problem, and all of the hardware vendors I have spoken with are still doing hardware because they see a benefit for their customers – the vendor I was talking to because they want to offer customers guarantees about performance they can’t make in a virtualized environment, we have custom hardware that does improve performance, though sometimes that hardware boost isn’t necessary. That gives you the option of the best tool for the job. So take your pick, blaster or The Force, they’re both available, depending upon what you need to do.

But don’t dismiss a good solution just because it is hardware or software only – pin the vendor down, make them justify their choice, and insist on usable performance numbers. After all, it’s not the form of the tool but how well the tool does the job that matters. And whatever you do, don’t go into the conversation in a helmet with the blast shield down… Unless your name is Luke. Who, I might add, used both light sabers and blasters… Right tool for the job and all.

Technorati Tags: Virtualization,Virtual Infrastructure,Star Wars,vADC,F5 Networks,Don MacVittie



Related Articles and Blogs

WikiQuote – Star Wars Episode IV

A Million VMs. Test Environments in Virtual and Cloud Infrastructures

100% Virtual? Sure, Run That Test

Dynamic Infrastructure: Are We Over That Hump?

Does This Application Make My Browser Look Fat?

Lori MacVittie — Sun, 11 Jul 2010 19:02:46 GMT

Web applications that count on the advantage of not having a bloated desktop footprint need to keep one eye on the scale…

A recent article on CloudAve that brought back the “browser versus native app” debate caught my eye last week. After reading it, the author is really focusing on that piece of the debate which dismisses SaaS and browser-based applications in general based on the disparity in functionality between them and their “bloated desktop” cousins.

Why do I have to spend money on powerful devices when I can get an experience almost similar to what I get from native apps? More importantly, the rate of innovation on browser based apps is much higher than what we see in the traditional desktop software.

[…]

Yes, today's SaaS applications still can't do everything a desktop application can do for us. But there is a higher rate of innovation on the SaaS side and it is just a matter of time before they catch up with desktop applications on the user experience angle.

When You Can Innovate With browser, Why Do You Need Native Apps?, Krishnan Subramanian

I don’t disagree with this assessment and Krishnan’s article is a good one – a reminder that when you move from one paradigm to another it takes time to “catch up”. This is true with cloud computing in general. We’re only at the early stages of maturity, after all, so comparing the “infrastructure services” available from today’s cloud computing implementations with well-established datacenters is a bit unfair.

But while I don’t disagree with Krishnan, his discussion reminded me that there’s another piece of this debate that needs to be examined – especially in light of the impact on the entire application delivery infrastructure as browser (and browser-based applications) capabilities to reproduce a desktop experience mature. At what point do we stop looking at browser-based applications as “thin” clients and start viewing them for what they must certainly become to support the kind of user-experience we want from cloud and web applications: bloated desktop applications.

The core fallacy here is that SaaS (or any other “cloud” or web application) is not a desktop application. It is. Make no mistake. The technology that makes that interface interactive and integrated is almost all enabled on the desktop, via the use of a whole lot of client-side scripting. Just because it’s loaded and executing from within a browser doesn’t mean it isn’t executing on the desktop. It is. It’s using your desktop’s compute resources and your desktop’s network connection and it’s a whole lot more bloated than anything Tim Berners-Lee could have envisioned back in the early days of HTML and the “World Wide Web.” In fact, I’d argue that with the move to Web 2.0 and a heavy reliance on client-side scripting to implement what is presentation-layer logic that the term “web application” became a misnomer. Previously, when the interface and functionality relied solely on HTML and was assembled completely on the web-side of the equation, these were correctly called “web” applications. Today? Today they’re very nearly a perfected client-server, three-tiered architectural implementation. Presentation layer on the client, application and data on the server. That the network is the Internet instead of the LAN changes nothing; it simply introduces additional challenges into the delivery chain.

THE MANAGEMENT and MAINTENTANCE DISTINCTION is NO MORE

But Lori! Desktop applications require installation! They require maintenance, and support! They are more costly! They are evil! An ancient horror that we have worked hard to eradicate from the face of laptops and desktops everywhere!

Let’s get a grip on reality, shall we?

As more and more of that presentation layer logic moves into the client – the browser, the desktop – those same issues are going to arise. Perhaps not on the same scale, but they will arise. When you start relying on libraries – frameworks like Prototype and XAJAX and the like – to provide the underlying framework through which all that interactivity and cool functionality is achieved, you are basically installing software. It’s just as hidden from the end-user as much of the desktop software, at least in an enterprise setting. It still requires maintenance and support and oh how problematic is the support! When an AJAX call goes “bad” it’s not uncommon at all for a web application to fail to provide the means by which the end-user is notified or that updating can be “restarted”. At some point that’s going to generate a support call to someone, and troubleshooting a browser-based application has got to be the most miserable experience on the face of the earth. You don’t have – can’t have – an exact duplicate of the user’s environment because, well, you may not have the same plug-ins installed, or the same version of the plug-ins. You may not be accessing the application over the same network with the same underlying infrastructure that may or may not be causing issues. You may not even be running on the same operating system – or device. But you’re still going to have to troubleshoot it and figure out what’s causing the problem somehow.

Good luck with that.

I’m not saying the evolution of the browser into the modern-day desktop – and that is what is happening – is a bad thing. But I am pointing out that at some point the same issues that once made us look upon browser-based applications as a godsend will crop up and cause the same kind of headaches for IT and end-users and in some cases those headaches will be migraines that never existed when IT had control over the environment.

The days of SaaS and “web applications” claiming an almost moral superiority over “bloated desktop applications” are over. They went the way of “Web 1.0” – lost to memory and saved only in the “wayback machine” to be called forth and viewed with awe and disbelief by digital millennials that any one ever used applications so simple.

Today’s “web” applications are anything but simple, and the environment within which they execute is anything but thin.

Related Posts

The New Distribution of The 3-Tiered Architecture Changes Everything

from tag frameworks

The Great Client-Server Architecture Myth

from tag client-server

How Reality Blew the Cloud Away at Interop

from tag browser

Facepalm: Google Wave Choice of XMPP Not the Death of HTTP

Firefox Reaches 20% Market Share for First Time Ever - ReadWriteWeb

Firebug: The Firefox add-on everybody else says I can't live without

Automatically detecting client speed

IE8: Robbing Peter to pay Paul

Is the Mozilla FireFox 3 SSL policy bad for the web?

from tag development

Is your epic application being dragged down by green shoes?

Rip and Replace Won't Solve Twitter's (Or Your) Security Problems

Let Your ENUMs Do the Talking with CSS

All your control are belong to us

(more..)



Technorati Tags: MacVittie,F5,frameworks,development,client-server,browser,architecture,3-tier,web,SaaS,cloud computing

Benefits of Desktop Virtualization

Jason Rahm — Thu, 08 Jul 2010 20:56:54 GMT

There is an abundance of mature desktop virtualization solutions that are outright free or at least reasonable. From VMware’s Workstation (at cost after 30-day trial, but entirely worth it) to Oracle’s VirtualBox and Microsoft’s Virtual PC, you can get started in literally minutes. Why would you want to?

Trivial backups. Tired of losing a drive and having to restore first the OS, then the applications, and finally your files? Once everything is hosted on a virtual disk, keeping that backed up frequently means a physical disk failure costs you only the time to restore the hardware OS and your virtualization. Major time saver. For enterprise environments, this also makes hardware changes for employees a cinch. New desktop? A single file copy and a short vm configuration, and the IT folks can move on to something more important.

Snapshots prior to patching, new applications. Ever had an antivirus or OS patch destroy your system? Snapshot your virtual machine prior to making changes, and if anything goes wonky, simply revert to the last snapshot.

Minimal, purposeful systems for online banking (or remote vpn access, shopping, etc). Do you know the malware that threads its tentacles into your system over time from browsing the internet? Keep one virtual machine that has nothing installed except the browser and plugins necessary to do your banking and investing, and once you’re done, shutting down the virtual machine restores it to original state so any potential malware threats are eliminated.

For the geeks in the audience (all of you?), quick test setups with teaming (VMware Desktop for sure, check on the others). Don had a great article on this relating to a BIG-IP LTM VE test setup. This is great becuase you mock up complete test environments and spin them together, but only when you need them. With several linux-based appliances, you can get virtual switches, routers, firewalls, lamp servers, all purpose-built and ready to go for your test environments.

Run nothing on the host OS except the hypervisor and antivirus. This ensures a minimal rebuild time should there be a failure. My exceptions to this rule are audio/video tools like skype, jing, camtasia, and iTunes. Lest you think I’ll need a file rebuild for iTunes, that library is off box anyway.

There are far more benefits, but these are the ones that jump out at me. Assuming you’re on windows, you can convert your physical disk to virtual with disk2vhd, another great tool from the Sysinternals team. Note that if you plan to use Virtual PC, you’ll need to resize the drive to less than 127G with diskpart and then tell the drive it has shrunk as well with a tool like VHD Resizer. If you want to use VMware Workstation, you’ll need an additional step that converts the drive from vhd format to vmdk with a tool like WinImage.

Technorati Tags: F5 DevCentral,Virtualization,Desktop Virtualization,VMware Workstation,Virtualbox,VirtualPC,WinImage,disk2vhd,SysInternals,diskpart,VHD Resizer,Don MacVittie,Jason Rahm

F5 Friday: Would You Like Some Transaction Integrity with Your Automation?

Lori MacVittie — Thu, 08 Jul 2010 18:34:22 GMT

If you thought the integration and collaboration required new networking capabilities, you ain’t seen nothing yet.

Anyone who has ever configured a network anything or worked with any of a number of cloud provider’s API to configure “auto-scaling” via a load balancing service recognizes that it isn’t simply point, click, and configure. Certain steps need to be configured in a certain order (based entirely on the solution and completely non-standardized across the industry) and it’s always a pain to handle errors and exceptions because if you want to “do over” you have to backtrack through the completed steps or leave the system cluttered or worse – unstable.

Developers and system operators have long understood the importance of a “transaction” in databases and in systems where a series of commands (processed in a “batch”) are “all or nothing”. Concepts like two-phase commit and transaction integrity are nothing new to developers and sysops and probably not to network folks, either. It’s just that the latter has never had that kind of support and have thus had to engineer some, shall we say, innovative solutions to recreating this concept.

Infrastructure 2.0 and cloud computing are pushing to the fore the need for transactional integrity and idempotent configuration management. Automation, which is similar to the early concepts of “batch” processing, requires that a series of commands be executed that individually configure the many moving pieces and parts of an overarching architecture that are required in order to “make the cloud go” and provide the basic support necessary to enable auto-scaling.

Because it is possible that one command in a sequence of ten, twenty, or more commands that make up an “automation” could fail, you need to handle it. You can catch it and try again, of course, but if it’s a problem that isn’t easily handled you’d wind up wanting to “roll back” the entire transaction until the situation could be dealt with, perhaps even manually. One way to accomplish this is to enable the ability to package up a set commands as a transaction and, if any command fails the transaction management system automagically hits the “undo” button and rolls back each command to the very beginning, making it like it never happened.

IT’S A LONG, LONG WAY TO TIPPERARY…and FULLY AUTOMATED DATACENTERS

We’re not there yet. If you were waiting for me to pronounce “we have arrived” I’m sorry. We haven’t, but we are at least recognizing that this is a problem that needs solutions. In order for the above scenario to be reality every system, every device, every component that is (or could be) part of the transaction must be enabled with such functionality.

Transactions are essential to maintaining the integrity of components across an orchestration in complex systems. That basically means most – if not all – datacenters will need transactions eventually if they go the route of automation to achieve operational efficiency. This is another one of those “dev” meets “ops” topics in which dev has an advantage over ops merely due to the exigencies of being in development and working with integration and databases and other development-focused systems that require transactional support. Basically we’re going to (one day) end up with layers of transactions: transactions at the orchestration layer that is comprised of individual transactions at the component layer and it will be imperative that both layers of transactions are able to handle errors, be idempotent (to handle restarts), and to provide integrity of the implied shared systems that make up a cloud computing environment (that’s fault isolation, by the way).

The necessity of transactional support for infrastructure 2.0 components is going to become more important as larger and more business critical systems become “cloudified”. The new network has to be more dynamic and that means it must be able to adapt its operating conditions to meet the challenges associated with integration and the dependencies between devices and applications that creates.

For F5, we’ve taken a few of the first steps down the road to Tipperary (and the eventually fully transactional-based automation implementation required to get there) by enabling our BIG-IP shell scripting solution (TMSH) with some basic transactional support. It isn’t the full kitchen sink, it’s just a start, and as you can guess you should “watch this space” for updates and improvements and broader support for this concept across the entire BIG-IP management space.

TMSH TRANSACTION SUPPORT

If you weren’t aware, back in a prior release (we called it Park City, you might know it as v10) there was a capability added called TMSH (TMOS SHell) which is an action/object based scripting shell for F5 BIG-IP that allows administrators to script the configuration and management of a BIG-IP in much the same way as they might develop system scripts in BASH and KORN or PowerShell or whatever their favorite shell scripting language may be.

What was really cool was the addition of the ability to wrap transactions around a series of TMSH commands. What transactions allow you to do is bundle a list of commands to be executed as a single atomic operation. Now, we’re not anywhere near the “two phase commit” capabilities of mature batch processing systems like IBM and databases might have, but we are a far sight further along than perhaps people realized. When you’re “in” a transaction, for example, the command is only validated for syntax. It isn’t executed until the transaction is submitted for execution, you can’t undo a submitted transaction, and right now you can’t create “named” transactions that you can queue up for approval or save to execute at a later date or what have you. You can, however, recreate a transaction you have deleted by using the cli history component.

1: root@bigip(Active)(tmos)# create transaction

2: [batch mode]root@bigip(Active)(tmos)# create ltm pool a_pool members add { 10.2.27.1:80 }

3: Command successfully added to the current transaction

4: [batch mode]root@bigip(Active)(tmos)# create ltm pool a_pool members add { 10.2.16.1:80 }

5: Command successfully added to the current transaction

6: [batch mode]root@bigip(Active)(tmos)# submit transaction

7: 01020066:3: The requested pool (a_pool) already exists in partition Common

8: [batch mode]root@bigip(Active)(tmos)# list ltm pool a_pool

9: 01020036:3: The requested pool (a_pool) was not found.

10: [batch mode]root@bigip(Active)(tmos)# list transaction

11: 1: (tmos)# create ltm pool a_pool members add { 10.2.27.1:80 }

12: 2: (tmos)# create ltm pool a_pool members add { 10.2.16.1:80 }

13: [batch mode]root@bigip(Active)(tmos)# modify transaction delete 1

14: [batch mode]root@bigip(Active)(tmos)# submit transaction

15: root@bigip(Active)(tmos)# list ltm pool a_pool

16: pool a_pool {

17: members {

18: 10.2.16.1:http { }

19: }

20: }

This is only one (small) piece of a much larger puzzle: the automated, dynamic infrastructure (a la infrastructure 2.0). And it’s a piece that’s not even wholly complete yet. But it’s a start on what’s going to be necessary if we’re going to automate the data center and realize the potential gains in efficiency and move even further into the “automated self-healing, self-optimizing” networks of the (further out) future. We can’t get even to a true implementation of “cloud” until we can automate the basic processes needed to implement elastic scalability because otherwise we’d just be shifting man hours from racking servers to tapping out commands on routers and switches and load balancers.

But it’s a step forward on the road that every one is going to have travel.

Related Posts

All F5 Friday Entries on DevCentral

TMSH Wiki

from tag CLI

The API Is the New CLI

from tag orchestration

Cloud Today is Just Capacity On-Demand

Repetition is the Key to Network Automation Success

If a Cat has Catness Does a Cloud have Cloudness?

Mobility Can Be a Pain in the aaS

Knowing is Half the Battle

The Thing Private Clouds Can Do that Public Clouds Can’t

Infrastructure 2.0 Isn’t Just For Cloud Computing

WILS: Automation versus Orchestration

And the Killer App for Private Cloud Computing Is...

from tag automation

OVF: A few layers short of a full stack

GoogleCL Puts Another Tool in the Devops Integration and Automation Toolbox

(more..)


Technorati Tags: MacVittie,F5,F5 Friday,automation,infrastructure 2.0,orchestration,integration,CLI,cloud computing,virtualization,devops

Heatmaps, iRules Style: Part2

Colin — Thu, 08 Jul 2010 13:22:00 GMT

Last week I talked about generating a heat map 100% via iRules, thanks to the Quova magic in LTM systems, and the good people over at Google letting us use their charting API. This was an outstanding way to visualize the the traffic coming to your application. For those interested in metrics it provides a great way to see this data in a visually pleasing manner. That said, it was pretty basic. All it showed was the United States which, for anyone that has used the internet much, is obviously not representative of the entire web. To be truly useful we’ll need to show the entire world. That’s simple enough. We’ll update the region the map we’re drawing zooms to, so it will look more like this:

Let’s take a look at how this is going to work. Since we were collecting data based on state abbreviations before, we’ll need to first switch that up to use country codes instead. We’ll then change up our Google call so that we’re setting the range covered by the map to the entire world, rather than just the US. While we’re at it, let’s change the name of the subtable we’re using from states to countries, just to keep things more clear. What we end up with is some code that looks very familiar, if you’ve already seen last week’s solution, with a few minor changes:

      set chld ""
      set chd ""
        foreach country [table keys -subtable countries] {
        append chld $country
        append chd "[table lookup -subtable countries $country],"
      }
      set chd [string trimright $chd ","]
      HTTP::respond 200 content "<HTML><center><font size=5>Here is your site's usage by Country:</font><br><br><br><img src='http://chart.apis.google.com/chart?cht=t&chd=&chs=440x220&chtm=world&chd=t:$chd&chld=$chld&chco=f5f5f5,edf0d4,6c9642,365e24,13390a' border='0'><br>br><br><br><a href='/resetmap'>Reset All Counters</a></center></HTML>"

So using that in place of the similar logic in last week’s solution you can get a simple world view of the traffic passing through your site. That’s great and all, but what if you can’t see the detail you’re looking for? What if you want to see the details of Asia’s traffic and be able to decipher the patterns in Japan and the Middle East? What we really need is to build a simple interface to make this more of an application, and less of a single image displayed on a web page.

Well, first of all, we already have all the data collected that we’ll need, if you think about it. We’re already tracking the requests per country, so all we need to do is build out options to allow for users to click a link and zoom to a different region of the map. To do this we’ll set up some simple HTML navigation links at the bottom of the page being generated via the iRule, and set up a switch structure to handle each URI the links pass back into the iRule, and use those to format the HTML appropriately so that we get the right Google charts call. That sounds more complicated than it is. Here’s what it looks like:

    "/heatmap" {
      set chld ""
      set chd ""
        foreach country [table keys -subtable countries] {
        append chld $country
        append chd "[table lookup -subtable countries $country],"
      }
      set chd [string trimright $chd ","]
      HTTP::respond 200 content "<HTML><center><font size=5>Here is your site's usage by Country:</font><br><br><br><img src='http://chart.apis.google.com/chart?cht=t&chd=&chs=440x220&chtm=world&chd=t:$chd&chld=$chld&chco=f5f5f5,edf0d4,6c9642,365e24,13390a' border='0'><br><br>Zoom to region: <a href='/asia'>Asia</a> | <a href='/africa'>Africa</a> | <a href='/europe'>Europe</a> | <a href='/middle_east'>Middle East</a> | <a href='/south_america'>South America</a> | <a href='/usa'>United States</a> | <a href='/heatmap'>World</a><br><br><br><a href='/resetmap'>Reset All Counters</a></center></HTML>"

    "/asia" {
      set chld ""
      set chd ""
        foreach country [table keys -subtable countries] {
        append chld $country
        append chd "[table lookup -subtable countries $country],"
      }
      set chd [string trimright $chd ","]
      HTTP::respond 200 content "<HTML><center><font size=5>Here is your site's usage by Country:</font><br><br><br><img src='http://chart.apis.google.com/chart?cht=t&chd=&chs=440x220&chtm=asia&chd=t:$chd&chld=$chld&chco=f5f5f5,edf0d4,6c9642,365e24,13390a' border='0'><br><br>Zoom to region: <a href='/asia'>Asia</a> | <a href='/africa'>Africa</a> | <a href='/europe'>Europe</a> | <a href='/middle_east'>Middle East</a> | <a href='/south_america'>South America</a> | <a href='/usa'>United States</a> | <a href='/heatmap'>World</a><br><br><br><a href='/resetmap'>Reset All Counters</a></center></HTML>"

…

That section can be repeated once for each available region that Google will let us view (Asia | Africa | Europe | Middle East | South America | United States | World). That then gives us something that looks like this:

As you can see, we now have a world view map that shows the heat of each country, and we have individual links that we can click on along the bottom to take us to a zoom of each country/region to get a more specific look at the info there. As an example, let’s take a look at the data from Asia:

So we now have a nice little heatmapping application. It pulls up a world view of app traffic going to your site or app, it allows you to click around to the different regions of the world to get a more detailed view, and it even lets you re-set the data at will. I can hear some among you asking “What about the states, though?”. If I take away the state view of the US and give a world view, then I’m really trading one limitation for another. Ideally we’d be able to see both, right?

If I want to be able to give a detailed view on both the countries around the world and the states within the US, then I need to expand my data collection a bit. I need to collect both country codes for incoming requests and state abbreviations, where applicable. This means creating a second sub-table within the iRule, and issuing a second whereis per request coming in. Something like this should do:

      set cloc [whereis ]IP::client_addr[ country]
      set sloc [whereis ]IP::client_addr[ abbrev]
      if {[table incr -subtable countries -mustexist $cloc] eq ""} {
          table set -subtable countries $cloc 1 indefinite indefinite
      }
      if {[table incr -subtable states -mustexist $sloc] eq ""} {
          table set -subtable states $sloc 1 indefinite indefinite
      }

Above we’re using the cloc (country location) and sloc (state location) variables to simultaneously track both country codes and state abbreviations in separate sub tables within the iRule. This way we don’t mix up CA (Canada) and CA (California) or similar crossovers and throw our counts off. When doing this, don’t forget to update the resetmap case as well to empty both sub tables, not just one. This also means that we’ll need to slightly change the logic in the “usa” case as opposed to all of the other cases when doing a lookup. If the user wants to view the USA details, we need to do a subtable lookup on the states sub table, everything else uses the countries sub table. Not too horrible.

Okay, we now have heatmaps for all countries, all available zoom regions and a zoom to state level in the US, complete with some rudimentary HTML to make this feel like an application, not just a static image on a web page. Unfortunately, we also have around 140 lines of code, much of which is being repeated. There’s no sense in repeating that HTML over and over, or those logic statements doing the lookups and whatnot. So it’s time to take out the scalpel and start slicing and dicing, looking for unnecessary code.

I started with the HTML. There’s just no reason to repeat that HTML in every single switch case. So I set that in some static variables in the RULE_INIT section and did away with that all together in each switch case. Next, the actual iRules logic is identical if I want to view asia or africa or europe or anything other than the US. The only difference is the HTML changing one word to tell the API where to zoom in. Using a little extra “zoom” logic, I was able to cut down most of that repetitive code as well, by having all of the switch cases other than the USA fall through to the world view case, giving us just two chunks of iRules logic to deal with. Not including the extra variables and tidbits, the core of those two chunks of logic are:

      foreach country [table keys -subtable countries] {
        append chld $country
        append chd "[table lookup -subtable countries $country],"
      }


      foreach state [table keys -subtable states] {
        append chld $state
        append chd "[table lookup -subtable states $state],"
      }

Don’t stop there, though, there’s more to trim! With some more advanced trickery we can combine these two table lookups into a single piece of logic. When all is said and done, here is the final iRule trimmed down to fighting form with a single switch case handling the presentation of all the possible heatmaps generated by Google…pretty cool stuff:

when RULE_INIT {
set static::resp1 "<HTML><center><font size=5>Here is your site's usage by Country:</font><br><br><br><img src='http://chart.apis.google.com/chart?cht=t&chd=&chs=440x220&chtm="
set static::resp2 "&chco=f5f5f5,edf0d4,6c9642,365e24,13390a' border='0'><br><br>Zoom to region: <a href='/asia'>Asia</a> | <a href='/africa'>Africa</a> | <a href='/europe'>Europe</a> | <a href='/middle_east'>Middle East</a> | <a href='/south_america'>South America</a> | <a href='/usa'>United States</a> | <a href='/heatmap'>World</a><br><br><br><a href='/resetmap'>Reset All Counters</a></center></HTML>"
}

when HTTP_REQUEST {
switch -glob [string tolower ]HTTP::uri[] {
    "/asia" -
    "/africa" -
    "/europe" -
    "/middle_east" -
    "/south_america" -
    "/usa" -
    "/world" -
    "/heatmap*" {
      set chld ""
      set chd ""
      set zoom [string map {"/" "" "heatmap" "world"} ]HTTP::uri[]
## Configure the table query to be based on the countries subtable or the states subtable ##
      if {$zoom eq "usa"} {
        set region "states"
      } else {
        set region "countries"
      }
## Get a list of all states or countries and the associated count of requests from that area ##
      foreach rg [table keys -subtable $region] {
        append chld $rg
        append chd "[table lookup -subtable $region $rg],"
      }
      set chd [string trimright $chd ","]
## Send back the pre-formatted response, set in RULE_INIT, combined with the map zoom, list of areas, and request count ##
      HTTP::respond 200 content "${static::resp1}${zoom}&chd=t:${chd}&chld=${chld}${static::resp2}"
    }
    "/resetmap" {
      foreach country [table keys -subtable countries] {
        table delete -subtable countries $country
      }
      foreach state [table keys -subtable states] {
        table delete -subtable states $state
      }
      HTTP::respond 200 Content "<HTML><center><br><br><br><br><br><br>Table Cleared.<br><br><br> <a href='/heatmap'>Return to Map</a></HTML>"
    }
    default {
## Look up country & state locations ##
      set cloc [whereis ]IP::client_addr[ country]
      set sloc [whereis ]IP::client_addr[ abbrev]
## If the IP doesn't resolve to anything, pick a random IP (useful for testing on private networks) ##
      if {($cloc eq "") and ($sloc eq "")} {
        set ip [expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }]
        set cloc [whereis $ip country]
        set sloc [whereis $ip abbrev]
      }
## Set Country ##
      if {[table incr -subtable countries -mustexist $cloc] eq ""} {
          table set -subtable countries $cloc 1 indefinite indefinite
      }
## Set State ##
      if {[table incr -subtable states -mustexist $sloc] eq ""} {
          table set -subtable states $sloc 1 indefinite indefinite
      }
      HTTP::respond 200 Content "Added"
    }
}
}

There we have it, an appropriately trimmed down and sleek application to provide worldwide or regional views of heatmaps showing traffic to your application, all generated 100% via iRules. Again, this couldn’t be done without the awesome Quova abilities of LTM or the Google charting API or, of course, iRules. In the next installment we’ll dig even deeper to see how to turn this application into something even more valuable to those interested in what the users of your site or app are up to.

DevCentral Weekly Podcast 139 – Plugging Into VMware vSphere

DevCentral Weekly Podcast — Thu, 08 Jul 2010 06:08:32 GMT

   Welcome to the one hundred and thirty ninth edition of the DevCentral Weekly Roundtable Podcast! A weekly recap of the interesting things that have been going on within the DevCentral community.

Hosts: Colin Walker, Jason Rahm, Joe Pruitt, Scott Koon, and special guest Charlie Cano.
In this weeks podcast we invited F5’s Charlie Cano on to talk about our new F5 Management Plug-In for VMware vSphere. We finished things up with some discussions on binary content from iRules, partitions with iControl, updates in the iControl Assemblies, and Colin discussion his two posts on Heatmaps with iRules.

All of the links from this recording can be found with the dcpodcast139 tag at Delicious.com. You can also watch the video recording from our UStream.TV channel: DevCentral Podcast 139.

Get the Flash Player to see this player.

Technorati Tags: F5,DevCentral,VMware,vSphere,iRules,partitions,iControl,Heatmaps,Colin Walker,Jason Rahm,Joe Pruitt,Scott Koon,Charlie Cano

Heatmaps, iRules Style: Part 1

Colin — Tue, 29 Jun 2010 15:41:00 GMT

When working with any type of traffic, metrics play an important role. Whether you’re dealing with web traffic, custom TCP application traffic, DNS requests…pretty much anything traveling over the wire, chances are at some point someone in your organization is going to want to know more than just “Yep, the traffic is flowing.”. This is where metrics come in. You’ll report on, well, something. Uptime, requests per x time period, utilization, or some other thing will need to be rolled up and reported to those interested parties so that they can make sure things are running properly, show off the great stats in presentations, or check the “metrics gathered” box off their quarterly objectives scorecard. Regardless, it’ll be up to you to gather that info.

I’ve been tinkering with a way to allow iRules to make that easier, and to allow those interested parties to see some usage statistics in a visually interesting, real-time manner, without adding much heavy lifting for you or your application. The idea is simple, create a heat map view of your HTTP requests, mapped to the locations around the United States (to start with). This will give you an idea which areas are most highly utilizing your application in a very easy on the eyes fashion. Best of all, of course, is that we’re going to generate this 100% with iRules. When all is said and done, you should end up with something that looks about like this.

This is possible for a couple reasons. First of all, iRules is awesome, and allows for all sorts of programmatic magic while processing traffic which we’ll get into below. Second though, and equally as important for this task is the somewhat new and completely amazing Quova database that iRules has access to. By issuing a single, simple command via iRules I’m able to easily look up the originating location of any IP address. I make great use of that in this example, and it wouldn’t be possible without their data.

Okay, on to the solution. We need to be able to keep a counter in memory and increment it. We naturally want to be able to do this in the most efficient way possible to minimize drag on your LTM. So we need something that is “permanent”, global (not scoped to a given session), highly efficient, easily counted or incremented…hmm. If you’re like me, and running 10.1 or later, tables should be leaping to mind right about now. So let’s take a look at a relatively simple counter mechanism using tables:

when HTTP_REQUEST {
set loc [whereis ]IP::client_addr[ abbrev]
if {$loc eq ""} {
    set ip [expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }]
    set loc [whereis $ip abbrev]
}
if {[table incr -subtable states -mustexist $loc] eq ""} {
    table set -subtable states $loc 1 indefinite indefinite
}
}

So above we have a relatively simple block that looks up the location of every inbound IP address using the whereis command. For testing and demonstration purposes it’s then setting a random IP address in case you can’t look up the incoming IP (all of my requests were on an internal network so none resolved and I wanted to test the functionality. You’ll want to remove that set ip set loc section if you aren’t looking for random IP info once you’re out of the testing or demo phase. It then increments a subtable entry for the state returned in the lookup. It’s setting the subtable entry to have an indefinite timeout and lifetime the first time a request for that state comes in, otherwise just incrementing it by 1.

That works great and all, but now we have this indefinite lifetime entry, many of them probably, with data building up. We need some mechanism to reset the tables to 0, but I don’t want those to occur on a given lifetime or timeout, since I want it to be manual. I.E. I want the data to collect until you push the button to say “reset data”, effectively. So now we need another section to reset things on demand. What we’ll want to do is get a list of all the entries we’ve created and delete them. Fortunately, since we’ve been using a subtable, that’s trivial. We’ll just get a list of the keys in the subtable and delete each one. That would look something like this:

if {[HTTP::uri] starts_with "/resetmap"} {
foreach state [table keys -subtable states] {
    table delete -subtable states $state
}
}

Now that we have a table counting every request that comes in, and a way to reset those tables as needed, we’re well on our way. All that’s left now is to actually output the heatmap that shows the visual representation of the compiled data we’re storing in the table shown above. To do that, we’ll query the table and get a list of all of the state abbreviation codes, along with their respective number of requests so far, and we’ll format that into a URL that is a call to Google’s charting API, using the chld and chd variables. Lastly, for presentation, we’ll wrap the whole mess in some very basic HTML (forgive me web-designers, for I am not one of you) that makes it feel a bit like an interface, as well as tweaking the default mapping color gradients a bit. The whole thing, when put together, looks like this:

when HTTP_REQUEST {
if {[HTTP::uri] starts_with "/heatmap"} {
    set chld ""
    set chd ""
    foreach state [table keys -subtable states] {
      append chld $state
      append chd "[table lookup -subtable states $state],"
    }
    set chd [string trimright $chd ","]
    HTTP::respond 200 content "<HTML><center><font size=5>Here is your site's usage by state:</font><br><br><br><img src='http://chart.apis.google.com/chart?cht=t&chd=&chs=440x220&chtm=usa&chd=t:$chd&chld=$chld&chco=f5f5f5,edf0d4,6c9642,365e24,13390a' border='0'><br><br><br><a href='/resetmap'>Reset Map</a></center></HTML>"
} elseif {[HTTP::uri] starts_with "/resetmap"} {
    foreach state [table keys -subtable states] {
    table delete -subtable states $state
}
    HTTP::respond 200 Content "<HTML><center><br><br><br>Table Cleared.<br><br><br> <a href='/heatmap'>Return to Map</a></HTML>"
} else {
    set loc [whereis ]IP::client_addr[ abbrev]
    if {$loc eq ""} {
      set ip [expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }].[expr { int(rand()*255) }]
      set loc [whereis $ip abbrev]
    }
    if {[table incr -subtable states -mustexist $loc] eq ""} {
      table set -subtable states $loc 1 indefinite indefinite
    }
}
}

There you have it, a fully functioning heatmapping application built entirely in iRules, with some much appreciated assistance from Quova and Google’s charting API. This is, of course, an amazingly basic look at what this type of functionality can do. In Part 2 of this series I’ll be showing you how you can make this even more powerful and granular to give you a look at not just overall usage information but some more specific bits that will likely prove even more useful. Many thanks to Matt Cauthorn, one of the many truly outstanding engineers here at F5, for getting the juices flowing on this one and riffing on it with me to come up with the idea and start the planning.

iRules 101 - #17 – Mapping Protocol Fields with the Binary Scan Command

citizen_elah — Thu, 24 Jun 2010 13:35:05 GMT

Introduction

An iRule is a powerful and flexible feature of BIG-IP devices based on F5's exclusive TMOS architecture. iRules provide you with unprecedented control to directly manipulate and manage any IP application traffic. iRules utilizes an easy to learn scripting syntax and enables you to customize how you intercept, inspect, transform, and direct inbound or outbound application traffic. In this series of tech tips, we'll talk about the TCL language, its usage and control structures, as well as iRule extensions to the TCL language.

The Binary Scan Command

The binary scan command, like the scan command covered in the last entry in this series, parses strings. Only, as the adjective indicates, it parses binary strings. In this article, I’ll highlight the command syntax and a few of the format string options below. Check the man page for the complete list of format options.

binary scan string formatString ?varName varName … ?

c - The data is turned into count 8-bit signed integers and stored in the corresponding variable as a list. If count is *, then all of the remaining bytes in string will be scanned. If count is omitted, then one 8-bit integer will be scanned.

S - The data is interpreted as count 16-bit signed integers represented in big-endian byte order. The integers are stored in the corresponding variable as a list. If count is *, then all of the remaining bytes in string will be scanned. If count is omitted, then one 16-bit integer will be scanned.

H - The data is turned into a string of count hexadecimal digits in high-to-low order represented as a sequence of characters in the set "0123456789abcdef". The data bytes are scanned in first to last order with the hex digits being taken in high-to-low order within each byte. Any extra bits in the last byte are ignored. If count is *, then all of the remaining hex digits in string will be scanned. If count is omitted, then one hex digit will be scanned.

Do the Research First!

Before you can start mapping fields, you need to know where the delimiters are, and that takes a little research. In this example, I want to list out the cipher suites presented to the BIG-IP LTM by the browser (more on that later), so I’m going to decode the fields in an SSLv3/TLSv1 client hello message. I took a capture with Wireshark to find the field delineations, but you could also pull this information from RFC 2246 if you are so inclined. Table 1 below shows the fields I’ll extract or skip over in the course of the iRule code.

So why do I want to look at the cipher suites? I was contacted by a friend who wanted to increase security to 256-bit ciphers for those browsers that supported it, but would still allow users that don’t to connect as well so they could be redirected to an error page requesting they upgrade. With the cipher settings as they were, the unsupported browser (in this case, IE6) could connect, but the newer browsers connected at 128-bit as well instead of 256-bit. I later learned (courtesy of none other than hoolio) that I could use @STRENGTH in my cipher settings to force the capable browsers up, but for the benefit of demonstrating the use the binary string command, I’ll proceed with the iRule.

Standard Behaviors

I set up a clientssl profile with these cipher settings: “DEFAULT:!ADH:!EXPORT40:!EXP:!LOW”. Those settings result in my test browsers (Chrome 5, FF 3.6.4, IE 8) all using the RC4-SHA 128-bit cipher. I used a simple iRule to display the information back to the client:

SSL Cipher in Use

when HTTP_REQUEST { set cipher_info "[SSL::cipher name]:[SSL::cipher bits]" HTTP::respond 200 content "$cipher_info" }

Again, the easy (and better) fix here is to add strength to the cipher settings in the SSL profile like so: “DEFAULT:!ADH:!EXPORT40:!EXP:!LOW:@STRENGTH”, but instead I created an additional clientssl profile, adding not medium to the settings: “DEFAULT:!ADH:!EXPORT40:!EXP:!LOW:!MEDIUM”. Once I had that profile in place, work on the iRule could begin.

Using Binary Scan

Enough prep talk already, let me get to the good stuff! The iRule starts in CLIENT_ACCEPTED, disabling SSL and doing a TCP collect:

Binary Scans

when CLIENT_ACCEPTED { set lsec 0 SSL::disable TCP::collect } when CLIENT_DATA { if { ! [info exists rlen] } { binary scan [TCP::payload] cSS rtype sslver rlen #log local0. "SSL Record Type $rtype, Version: $sslver, Record Length: $rlen" # SSLv3 / TLSv1.0 if { $sslver > 767 && $sslver < 770 } { if { $rtype != 22 } { log local0. "Client-Hello expected, Rejecting. \ Src: [IP::client_addr]:[TCP::remote_port] -> \ Dst: [IP::local_addr]:[TCP::local_port]" reject return }

As shown in the binary scan syntax, “c” grabs one byte and returns a signed integer, and “S” grabs two-bytes and returns a signed integer. The rest of the payload is thrown away. From table 1, we know the first three fields of SSLv3/TLSv1 client hello’s are one-byte, two-byte, two-byte, so a format string of cSS followed by three variable names takes care of the parsing. The clientssl profile doesn’t yet support TLS 1.1 or 1.2, so I’m only testing for SSLv3/TLSv1. The SSLv2 and earlier record format is different altogether, so the initial fields wouldn’t map this way anyway. This is true of IE6, which by default issues a SSLv2 client hello but supports SSLv3/TLSv1. Confused yet? Anyway, once the test for SSLv3/TLSv1 is complete, I then make sure the record type is a client hello, if not, no reason to continue.

Once the verification is complete that this really is a client hello, there is a string (pun intended!) of binary scans to sort the rest of the details out:

Binary Scans

#Collect rest of the record if necessary if { [TCP::payload length] < $rlen } { TCP::collect $rlen #log local0. "Length is $len" } #skip record header and random data set field_offset 43 #set the offset binary scan [TCP::payload] @${field_offset}c sessID_len set field_offset [expr {$field_offset + 1 + $sessID_len}] #Get cipherlist length binary scan [TCP::payload] @${field_offset}S cipherList_len #log local0. "Cipher list length: $cipherList_len" #Get ciphers, separate into a list of elements set field_offset [expr {$field_offset + 2}] set cipherList_len [expr {$cipherList_len * 2}] binary scan [TCP::payload] @${field_offset}H${cipherList_len} cipherlist #log local0. "Cipher list: $cipherlist"

I’m using the “@” symbol to skip bytes in the binary string. Also, the curly brackets are necessary to surround the variable name to distinguish your variable from the significance of the format string characters. In the first binary scan above, I’m skipping the record header and all the random data as I don’t need that information, but I do need the session ID length so I can set my offset properly to get to the cipher list. In the second binary scan, again I’m skipping (from original payload) the new offset and then storing another two-byte signed integer that tells me how long the cipher list is. From that information, I set a new offset, set the cipher list length (number of ciphers * 2, each cipher is two-bytes) and then perform the final binary scan to store the entire cipher list (in hex high->low characters, hence the “H” in the string format).

The rest of the rule is processing that cipher list (thanks to Colin for some tcl-fu in the for loop below), and sending users to a different clientssl profile if any of the ciphers in the client hello list match the ciphers in a string datagroup you define, in this case, accepted_ciphers (values were four hex character pairs, like c005, 0088, etc). For browsers incapable of the higher security profile, they would be redirected to an error page instructing them to upgrade.

Rule Conclusion

set profile_select 0 for { set x 0 } { $x < $cipherList_len } { incr x 4 } { set start $x set end [expr {$x+3}] #Check cipher against 256-bit ciphers in accepted_ciphers data group if { [matchclass ]string range $cipherlist $start $end[ equals accepted_ciphers] } { #If 256-bit capable, select High Security profile set profile_select "SSL::profile clientssl_highSecurity" break } } #Set New Profile with eval since SSL::profile errors in CLIENT_ACCEPTED event if { $profile_select != 0 } { eval $profile_select #log local0. "Profile changed to High Security" } else { set lsec 1 } event CLIENT_DATA disable SSL::enable TCP::release } else { set lsec 1 event CLIENT_DATA disable SSL::enable TCP::release } } }

Conclusion

The rule is useless in production, but is a good teaching example on the use of binary scan. There are a few more good examples in the wiki. Check out this DNS modification example and this tftp file server iRule (companion article for the tftp iRule here). That one throws in a binary format to boot. Happy coding!

DevCentral Weekly Podcast 137 – Next Week Will Be HOT

DevCentral Weekly Podcast — Thu, 24 Jun 2010 06:13:32 GMT

   Welcome to the one hundred and thirty seventh edition of the DevCentral Weekly Roundtable Podcast! A weekly recap of the interesting things that have been going on within the DevCentral community.

Hosts: Colin Walker, Jason Rahm, Joe Pruitt, and Scott Koon.
In this weeks podcast we discussed IPv6 to IPv4 DNS handling, Switch Statement Conditions, Mapping Protocol Fields With Binary Scan, pyControl with GTM, Optimizing if statement conditions, and a preview of next weeks hotness.

All of the links from this recording can be found with the dcpodcast137 tag at Delicious.com. You can also watch the video recording from our UStream.TV channel: DevCentral Podcast 137.

Get the Flash Player to see this player.

Technorati Tags: F5,DevCentral,IPv6,IPv4,DNS,Binary,pyControl,GTM,Colin Walker,Jason Rahm,Joe Pruitt,Scott Koon

DevCentral Weekly Podcast 136 – How To Keep The Giant At Bay

DevCentral Weekly Podcast — Thu, 17 Jun 2010 06:14:18 GMT

   Welcome to the one hundred and thirty sixth edition of the DevCentral Weekly Roundtable Podcast! A weekly recap of the interesting things that have been going on within the DevCentral community.

Hosts: Colin Walker, George Watkins, Jason Rahm, Joe Pruitt, and Scott Koon.
In this weeks podcast we welcomed George Watkins to the DevCentral team! We discussed Migration to GTM, Query iRule Priority Numbers, CIO Conversation with John Matthews, Correlation Numbers in iRules, iRule logging, a fix in the iRule Editor, Strategic Points of Control, iRule Parsing Tricks, Cookie Based Connection Limiting, Scanning for Unsafe URLs, Firefox Losing Foothold, and Learning from F5’s Success.

All of the links from this recording can be found with the dcpodcast136 tag at Delicious.com. You can also watch the video recording from our UStream.TV channel: DevCentral Podcast 136.

Get the Flash Player to see this player.

Technorati Tags: F5,DevCentral,GTM,iRules,CIO,John Matthews,logging,iRule Editor,Cookies,URLs,FireFox,Colin Walker,George Watkins,Jason Rahm,Joe Pruitt,Scott Koon

CIO Conversation – John Matthews On Intelligent File Management

DevCentral Interview — Tue, 15 Jun 2010 06:06:52 GMT


IT organizations in every industry face a common set of challenges as they work to keep the business up and running. Like many companies, F5 Networks was experiencing a host of problems related to the rapid growth of unstructured file data. Spending on storage was rising quarterly, routine management tasks required frequent user interruptions and backups were taking 96 hours to complete.

After adding the ARX technology to its product portfolio via an acquisition, F5 put the solution to work in its own environment. Listen to this podcast to hear F5 CIO and Vice President of IT John Matthews explain how the ARX solution enabled the company to dramatically reduce storage costs, even as data storage continued to grow. Through the use of intelligent file management, the IT team slashed backup times to just 4 hours. Downtime is a rare occurrence instead of a regular event, and the IT organization was able to introduce vendor diversity into its environment. The icing on the cake for a CIO? An ROI of less than 4 months.

Get the Flash Player to see this player.

Technorati Tags: F5,DevCentral,ARX,File Virtualization,Storage,John Matthews

DevCentral Weekly Podcast 135 – Sweating in Nola

DevCentral Weekly Podcast — Thu, 10 Jun 2010 06:03:45 GMT

   Welcome to the one hundred and thirty fifth edition of the DevCentral Weekly Roundtable Podcast! A weekly recap of the interesting things that have been going on within the DevCentral community.

Hosts: Colin Walker, Jason Rahm, Joe Pruitt, and Scott Koon.
In this weeks podcast we discussed Jason’s first conference as an F5 employee at TechEd New Orleans, a blog post about our Microsoft BranchCache iRule, Gomez Automated Perfomance Monitoring with Datacenter Identification, Stream expressions from Data Groups, Java iControl Sources, and our new F5 Careers topic group.

All of the links from this recording can be found with the dcpodcast135 tag at Delicious.com. You can also watch the video recording from our UStream.TV channel: DevCentral Podcast 135.

Get the Flash Player to see this player.

Technorati Tags: F5,DevCentral,TechEd,Microsoft,BranchCache,Gomez,Datacenter,Stream Profile,Data Groups,Java,iControl,Careers,Colin Walker,Jason Rahm,Joe Pruitt,Scott Koon

Automated Gomez Performance Monitoring Part 2 – DataCenter Identification

Joe — Mon, 07 Jun 2010 13:34:22 GMT

In my previous article titled “Automated Gomez Performance Monitoring”, I gave an overview of Gomez and the partnership between F5 Networks and Gomez to assist users of both technologies to integrate the two together.

In that article, I illustrated how to use iRules to serve up the Gomez bootstrap JavaScript code eliminating the need to install in on a backend web server. The iRule is also used to automatically inject the client side JavaScript into the response stream for your webpages.

The iRule is completely standalone and requires no knowledge of how your web applications are configured or where they are located. You simply apply the iRule to the virtual server fronting your applications and Gomez takes care of the rest by organizing the requests by the URLs of your pages. The Gomez JavaScript looked like this:

1: <SCRIPT LANGUAGE="JavaScript"></SCRIPT>

4: <script src="/js/axfTag.js" type="text/javascript"></script>

The only configuration change needed in the iRule was to set the Gomez Account Id in the “acctId” parameter. But, you may notice there are two additional parameters in there that I didn’t use. The first one is the Page Identifier (pgId) which I will discuss in a future article. What I will discuss here is the Group Identifier (grpId).

The Group Identifer

On problem with performance testing when you are running a multi-datacenter infrastructure using the same domain name, like we are here at devcentral.f5.com, is that when you look at your reports, all application requests will show up on the same graph. If you are having a datacenter specific issue, it is not easy to narrow down where the bottleneck occurs as the reports will treat all requests as a single application.

The Group Identifier property allows you to give one additional search criteria to the Gomez reporting system. It would make perfect sense to use this feature to mark the application request to a given datacenter, or even to a given virtual server and that is exactly what we did on our implementation. We run DevCentral.f5.com out of multiple data centers for redundancy. And top top that off, within the datacenters, we have multiple virtual servers running the application. We have production virtuals with WebAccelerator and development virtuals without the web optimization features enabled. Since the iRule is running in the network layer and has information about the client side of the connection, it was very straightforward to build a lookup table that mapped the client side address to a specific virtual server.

1: set GOMEZ_GROUPID_LIST [list \

2: "xxx.xxx.xxx.1 DC_PRD_DC1" \

3: "xxx.xxx.xxx.2 DC_PRD_DC2_C1" \

4: "xxx.xxx.xxx.3 DC_PRD_DC2_C2" \

5: "xxx.xxx.xxx.4 DC_PRD_DC2_C2_NO_WAM" \

6: "xxx.xxx.xxx.5 DC_PRD_DC3" \

7: "xxx.xxx.xxx.6 DC_DEV_DC1" \

8: "xxx.xxx.xxx.8 DC_DEV_DC2" \

9: "xxx.xxx.xxx.9 DC_STG_DC1" \

10: ];

11:

12: set gomez_group_id [findclass ]IP::local_addr[ $::GOMEZ_GROUPID_LIST " "]

In addition to targeting production virtuals, we were also able to target internal development systems so that they run the Gomez injection tests and then be excluded from the final reports.

The above code builds a lookup table (replace xxx.xxx.xxx.xxx with your local ip addresses for your given virtual servers). Then the findclass method is used to lookup the address and return a user friendly name for the data center.

Once the user friendly group name is found, it can be included with the above JavaScript by the following code:

1: set gomez_client [subst {<SCRIPT LANGUAGE="JavaScript"></SCRIPT>

4: <script src="/js/axfTag.js" type="text/javascript"></script>

5: } ];

You’ll notice I had to throw a “subst” command around the code snippet so that TCL would correctly replace the value of the $gomez_group_id variable in the gomez_client variable. After this, the iRule is the same as in the previous example.

Conclusion

If you are running a multi-datacenter environment and would like to enable Gomez performance monitoring to your applications, this iRule be all you need to help you target down datacenter specific issues with your application.

Download

You can download the full script in the iRules CodeShare under GomezInjectionWithGroupId.

DevCentral Weekly Roundup Episode 134 – Blogizing The Tutorials

DevCentral Weekly Podcast — Thu, 03 Jun 2010 06:09:02 GMT

   Welcome to the one hundred and thirty forth edition of the DevCentral Weekly Roundtable Podcast! A weekly recap of the interesting things that have been going on within the DevCentral community.

Hosts: Colin Walker, Jason Rahm, Jeff Browning, Joe Pruitt, and Scott Koon.
In this weeks podcast we discussed Blogizing The Tutorials, The Infrastructure Beef, F5 Microsoft Virtualization Guide, The Problem with Storage Growth, a few posts by The F5 Guy on ASM Training, and The Box of Awesomeness, f5poolmgr, iControl SDK 10.2 downloads and .NET Assembly, and a new Cacti Template for F5.

Oh, and stop by the F5 Booth at Microsoft TechEd 2010 in New Orleans 6/7-10 and say hi to Jason!

All of the links from this recording can be found with the dcpodcast134 tag at Delicious.com. You can also watch the video recording from our UStream.TV channel: DevCentral Podcast 134.

Get the Flash Player to see this player.

Technorati Tags: F5,DevCentral,Tutorials,Infrastructure,Microsoft,Virtualization,Storage,The F5 Guy,ASM,iControl,Cacti,Colin Walker,Jason Rahm,Jeff Browning,Joe Pruitt,Scott Koon

Tutorial: Remove iRules from Virtual Servers with Tmsh

DevCentral — Mon, 31 May 2010 21:42:07 GMT

The companion article for this video can be found here.

Tutorial: Site Navigation Overview

DevCentral — Mon, 31 May 2010 21:39:43 GMT

Tutorial: Vim iRule Editor Customizations

DevCentral — Mon, 31 May 2010 21:38:35 GMT

Tutorial: Where Did the Forums Go?

DevCentral — Mon, 31 May 2010 21:37:21 GMT

Tutorial: Where are the Lab Projects?

DevCentral — Mon, 31 May 2010 21:36:37 GMT

Tutorial: Posting to the Forums

DevCentral — Mon, 31 May 2010 21:34:58 GMT

Tutorial: Vim iRule Editor Ubuntu Installation

DevCentral — Mon, 31 May 2010 21:32:58 GMT

Tutorial: Providing Feedback

DevCentral — Mon, 31 May 2010 21:31:38 GMT

Tutorial: Vim iRule Editor Usage

DevCentral — Mon, 31 May 2010 21:30:44 GMT

Tutorial: Configuring Jing Pro for Screencast Contributions to DevCentral

DevCentral — Mon, 31 May 2010 21:28:56 GMT

Tutorial: Configuring Windows Live Writer for Tech Tips

DevCentral — Mon, 31 May 2010 21:26:40 GMT

Tutorial: Contributing a Tech Tip

DevCentral — Mon, 31 May 2010 21:25:56 GMT

Tutorial: Group Social Features

DevCentral — Mon, 31 May 2010 21:24:45 GMT

Tutorial: Finding Groups

DevCentral — Mon, 31 May 2010 21:23:26 GMT

Tutorial: What’s a Group?

DevCentral — Mon, 31 May 2010 21:22:11 GMT

Tutorial: Customizing your Forum Experience

DevCentral — Mon, 31 May 2010 21:21:03 GMT

The DevCentral Homepage

DevCentral — Mon, 31 May 2010 21:20:00 GMT

DevCentral Top Contributors

DevCentral — Mon, 31 May 2010 21:18:47 GMT

5-Minute iRules Tutorial: Variables

DevCentral — Mon, 31 May 2010 21:17:06 GMT

Writing to and rotating custom log files

smp — Thu, 27 May 2010 11:27:00 GMT

Sometimes I need to log information from iRules to debug something. So I add a simple log statement, like this:

when HTTP_REQUEST {
if { [HTTP::uri] equals "/secure" } {
    log local0. "[IP::remote_addr] attempted to access /secure"
}
}

This is fine, but it clutters up the /var/log/ltm log file. Ideally I want to log this information into a separate log file. To accomplish this, I first change the log statement to incorporate a custom string - I chose the string "##":

when HTTP_REQUEST {
if { [HTTP::uri] equals "/secure" } {
    log local0. "##[IP::remote_addr] attempted to access /secure"
}
}

Now I have to customize syslog to catch this string, and send it somewhere other than /var/log/ltm. I do this by customizing syslog with an include statement:

tmsh modify sys syslog include '"
filter f_local0 {
    facility(local0) and not match(\": ##\");
};

filter f_local0_customlog {
    facility(local0) and match(\": ##\");
};

destination d_customlog {
    file(\"/var/log/customlog\" create_dirs(yes));
};

log {
    source(local);
    filter(f_local0_customlog);
    destination(d_customlog);
};
"'

save the configuration change:

tmsh save / sys config

and restarting the syslog-ng service:

tmsh restart sys service syslog-ng

The included "f_local0" filter overrides the built-in "f_local0" syslog-ng filter, since the include statement will be the last one to load. The "not match" statement is regex which will prevent any statement containing a “##” string from being written to the /var/log/ltm log. The next filter, "f_local0_customlog", catches the "##" log statement and the remaining include statements handle the job of sending them to a new destination which is a file I chose to name "/var/log/customlog".

You may be asking yourself why I chose to match the string ": ##" instead of just "##". It turns out that specifying just "##" also catches AUDIT log entries which (in my configuration) are written every time an iRule with the string "##" is modified. But only the log statement from the actual iRule itself will contain the ": ##" string. This slight tweak keeps those two entries separated from each other.

So now I have a way to force my iRule logging statements to a custom log file. This is great, but how do I incorporate this custom log file into the log rotation scheme like most other log files? The answer is with a logrotate include statement:

tmsh modify sys log-rotate syslog-include '"
/var/log/customlog {
    compress
    missingok
    notifempty
}"'

and save the configuration change:

tmsh save / sys config

Logrotate is kicked off by cron, and the change should get picked up the next time it is scheduled to run.

And that's it. I now have a way to force iRule log statements to a custom log file which is rotated just like every other log file. It’s important to note that you must save the configuration with "tmsh save / sys config" whenever you execute an include statement. If you don't, your changes will be lost then next time your configuration is loaded. That's why I think this solution is so great - it's visible in the bigip_sys.conf file - not like customizing configuration files directly. And it's portable.

Problems Overcome During a Major LTM Software/Hardware Upgrade

smp — Thu, 27 May 2010 10:06:00 GMT

I recently completed a successful major LTM hardware and software migration which accomplished two high-level goals:

·         Software upgrade from v9.3.1HF8 to v10.1.0HF1

·         Hardware platform migration from 6400 to 6900

I encountered several problems during the migration event that would have stopped me in my tracks had I not (in most cases) encountered them already during my testing. This is a list of those issues and what I did to address them. While I may not have all the documentation about these problems or even fully understand all the details, the bottom line is that they worked. My hope is that someone else will benefit from it when it counts the most (and you know what I mean).

Problem #1 – Unable to Access the Configuration Utility (admin GUI)

The first issue I had to resolve was apparent immediately after the upgrade finished. When I tried to access the Configuration utility, I was denied:

Access forbidden!

You don't have permission to access the requested object.

Error 403

I happened to find the resolution in SOL7448: Restricting access to the Configuration utility by source IP address. The SOL refers to bigpipe commands, which is what I used initially:

bigpipe httpd allow all add

bigpipe save

Since then, I’ve developed the corresponding TMSH commands, which is F5’s long-term direction toward managing the system:

tmsh modify sys httpd allow replace-all-with {all}

tmsh save / sys config

Problem #2 – Incompatible Profile

I encountered the second issue after the upgraded configuration was loaded for the first time:

[root@bigip2:INOPERATIVE] config # BIGpipe unknown operation error: 01070752:3: Virtual server vs_0_0_0_0_22 (forwarding type) has an incompatible profile.

By reviewing the /config/bigip.conf file, I found that my forwarding virtual servers had a TCP profile applied:

virtual vs_0_0_0_0_22 {

destination any:22

ip forward

ip protocol tcp

translate service disable

profile custom_tcp

}

Apparently v9 did not care about this, but v10 would not load until I manually removed these TCP profile references from all of my forwarding virtual servers.

Problem #3 – BIGpipe parsing error

Then I encountered a second problem while attempting to load the configuration for the first time:

BIGpipe parsing error (/config/bigip.conf Line 6870): 012e0022:3: The requested value (x.x.x.x:3d-nfsd {) is invalid (show | <pool member list> | none) [add | delete]) for 'members' in 'pool'

While examining this error, I noticed that the port number was translated into a service name – “3d-nfsd”. Fortunately during my initial v10 research, I came across SOL11293 - The default /etc/services file in BIG-IP version 10.1.0 contains service names that may cause a configuration load failure. While I had added a step in my upgrade process to prevent the LTM from service translation, it was not scheduled until after the configuration had been successfully loaded on the new hardware. Instead I had to move this step up in the overall process flow:

bigpipe cli service number

b save

The corresponding TMSH commands are:

tmsh modify cli global-settings service number

tmsh save / sys config

Problem #4 – Command is not valid in current event context

This was the final error we encountered when trying to load the upgraded configuration for the first time:

BIGpipe rule creation error: 01070151:3: Rule [www.mycompany.com] error: line 28: [command is not valid in current event context (HTTP_RESPONSE)] [HTTP::host]

While reviewing the iRule it was obvious that we had a statement which didn’t make any sense, since there is no Host header in an HTTP response. Apparently it didn’t bother v9, but v10 didn’t like it:

when HTTP_RESPONSE {
switch -glob [string tolower ]HTTP::host[] {
    <do some stuff>
}
}

We simply removed that event from the iRule.

Problem #5: Failed Log Rotation

After I finished my first migration, I found myself in a situation where none of the logs in the /var/log directory were not being rotated. The /var/log/secure log file held the best clue about the underlying issue:

warning crond[7634]: Deprecated pam_stack module called from service "crond"

I had to open a case with F5, who found that the PAM crond configuration file (/config/bigip/auth/pam.d/crond) had been pulled from the old unit:

#

# The PAM configuration file for the cron daemon

#

#

auth    sufficient      pam_rootok.so

auth    required        pam_stack.so service=system-auth

auth    required        pam_env.so

account required        pam_stack.so service=system-auth

session required        pam_limits.so

#session        optional        pam_krb5.so

I had to update the file from a clean unit (which I was fortunate enough to have at my disposal):

#

# The PAM configuration file for the cron daemon

#

#

auth       sufficient pam_rootok.so

auth       required   pam_env.so

auth       include    system-auth

account    required   pam_access.so

account    sufficient pam_permit.so

account    include    system-auth

session    required   pam_loginuid.so

session    include    system-auth

and restart crond:

bigstart restart crond

or in the v10 world:

tmsh restart sys service crond

Problem #6: LTM/GTM SSL Communication Failure

This particular issue is the sole reason that my most recent migration process took 10 hours instead of four. Even if you do have a GTM, you are not likely to encounter it since it was a result of our own configuration. But I thought I’d include it since it isn’t something you’ll see documented by F5. One of the steps in my migration plan was to validate successful LTM/GTM communication with iqdump. When I got to this point in the migration process, I found that iqdump was failing in both directions because of SSL certificate verification despite having installed the new Trusted Server Certificate on the GTM, and Trusted Device Certificates on both the LTM and GTM. After several hours of troubleshooting, I decided to perform a tcpdump to see if I could gain any insight based on what was happening on the wire. I didn’t notice it at first, but when I looked at the trace again later I noticed the hostname on the certificate that the LTM was presenting was not correct. It was a very small detail that could have easily been missed, but was the key in identifying the root cause.

Having dealt with Device Certificates in the past, I knew that the Device Certificate file was /config/httpd/conf/ssl.crt/server.crt. When I looked in that directory on the filesystem, there I found a number of certificates (and subsequently, private keys in /config/httpd/conf/ssl.key) that should not have been there. I also found that these certificates and keys were pulled from the configuration on the old hardware. So I removed the extraneous certificates and keys from these directories and restarted the httpd service (“bigstart restart httpd”, or “tmsh restart sys service crond”). After I did that, the LTM presented the correct Device Certificate and LTM/GTM communication was restored. I'm still not sure to this day how those certificates got there in the first place...

Investigating Efficiencies in iRules: Handling Wildcards in Hostnames

citizen_elah — Tue, 25 May 2010 09:57:00 GMT

This isn’t the first time I’ve looked at optimization techniques with iRules. I’ve blogged about why testing matters, and hosted a small contest as a result of that testing (hmm, I think I still owe natty76 a t-shirt…) to give other iRulers out there an opportunity to find efficiencies. Well, every now and then a post catches my eye and I start to think about the fundamental differences in approach to a problem. A thread in the Adv Design/Config General Discussion forum, Bypassing a VIP based on destination address, was searching for a way to match a host that had a wildcard in the middle: webmail.<something>.server.com. Hamish, one of our MVPs, suggested using the matches_glob operator. (Note: the html tags in the HTTP::respond content section were removed for clarity. The editor wasn’t displaying them properly.)

iRule Solutions

matches_glob

when HTTP_REQUEST { if { [HTTP::host] matches_glob "webmail.*.server.com" } { HTTP::respond 200 content { OK } } }

I came up with a couple more alternatives, one utilizing nested if conditionals and the other using the string match command.

nested conditional & string match

when HTTP_REQUEST { if { [llength ]split [HTTP::host] "."[] == 4 } { if { [HTTP::host] starts_with "webmail" } { if { [HTTP::host] ends_with "server.com" } { HTTP::respond 200 content { OK } } } } } ********** when HTTP_REQUEST { if { [string match -nocase webmail.*.server.com ]HTTP::host[] } { HTTP::respond 200 content { OK } } }

After reaching out to unRuleY with some testing results (more on that later), he proposed a couple more options, both the same approach, but one using the domain command and the other using getfield.

domain & getfield

when HTTP_REQUEST { if { ([domain ]HTTP::host[ 4] eq [HTTP::host]) && \ ([HTTP::host] starts_with "webmail.") && \ ([HTTP::host] ends_with ".server.com") } { HTTP::respond 200 content { OK } } } ********** when HTTP_REQUEST { if { ([getfield ]HTTP::host[ "." 5] eq "") && \ ([HTTP::host] starts_with "webmail.") && \ ([HTTP::host] ends_with ".server.com") } { HTTP::respond 200 content { OK } } }

If you’re counting, that’s five iRule solutions to one problem. Three unique solutions, really. If I were to classify these, I’d make the nested conditional one solution, matches_glob/string match another solution, and domain/getfield the final solution. So how do all these iRules stack up against one another?

Testing Methodology

Testing was performed on a BIG-IP 3600 platform running the 10.2 release. The device under test had no other traffic flowing during testing (except a control ssh session and iControl sessions from the editor) and only the rule under test applied to the virtual server. Three runs of 10,000 requests from apachebench were averaged together for each iRule solution. The iRules themselves were devoid of any logic not necessary for the solution other than the HTTP::respond to confirm the rule was working.

Results

If you’re the pretty picture type, the results are shown in Figure 1 to the right. So it’s safe to say that my idea of the nested conditional was not the best. It works, but it is not efficient. The matches_glob and string match solution results are the most efficient, and performance is really close, so close that I reached out to get some insight on this one. Straight from unRuleY:

The reason is that operators are compiled directly into byte-code and hence no further parsing is necessary when the iRule is evaluated. In the [string match] command case, the string command can’t be completely compiled into byte-code. Instead, the string command function is invoked. This function then also has to determine which arguments are options.

The second grouping of solutions, domain & getfield, are even closer in relation to each other, though significantly less efficient (as a group) than the first. The differences in domain and getfield are most likely due to the fact that the domain command has a built in split function (on “.”), whereas with the getfield command, it must be supplied as an argument and evaluated. Actual numbers and efficiencies in relation to the other solutions are shown in the table below. Note that the percentages are in relation to the left column, so the matches_glob solution is 4.25% more efficient than the string match solution, and the domain solution is 10.55% less efficient than the matches_glob solution.

Approach Average CPU Cycles matches_glob string match domain getfield nested conditional

matches_glob
31116 4.25% 9.55% 10.89% 16.84%

string match
32496 -4.44% 5.53% 6.94% 13.15%

domain
34400 -10.55% -5.86% 1.49% 8.06%

getfield
34919 -12.22% -7.46% -1.51% 6.67%

nested conditional
37416 -20.25% -15.14% -8.77% -7.15%

Conclusion

Be considerate of the weapons you have at your disposal in the iRules arsenal. Get the solution working, then comb every line and event looking for efficiencies. For example, I had never considered using the domain command to find an empty string. Test everything. Not only will you find efficiencies, you just might learn something as well.

Rapid iRule Removal via Tmsh Script

citizen_elah — Fri, 14 May 2010 10:55:00 GMT

The BIG-IP GUI is pretty slick and there have been massive improvements in function and efficiency since my first exposure in version 4.2. As good as it is, however, some tasks are just better suited to the CLI. Take maintenance, for example. How long will it take you to remove an iRule from a virtual server? A quick attempt from login to removal was about 14 seconds. While already logged in, it took me about nine seconds. So assuming you know exactly what virtuals an iRule needs to be removed from, and that you can stay on top of which you’ve pulled from without duplicating effort, you can see how efficiency flies right out the window if you have 20, or 30, or maybe 300 virtuals to touch, right?

Enter scripting. This scenario is why scripts exist. It takes a mundane manual task and automates it, taking not only less time, but removing the human error element as well (assuming you don’t design the errors into your script!) In this tech tip, I’m going to look at using tmsh scripting to evaluate all the virtual servers on the BIG-IP and if an iRule called “x” is present, remove it.

Why remove iRules? They Rock, right?

Well, yes, they do! But, there are differently purposed iRules deployed. Some are critical to site functionality—remove them, and the site is down hard. Other rules are present for logging, d

ebugging, injecting content, etc. These aren’t necessary, and during times of troubleshooting or resource concerns, could be removed. As indicated above, this is trivial from a standpoint of procedure, but it’s the time involved that counts. So scripting it is. The problem now is what kind of script to go with? The options are a shell script, a perl script, a tmsh script, or an iControl script. The first two are certainly possible, but must utilize raw BIG-IP configuration data and then parse it into objects for manipulation. For the latter two, the data is already presented as objects, so the complexity of the script is reduced dramatically. iControl gets the stage all the time, so I chose tmsh for this job.

A Little Planning Goes a Long Way

Sometimes I get ahead of myself and start scripting immediately. This is more fun, but always results in rewriting something as I’ve certainly made logic errors and assumptions that don’t pan out. So let’s start with the high level approach, and then work our way into code.

Consider this virtual server configuration:

Virtual Server Configuration

ltm virtual tmshtest10 { destination 10.10.20.159:http ip-protocol tcp mask 255.255.255.255 pool att_rtr profiles { http { } stream { } tcp { } } rules { filler1 filler3 filler4 filler5 } }

Each of these keywords and values can be accessed in tmsh as an object. I don’t really care about any of the keywords in a virtual except rules, since it’s only a specific iRule I want to remove. So that’s the first test in the script. After collecting all the vips into a variable, I want to loop through that list and test for the presence of that keyword. If it is not defined, I can move on to the next virtual. Once I’ve verified the keyword is present, however, I next need to see if the iRule I want to remove is defined within the rules keyword. If it is, then I want to take action on this virtual server. Since there is no remove action associated with iRules, I need to test how many iRules are currently applied. If the iRule I want to remove is the only one, I can just modify the virtual server with the rules none attributes. However, if there are more than one, I need to “pop” the specific rule from the list, then re-apply the iRules that will remain. Finally, I’ll wrap all the looping and modification into a transaction, so if there are any failures in the script the desired actions will not be committed. If you are a visual learner, the approach I’ve just described is presented in a workflow diagram in Figure 1.

The Code

Every tmsh script requires the script::run procedure. I start with a conditional to make sure the rule name has been supplied as an argument (the only argument). I then store the rule name and the virtual server configurations in a couple variables, and set a couple other variables for use in the loop. The first two conditionals in the for loop test for presence of rules and then the specific rule. If either conditional fails, that instance of the loop is terminated and the loop moves on to the next virtual. I originally had a little more “logic” in there on both conditionals, but a second set of eyes from Mark Crosland, the author of tmsh, cleaned it up significantly, taking advantage of the zero return from tmsh::get_field_value in the absence of the keyword. The third conditional evaluates the number of rules present. If only one, the script utilizes the none attribute for rules. If more than that, the rule must be removed from the list, which is done with lreplace. The lappend action is just for printing the actions the script took on the active configuration.

tmsh script

proc script::run {} { if { $tmsh::argc != 2 } { puts "A single rule name must be provided" exit } set rulename [lindex $tmsh::argv 1] set rules "" set vips [tmsh::get_config /ltm virtual] set vips_in_play "" tmsh::begin_transaction foreach vip $vips { if { [tmsh::get_field_value $vip "rules" rules] == 0 } { continue } if { [lsearch -exact $rules $rulename] == -1 } { continue } if { [llength $rules] < 2 } { tmsh::modify /ltm virtual [tmsh::get_name $vip] rules none lappend vips_in_play $vip } else { set id [lsearch -exact $rules $rulename] set keepers [lreplace $rules $id $id] tmsh::modify /ltm virtual [tmsh::get_name $vip] rules "{ $keepers }" lappend vips_in_play $vip } } tmsh::commit_transaction puts "The $rulename iRule was removed from the following virtuals: " foreach x $vips_in_play { puts "\t[tmsh::get_name $x]" } }

The Demo

For a quick walkthrough of the process involved in removing an iRule, and a demonstration of the script, check out the video below, which is also viewable in the Monitoring & Management section of the Tutorials Page. The script above is posted in the tmsh wiki.

Automated Gomez Performance Monitoring

Joe — Thu, 13 May 2010 13:44:00 GMT

Gomez provides an on-demand platform that you use to optimize the performance, availability and quality of your Web and mobile applications. It identifies business-impacting issues by testing and measuring Web applications from the “outside-in” — across your users, browsers, mobile devices and geographies — using a global network of 100,000+ locations, including the industry’s only true Last Mile.

F5 and Gomez have partnered to provide a users of both technologies an easy way to integrate the two together. In this article I will focus on the Web Performance Management component of the Gomez Platform. Key features of their Web Performance Management solution include

Rapidly detect and troubleshoot problems that directly impact your customers.

Identify the root cause of issues using real-user passive monitoring.

Ensure web application availability and performance for key user segments.

Optimize web application performance

Improve web experiences across mobile, streaming, and web applications.

Reduce downtime and response times with detailed object-level, page, connection, and host data across multiple browsers.

Make smart technology investments and manage service providers effectively

Quantify the benefits of technology investments such as Content Delivery Networks (CDNs), virtualization, and infrastructure changes.

Ensure service level agreement compliance

All of these features can be possible by simply inserting a small section of client side JavaScript code into your web page requests. Now, for a small site with only a couple of pages, this is fairly simple to do, but for sites like DevCentral with many 1000’s of pages, it is very cumbersome and requires developer support for integration. In addition to that, application level changes typically require a testing cycle before being deployed into production.

The Client-side Script

The Gomez script looks something like this:

1: <SCRIPT LANGUAGE="JavaScript"></SCRIPT>

4: <script src="/js/axfTag.js" type="text/javascript"></script>

The key configuration components are the following variables

acctId – Your Gomez Application id to link client page request to your Gomez Account

pgId – An optional page identifier that allows you to give a page, or set of pages, a unique user friendly name in the reporting.

grpId – An optional group identifier that you can use to further identify your page views. This is highly useful in multi-datacenter deployments when you would like to differentiate the servers serving up your applications.

Enter iRules

With our embedded scripting technology iRules on the BIG-IP, deploying this client script code across all of your application becomes very simple. Since all of your application traffic is traveling through the BIG-IP, it makes perfect sense to “inject” this code at the network layer. That is exactly what I’m going to do with this solution.

Prerequsites

Make sure your Virtual Server is configured as follows:

Ensure that your Virtual Server has the default (or custom) “stream” profile assigned in it’s configuration.

Your Virtual Server must also have an associated HTTP Profile with the “Response Chunking” property set to “Rechunk”.

Handling The Request

You’ll notice I utilized the ability to serve up content directly from the iRule to eliminate the need to deploy the Gomez bootstrap code to a separate server. Also, by implementing it this way instead of injecting the code directly into the page response, allows the browser to cache that data and reduce the size of your page request.

1: when HTTP_REQUEST {

2:

3: set GOMEZ_APP_ID "XXXXXX";

4: set GOMEZ_DEBUG 0;

5: set gws_luri [string tolower ]HTTP::uri[]

6:

7: if { $gws_luri eq "/js/axftag.js" } {

8:

9: # -----------------------------------------------------------------------

10: # Serve up Gomez bootstrap javascript

11: # -----------------------------------------------------------------------

12: HTTP::respond 200 content [string map {LBR \{ RBR \}} {/*Gomez ...}] "Content-Type" "application/x-javascript";

13:

14: } else {

15:

16: # ---------------------------------------------------------------------

17: # Don't allow response data to be chunked

18: # ---------------------------------------------------------------------

19: if { [HTTP::version] eq "1.1" } {

20: # -------------------------------------------------------------------

21: # Force downgrade to HTTP 1.0, but still allow keep-alive connections.

22: # Since HTTP 1.1 is keep-alive by default, and 1.0 isn't,

23: # we need make sure the headers reflect the keep-alive status.

24: # -------------------------------------------------------------------

25: if { [HTTP::header is_keepalive] } {

26: HTTP::header replace "Connection" "Keep-Alive";

27: }

28: }

29: }

30: }

So, for request for the Gomez Bootstrap code (“/js/axftag.js”), the iRule will serve it up. Otherwise, it will allow the request to pass through to the backend server.

Also, you will need to update your GOMEZ_APP_ID variable with your personal Application Id supplied to you from your Gomez account.

Injecting The Client-Side JavaScript

Now that the request has gone through to the backend application, we are ready to handle the response from the servers. We are going to utilize the Stream profile to do the actual content replacement. It’s fast and very easy to configure.

Since we only want to inject the client code in valid web page responses, I’ve added a check for Content-Type “text/html” as well as a success HTTP Response code (200). Then we’ll create a variable for the Gomez client code with the embedded Application Id and set the Stream expression for the Stream Filter to insert it right before the end of the Head element (or Body element if the Head element doesn’t exist in the response).

1: when HTTP_RESPONSE {

2:

3: # -------------------------------------------------------------------------

4: # Stream filter is disabled by default

5: # -------------------------------------------------------------------------

6: STREAM::disable;

7:

8: # -------------------------------------------------------------------------

9: # Only process stream replacement for a valid response and content type is html.

10: # -------------------------------------------------------------------------

11: if { ([HTTP::header Content-Type] starts_with "text/html") && ([HTTP::status] == 200) } {

12:

13: set gomez_client [subst {<SCRIPT LANGUAGE="JavaScript"></SCRIPT>

16: script src="/js/axfTag.js" type="text/javascript"></script>

17: ];

18:

19: if { $::GOMEZ_DEBUG > 1 } { log local0. "Adding Gomez JavaScript"; }

20: if { $::GOMEZ_DEBUG > 2 } { log local0. "$gomez_client"; }

21:

22: # -----------------------------------------------------------------------

23: # Set the stream replacement expression

24: # -----------------------------------------------------------------------

25: set stream_expression "@</\[Hh]\[Ee]\[Aa]\[Dd]>@$gomez_client</head>@@<\[Bb]\[Oo]\[Dd]\[Yy]>@$gomez_client<body>@";

26: STREAM::expression $stream_expression;

27:

28: if { $::GOMEZ_DEBUG > 2 } { log local0. "Current Stream Expression: $stream_expression"; }

29:

30: # -----------------------------------------------------------------------

31: # Enable the stream filter for this request.

32: # -----------------------------------------------------------------------

33: STREAM::enable;

34:

35: } else {

36: if { $::GOMEZ_DEBUG > 1 } { log local0. "Ignoring response with content type [HTTP::header Content-Type], status = [HTTP::status]" }

37: }

38:

I’ve included the STREAM_MATCHED event in the case that the Stream profile finds multiple matches. After the first match, the Stream profile is disabled so that only one replacement occurs.

1: when STREAM_MATCHED {

2: if { $::GOMEZ_DEBUG > 0 } { log local0. "URI: $gws_luri matched [STREAM::match]"; }

3:

4: # -------------------------------------------------------------------------

5: # We've found a match, so disable the stream profile for all subsequent

6: # matches in the response.

7: # -------------------------------------------------------------------------

8: STREAM::disable;

9: }

Conclusion

With this simple iRule, you can now inject the Gomez client side JavaScript code in all of your applications without effecting your application teams. In a future article, I will discuss how you can further customize this iRule by adding support for their Page and Group identifiers so if you want a little more granularity in the reporting, stay tuned.

Download

You can download the full script in the iRules CodeShare under GomezInjection

New Performance Statistics in the F5 Management Pack v2.1.0.43

Blog Author — Thu, 13 May 2010 07:44:17 GMT

We’ve almost doubled the total number of statistics that are gathered in this latest version of the F5 Management Pack with the addition of 100 new http and client SSL profile statistics.

As with all of the statistics, there are several built in Performance reports you can use to graph them over time and drill down into details. You can also generate alerts and recovery actions based on configurable thresholds.

The full list of statistics and monitors comprising the BIG-IP health model can be found on our documentation site.

New HTTP profile statistics (* means only available in BIG-IP 10.1+)

Set-Cookie header insertions (avg/sec)
Successful Responses (avg/sec)
Redirection Responses (avg/sec)
Client Error Responses (avg/sec)
Server Error Responses (avg/sec)
Requests (avg/sec)
Get Requests (avg/sec)
Post Requests (avg/sec)
Version 0.9 Requests (avg/sec)
Version 1.0 Requests (avg/sec)
Version 1.1 Requests (avg/sec)
Version 0.9 Responses (avg/sec)
Version 1.0 Responses (avg/sec)
Version 1.1 Responses (avg/sec)
Max Requests
Responses (00-01 Kilobytes) (avg/sec)
Responses (01-04 Kilobytes) (avg/sec)
Responses (04-16 Kilobytes) (avg/sec)
Responses (16-32 Kilobytes) (avg/sec)
Responses (32-64 Kilobytes) (avg/sec)
Responses (64-128 Kilobytes) (avg/sec)
* Responses (128-512 Kilobytes) (avg/sec)
* Responses (512K-2 Megabytes) (avg/sec)
* Responses (Over 2 Megabytes) (avg/sec)
Idle Connection Splices (avg/sec)
Idle Connections (avg/sec)
Pre-compression Bytes (avg/sec)
Post-compression Bytes (avg/sec)
NULL-compression Bytes (avg/sec)
HTML Pre-compression Bytes (avg/sec)
HTML Post-compression Bytes (avg/sec)
CSS Pre-compression Bytes (avg/sec)
CSS Post-compression Bytes (avg/sec)
JS Pre-compression Bytes (avg/sec)
JS Post-compression Bytes (avg/sec)
XML Pre-compression Bytes (avg/sec)
XML Post-compression Bytes (avg/sec)
SGML Pre-compression Bytes (avg/sec)
SGML Post-compression Bytes (avg/sec)
Plain Pre-compression Bytes (avg/sec)
Plain Post-compression Bytes (avg/sec)
Octet Pre-compression Bytes (avg/sec)
Octet Post-compression Bytes (avg/sec)
Image Pre-compression Bytes (avg/sec)
Image Post-compression Bytes (avg/sec)
Video Pre-compression Bytes (avg/sec)
Video Post-compression Bytes (avg/sec)
Audio Pre-compression Bytes (avg/sec)
Audio Post-compression Bytes (avg/sec)
Other Pre-compression Bytes (avg/sec)
Other Post-compression Bytes (avg/sec)
RAM Cache Hits (avg/sec)
RAM Cache Misses (Cacheable) (avg/sec)
RAM Cache Misses (Uncacheable) (avg/sec)
RAM Cache Hits (Bytes) (avg/sec)
RAM Cache Misses (Bytes) (Cacheable) (avg/sec)
RAM Cache Misses (Bytes) (Uncacheable) (avg/sec)
RAM Cache Size
RAM Cache Count (avg/sec)
RAM Cache Evictions (avg/sec)
* RAM Cache Inter-Stripe Hit Count (avg/sec)
* RAM Cache Inter-Stripe Miss Count (avg/sec)
* RAM Cache Inter-Stripe Hit Byte Count (avg/sec)
* RAM Cache Inter-Stripe Cache Size (Bytes)
* RAM Cache Inter-Stripe Cache Entity Count
* RAM Cache Inter-Stripe Cache Evictions (avg/sec)
* RAM Cache Remote Hit Count (avg/sec)
* RAM Cache Remote Miss Count (avg/sec)
* RAM Cache remote Hit Byte Count (avg/sec)

New Client SSL profile statistics:

Bad Records (avg/sec)
Current Compat-mode Connections
Current Connections
Current Native Connections
Decrypted Bytes In (avg/sec)
Decrypted Bytes Out (avg/sec)
Encrypted Bytes In (avg/sec)
Encrypted Bytes Out (avg/sec)
Fatal Alerts (avg/sec)
Offloaded Connections (avg/sec)
Non-SSL Greeting Failures (avg/sec)
Handshake Failures (avg/sec)
Max Compat-mode Connections
Max Connections
Max Native Connections
Mid-connection Handshakes (avg/sec)
Software Connections (avg/sec)
Assisted Connections (avg/sec)
Invalid Certificates (avg/sec)
Connections Without Certificates (avg/sec)
Valid Certificates (avg/sec)
Unclean Shutdowns (avg/sec)
Records Received (avg/sec)
Records Transmitted (avg/sec)
Session Cache Entries (avg/sec)
Session Cache Hits (avg/sec)
Session Cache Invalidations (avg/sec)
Session Cache Lookups (avg/sec)
Session Cache Overflows (avg/sec)
Compat-mode Connections (avg/sec)
Native Connections (avg/sec)

F5 Networks Management Pack welcomes Microsoft SharePoint 2010 to the world

Blog Author — Thu, 13 May 2010 04:53:41 GMT

To celebrate today’s global launch of Microsoft Office and SharePoint 2010, F5 Networks has popped the cork off a bottle of L7 Champagne with our new version of the Management Pack, version 2.1… which includes Distributed Application Designer support for SharePoint 2010.

With this latest Distributed Application Designer, your data center administrator can quickly and easily deploy Microsoft SharePoint 2010 into their data center. As with all of our Distributed Application Designers, the SP 2010 Designer integrates System Center application and network monitoring, health and alerting, along with automatic discovery and automatic-configuration of existing F5 Networks BIG-IP infrastructure.

We’re particularly excited about the new release of the Microsoft SharePoint 2010 Management Pack for System Center Operations Manager, as it includes a much richer set of health and performance objects, which we can link directly against the F5 Networks Local Traffic Manager for real-time capacity and health-based load balancing.

David Ruddell, an engineer on the F5 MPk development team, has posted a blog with more info and a video tutorial on the SharePoint 2010 integration.

This latest release of the Management Pack also includes over a 100 new network performance statistics and thresholds, covering http, ssl, and device level statistics.

As always, our releases include rolled-up maintenance fixes, in-place upgrade, support via the F5 DevCentral Management Pack Forums.

Look for more blog posts and videos describing and detailing the features in the latest release.

Thank you,
Stephen Fisher
Product Development Manager, F5 Networks Management Pack and Data Center Orchestration

F5 Management Pack v2.1.0.43 Released with App Designer Support for SharePoint 2010

Blog Author — Thu, 13 May 2010 04:48:02 GMT

In conjunction with the release of Microsoft SharePoint 2010, the F5 Management Pack team has released v2.1.0.43 of our Management Pack. Along side this release we are introducing the F5 Application Designer for SharePoint 2010. This new application designer is actually very similar to the SharePoint 2007 Application Designer released last month, but with the SharePoint 2010 Management Pack helping run the show, a whole new level of visibility and capabilities have become available to the user surrounding a SharePoint deployment.

The Application Designer for SharePoint 2010 has a lot of the same objects and features as the SharePoint 2007 designer, but with the new capabilities of the SharePoint Foundation 2010 Management Pack, we are really able to get the application specific objects we require to make the most sense out of the whole picture. On the right is a sample diagram of a single SharePoint Server farm with the basic elements of the designer discovered: F5 LTM Virtual Server, F5 LTM Pool Member, SharePoint Server, Operating System, Website, SharePoint Web Application, and the SQL Content Database for the Website and SharePoint Web Application in the diagram. There is also an object for Hyper-V Virtual Machine if the server discovered is running under Hyper-V and you have the Virtual Machine Manager Management Pack installed.

If you have your SharePoint 2010 deployment up and running and are ready to dive into the wonderful world of F5 Application Designers, then you may find the following fairly useful. First we have the Wiki documentation for all of our App Designers, but if you scroll down, you will find the SharePoint 2010 specific deployment guide. Along with the documentation, I have made up a somewhat lengthy tutorial video to help show you how to get things working, as well as walk you through downloading, installing, and modifying the scripts necessary to successfully run our SharePoint designer. That video can be located here:

Working with the F5 Application Designer for SharePoint 2010

If you have any additional questions or concerns, feel free to visit our F5 Management Pack Group and post in our forums.

ARX: Getting through the Interview Wizard

MichaelAghajanian — Thu, 29 Apr 2010 10:30:00 GMT

So you have gotten your fancy new shiny ARX devices racked and powered as you admire the really nifty red glowing F5 Logo on the front you think "Now what?" The next step is to connect a console cable to the device and run through the interview wizard. Here is where I will step you through the wizard to get your devices ready for an implementation.

Step 1: Console Cable

What's that you ask? It's the Cable that is usually Powder Blue with a DB9 connector on one end and a RJ 45 connector on the other. It's also called a roll over cable ( not sure why).

Step 2: Terminal

You need a program to open a connection through the serial port to the ARX. You can use windows HyperTerm or something that is more user friendly and free like PuTTY Download Here. After you install the program of your choice you will have to open the COM Port the serial cable is attached to and set the correct communication settings. If you have a physical serial port in your laptop ( new models after late 2009 do not anymore) it will be COM1 most likely if you have a USB to Serial it will be a higher number. Mine is COM8.

After you find what COM port the settings are Baud Rate: 9600, Data Bits 8, Parity None, Stop Bits 1.

Next open the session! If you don't see anything hit a key. If that doesn't work check your settings. You should see:

Press <Enter> to start the Switch Configuration Wizard.

After you hit enter sit back with a cup of coffee and enjoy reading the LICENSE AGREEMENT.

Step 3: The Wizard!

2. Enter 'yes' to accept these terms and conditions
   in the format 'yes' or 'no'. # yes

The switch's management port requires an IP address and mask.

3. Enter the management port IP address

   in the format nnn.nnn.nnn.nnn or 'none'. # 172.30.71.10 <- Out of Band IP, needs to be on a different network than any other IP on the box to include the storage, proxy IP etc. or you can type ‘none’ if you do not want to use one.

4. Enter the management port subnet mask

   in the format nnn.nnn.nnn.nnn.(default=255.255.0.0) # 255.255.255.0

The switch's management port requires a gateway IP address.

5. Enter the gateway IP address for the management interface

   in the format nnn.nnn.nnn.nnn or 'none'. # 172.30.71.1

A switch replacement requires additional configuration questions.

6. Are you doing a switch replacement?

   in the format 'yes' or 'no'.(default=no) # no

The crypto-officer is the most privileged user in the system.

7. Enter the crypto-officer username

   in the format text (1-28 characters). # admin <- We use admin

8. Enter the crypto-officer password

   in the format text (6-28 characters). # ****** <- We use acopia

         Confirm the crypto-officer password # ******

A system password is required for access to the master key.

9. Enter a system password

   in the format text (12-28 characters). # ************** <- We use Acopia01Acopia

         Confirm the system password # **************

The master key is used to encrypt critical security parameters.

10. Enter the master key

   in the format base64-encoded key or keyword 'generate'.(default=generate) # <- When installing the first box generate this. When we are installing the second box you will need the key generated from the first box to be pasted in here. To find out the master key that was generated issue this command when you log into the first box you installed enter these commands “switch> en” then “switch# sho master-key”.

Configuration Summary

    Management IP Address    172.30.71.10

    Management IP Mask       255.255.255.0

    Management Gateway       172.30.71.1

    Private IP Subnet        169.254.253.0

    Private IP Mask          255.255.255.192

    Private VLAN             1002

    Chassis GUID             2af5da5e-fe5b-11dc-a435-2bfb40b0a892

    Switch Password          #######

    Switch Master Key        generate

    Crypto-officer Username admin

    Crypto-officer Password #######

Enter 'yes' to load configuration or 'r' to restart #

Repeat this for the second switch. Do not worry about any other configuration. After the switches are configured, if the Out of Band I is configured you can try to ping the gateway and the management ports to make sure they can be accessed. We will configure the HA pair when I remote in.

Enjoy! Any problems don’t hesitate to call us up!

ARX: Common Helpful Troubleshooting CLI Commands

MichaelAghajanian — Thu, 29 Apr 2010 10:00:00 GMT

Show Health

Quick view if there are any active alarms the ARX is sending out that the admin needs to be aware of.

Show Health Time-Skew

Will show the health of the time-skew between ARX's in HA and domain controllers and the backend filers. Great for troubleshooting Kerberos if there is a time skew.

Show cifs-service user-sessions

Shows what CIFS users are connecting to the ARX. user@domain.com are Kerberos connections and domain/user are NTLM

Show redundancy

Show the status of the redundancy and which switch is active. The * is the active switch.

Show redundancy his

Shows the history of the redundancy connection. Great for troubleshooting

Show running-config

Shows the running config

Show global-config

Shows the global config where all the namespaces, volumes, shares etc.

Tail logs syslog

Good for immediate troubleshooting. Shows the last 25 lines of the syslog. You can also see the syslog in the GUI, but the file can get large.

Log format:

utc.uuu‐tz:switch:slot‐proc‐board‐pid:cmp‐ins‐sev‐id:: msg

utc is the time in UTC.

uuu is the millisecond time fraction.

tz is the hours off UTC (+nnnn or -nnnn).

switch is the switch name.

slot-proc-board is the chassis slot, processor number, and board type (ACM, ASM, SCM, or NSM). Use the show processors

pid is the Process ID (PID).

cmp is the log-component name. Refer to Log Components, above.

ins is the process’s instance number.

sev indicates the message severity: DBG (debug) is the least severe, followed by INFO, NOTE, WARN, ERR, and CRIT (critical).

id is the message-catalog ID, a unique ID for each log message. MSGn is an ID for an uncatologued message. The n is the severity of the message, from 7 (debug) down to 2 (critical). Refer to the

msg is the message text itself.

Show Chassis

Shows the status of the hardware. Can also see this in the GUI

Show ntlm-auth-server status

Great for trouble shooting NTLM connections. This will show you if the ARX is connecting to the NTLM secure agent.

Show ron tunnel

Good to see if the RON tunnels are connected to the other switches. When running Shadow Copies to a remote site this is a good command to use to troubleshoot if the copies are not working

Show ip route

Troubleshoot network issues

Show arp

Troubleshoot network issues

Show system tasks

Shows all running process and cpu utilization

Ping ip‐address from slot.processor [count number]

Ping from a specific processor. Great for troubleshooting network issues from specific processors.

Show statistics namespace fastpath

Shows statistics of shares.

BIG-IP LTM and Live Meeting Portal Server

naladar — Tue, 27 Apr 2010 14:34:38 GMT

BIG-IP LTM and Live Meeting Portal Server

I setup Live Meeting Portal Server in a production environment the other day and wanted to share a few things that are not mentioned in Microsoft’s documentation. The BIG-IP portion of this configuration is super easy, but it is understanding how both the application and the BIG-IP work together that can be the hardest part of any deployment.

Prerequisites:

Please consult the Live Meeting Portal Server documentation and ensure that your servers meet all the perquisites before installation. All the examples in this guide are setup so that you will end up with a website at this URL: https://livemeeting.mycompany.com/lmportal. Please feel free to substitute your company’s name for “mycompany”.

IIS Setup:

1. Download the latest version of Office Live Meeting Service Portal. As of 4/20/2010 that can be found here:

http://www.microsoft.com/downloads/details.aspx?FamilyID=429bb528-fd1b-45b7-af2b-cbbf4a8e65ff&displaylang=en

2. Create a basic website in IIS and name it Live Meeting. This empty shell of a website will be used by the Live Meeting installer and will basically be taken over by it after you run through the installation.

3. Create a folder named “Livemeeting” in the directory of your choice. In this example we will use ”E:\web\content\”

4. Double click the lmportal.exe to begin the installation and choose custom when the option appears. Then select the directory you created above so the files will be placed in your normal custom web content location.

5. Remote Desktop (RDP) to the web server and open IIS. DO NOT USE THE IIS CONSOLE ON YOUR LOCAL MACHINE as you will not have access to everything that you need.

6. The screenshots below will help guide you through the configuration of the web site in IIS. Things that do need to be changed:

a. Add 443 to the SSL port and select the unique IP address for the site to use. We will be terminating SSL on the F5 BIG-IP and then re-encrypting before sending it back on to the server.

b. Allow Scripts and Executables under execute permissions. Verify application pool is set to Live Meeting Intranet Portal AppPool.

c. Verify that ASP.NET is set to version 1.1.4.322

d. Under Directory Security, click Edit and make sure there is a check mark on the “Enable anonymous access” and “Integrated Windows authentication” box.

e. Go to the application pool, right click and go to properties. Click the Health tab and uncheck “Enable Rapid-Fail protection”. Not including a screenshot of this one.

7. Navigate to “E:\web\content\Livemeeting\Portal” on the server. Then find the file named “Portal.config”, right-click it and click the Security tab. Click Add and then add the “Network Service” user account and give it full control. You have to do this or you cannot modify the configuration settings from the GUI.

8. Do the same thing listed in step 7 for the “PortalExport” folder located in the directory you should currently be in: “E:\web\content\Livemeeting\Portal”

9. Now you have to import the SSL certificate that you are going to use into IIS website that you just set up. You will need to obtain the .crt file for the SSL certificate and the .key file for that certificate. We terminate our SSL on the BIG-IP so these can both be obtained from there. I will skip the steps regarding purchasing an SSL certificate for a site if you do not already have one. It kind of falls outside the scope of this guide.

10. Use a search engine and search for OpenSSL. You should find their homepage at: http://www.openssl.org/

11. Download OpenSSL and install it on your Local machine. I don’t recommend installing it on the server for a wide variety of reasons. I installed my copy of OpenSSL into “C:\OpenSSL”.

12. Take the .key file and the .crt file and put them into OpenSSL’s “bin” directory. It’s just a folder inside of your OpenSSL folder called bin.

13. Open a command line and change directory over to C:\OpenSSL\bin. The example I am going to provide is for a fictitious company named “MyCompany” that is using a wildcard ssl certificate on a few of their websites.

14. Then type in the following command:

This all needs to be on one line. Spaces are ok, but no carriage returns or anything like that. This command is modeled after this example for future reference:

openssl pkcs12 -export -out certificate.pfx -inkey privateKey.key -in certificate.crt -certfile CACert.crt

certificate.pfx = the name of the new .pfx file you want to create

privateKey.key = the private key you got off of the F5 BIG-IP

certificate.crt = the crt file that you got off the F5 BIG-IP

CACert.crt = the crt file that you got off the F5 BIG-IP

15. After you type the command and hit enter, you will be prompted for a password. You can use any password that you like but you will need to remember it because IIS asks you for the same password when you go to import it.

16. OpenSSL will compile a new .pfx file for you in the C:/OpenSSL/bin directory. Take that SSL certificate and copy it over to your web server.

17. RDP over to the server and open IIS. Again here is the disclaimer, DO NOT USE THE IIS CONSOLE ON YOUR LOCAL MACHINE. Right-click on the Live Meeting web site that you created and click on the Directory Security tab. Under “Secure Communications”, click the “Server Certificate…” button.

18. Click Next and then click the “Import a certificate from a .pfx file” radio button and click next. Browse to the .pfx file that you uploaded to the web server. Click next and enter your password information that you used when you created the certificate. Then finish clicking through the wizard. Then restart IIS on the server and delete the certificate off of your local machine. This completes the IIS setup. Now move on to the Live Meeting Portal setup.

Live Meeting Portal Setup

19. Navigate to the URL:

https://livemeeting.mycompany.com/LMPortal/settings.aspx

Where livemeeting.mycompany.com is the name of the website you setup. The screen will look like the one shown on the next page. This is the Settings-Portal Configuration page. You will want to use the following settings which are also pictured in the screenshot on the next page.

Conference Center URL = https://www.livemeeting.com/cc/mycompany

Conference Center Administrator

User Id =

Password =

Email address for escalation =

Enabled Portal Services = Check the Account Create, Account Login, Account Update and Web Method Calls

Ticket Timeout = 300 Seconds

Directory Service Parameters = AccountNamePolicy=LogonUsername

20. Then click Save. If you receive an error at this point, refer back to step #7.

21. Click on the Roles link on the left side of the page. This will take you to the Roles-Portal Configuration page. Under “Live Meeting Administrators” add the users who will be the Live Meeting Administrators. Use domain\name format. IE: mydomain\username

22. Then under the “Live Meeting Organizers” settings I recommend adding the “Domain Users” from the varies domains on your network. So if you have three domains on you network named ABC, 123 and XYZ you would list ABC\Domain Users, 123\Domain Users and XYZ\Domain Users.

23. Then click the “Export Configurations Settings” link on the left hand side of the page. This is not really labeled right because what it actually does is back up your configuration. If you mess something up in the running configuration, simply click on the “Import Configuration Settings” to restore the last configuration that you exported.

24. Then click on the “Events” link on the left side of the page. Change the log file directory to a directory that you want to have all the logs written into. In this example I chose the E: drive of the server I was working on. Whether you create a new one or use an existing one you must make sure that the “Network Service” account has permissions on that folder to Read, Write and Modify. Otherwise you will receive a nasty .NET error when you go to save the changes you just made. Click Save.

Live Meeting Portal Server BIG-IP LTM Setup

The BIG-IP LTM set up for this is very easy to configure. You will need to create nodes for each of your web servers, assign them to a pool named “Live_Meeting_Pool” and then create a Virtual Server for the application. I named my virtual server “Live Meeting” in the example pictured below. You may need to customize it to match your environment, but the basic settings are:

Service Port: 443

Type: Standard

Protocol: TCP

Protocol Profile (Client): tcp

HTTP Profile: http

SSL Profile (Client): wildcard

SSL Profile (Server): serverssl

I also assigned the Live_Meeting_Pool to the Virtual Server, set the Default Persistence Profile to “Cookie” and Fallback Persistence Profile to source address.

-naladar

www.TheF5Guy.com

Holistic Monitoring and Auto Configuration using the F5 Management Pack and IIS 7

Blog Author — Fri, 23 Apr 2010 03:29:23 GMT

I think we're all aware that the overall health of anything but a very simple application depends on all of the various components’ health, such as the web server, the overall operating system, the BIG-IP pool members, and potentially the virtual machine that the web server runs on. The traditional way of taking all of these components into account is to periodically test the end result: Is the site responding to pings? Is anyone complaining?

The Application Designer for IIS 7 is an extension to the F5 Management Pack that enables a more comprehensive, holistic view of your application's health in Operations Manager:

In the diagram above (annotations in red text), each BIG-IP pool member is paired with it's related components, and together represent an 'Application Instance'. The health of the App Instance by default represents the worst health of any of it's member components. In turn, the overall application's health depends on at least one of the App Instances being healthy as well as the health of the virtual server.

This has some great practical uses beyond just the pretty diagram. For example:

Your BIG-IP pool member may healthy because the HTTP monitor on the BIG-IP is succeeding, but one of the other components may have detected a serious problem such as dangerously low memory or compromised security. In this case, the app instance will go into a critical state and can be configured to fire an alert or take remedial action such as disabling the BIG-IP pool member.

If the BIG-IP’s monitor indicates that the web site is down, many of possibly culprits of the outage are quickly in view for root cause analysis.

Statistics from the various components are all available and can be aggregated to gain a much better idea of how your application is responding and whether it needs to be scaled up

Monitoring on the BIG-IP can be replaced by the monitoring included in this pack, reducing load on the BIG-IP

This application designer has two primary uses cases:

BIG-IP Auto-Configuration: You have the option of creating applications where the BIG-IP has not yet been configured with any pool members, pools, or virtual servers. Here is an example of a newly created application that targets a BIG-IP whose LTM configuration is blank:

As you can see, there is no virtual server and there are no pool members shown. The application and the applications instances are red, in this case meaning that the BIG-IP objects need to be configured. Here are the related alerts:

Tasks are available for each of these alerts that create the VIP, pool, and pool members on the BIG-IP according to best practices for IIS websites including configuring profiles for optimal performance.

Application Monitoring: If your BIG-IP is already configured or you would like to manually configure it, you can opt to simply discover your existing configuration. When you discover an existing configuration, the diagram looks like the first screenshot in this post right out of the box.

I hope that you’ll find this extension brings the dream of a “single pane of glass” closer to a reality. Here are some resources on this pack – watch for more to emerge in the near future:

Download the extension pack

Video Tutorials:

App Designer for IIS Part I: Auto-Configuring a BIG-IP

App Designer for IIS Part II: Discovering an Existing App

App Designer for IIS Part III: Modifying Mapping Scripts

F5 Management Pack Application Designer Wiki Documentation

Building a Microsoft SharePoint Server Application with the F5 Networks Management Pack

Blog Author — Fri, 23 Apr 2010 01:55:30 GMT

Shipping along side the F5 Application Designer Packs, there are two typical application extensions as well. One of these happens to be targeted at Microsoft SharePoint Server 2007. With this extension pack, it is easy to deploy and monitor your entire SharePoint application including the important pieces of the Big-IP configuration as well. This Application Designer profile brings together the key components of your SharePoint deployment, including the SharePoint Server objects, SharePoint Application objects, associated websites as well as any Virtual Machines or SQL Server databases you may be using.

Want more? You got it! Not only will this extension pack help you detect and monitor your SharePoint application, it will also walk you through configuring of your Big-IP from scratch. This even includes creating all of the SharePoint specific profiles, LTM Virtual Server, Pool and Pool Members settings designed to work with your SharePoint deployment. With minimal effort, you can have your SharePoint app up and running in no time. The configuration that we deploy for you is based on the Big-IP v9.x and Microsoft Office SharePoint 2007 Deployment Guide. This guide and others can be found here: F5 Networks Deployment Guides. Keep in mind that the configuration we deploy is the very basic configuration to get you up and running. If you wish to tweak the settings to use your Web Accelerator or other Big-IP Modules, you will have to finish those steps manually through the devices Configuration Utility.

Below are three videos to demonstrate how easy it can be to discover and configure the F5 SharePoint Application Designer.

Demonstration of the F5 Application Designer SharePoint 2007 Pack

Demonstration of the F5 Application Designer SharePoint 2007 Pack With No Big-IP Configuration

How to Setup and Configure the F5 Application Designer SharePoint 2007 Pack

For further information, please visit our Wiki documentation from the following link: F5 Management Pack Application Designer Wiki Documentation

Announcing the 2.0 Wave of the F5 Management Pack and Data Center Orchestration

Blog Author — Mon, 19 Apr 2010 03:42:08 GMT

I’m excited to announce the next wave of the F5 Management Pack and related Data Center Orchestration technologies. Some of the scenarios that we are enabling in this release include It’s no coincidence that this week is also the annual Microsoft Management Summit, currently being held in Las Vegas, Nevada.

New and Updated F5 Management Pack downloads available in the 2.0 release include:

Core F5 Management Pack (x86 and x64): Improved support for GTM Virtual Server health, more graceful and transparent handling of discovery and configuration update errors, centralized Orchestration logging configuration.

PRO-Enabled F5 Management Pack for System Center Virtual Machine Manager: Improved PRO Tip diagnostics and recovery information

Network Application Designers

Tools and views that enable Data Center Administrators to quickly and easily model, configure, and optimize their network applications. The Designers provide alerts, tasks, and views for automatically discovering application servers and one-step configuring of the BIG-IP tailored to each application (e.g. Microsoft SharePoint 2007, Microsoft IIS 2008, etc.).

With the newly integrated network application designers, the F5 Management Pack enables significant improvements to application-based load balancing. For example, your BIG-IP LTM will now receive direct diagnostic and performance information from the F5 MPk about application availability, failure analysis, and application server performance.

For example, your Data Center Administrator will now be able to reprovision Application Server hardware and software resources, and the BIG-IP LTM will automatically be fed the updated server capacity information to rebalance traffic appropriately. In virtualized and heterogenous hardware environments, this will save configuration costs, streamline process communication, and reduce traffic bottleneck issues due to misaligned capacity.

With the Designer’s automatic discovery and configuration framework, your Data Center Administrator can add and remove application servers from service pools, and the BIG-IP can be quickly, directly, and more easily configured.

Finally, with a single pane of glass to model and view your service front end and the application server and network objects, you can more directly identify issues on a per instance basis, separate server from network issues, while monitoring, aligning and synchronizing all of the health and configuration for each instance.

Throughout the week, the F5 Management Pack and Data Center Orchestration team will be posting more blogs and videos explaining all of the new features, easy how-to-videos on enabling and configuring the new features, and updates to the F5 DevCentral Management Pack wiki.

We’re not only excited to releasing a tremendous amount of innovative technology bridging the application and network. We’re really excited to help you better leverage your existing application server and BIG-IP investments, simplify your administration tasks, and save you time and resources.

Thank you,

Stephen Fisher

Product Development Manager, F5 Management Pack & Data Center Orchestration Technologies

F5 Networks

Working with High-Availability Pairs and the F5 Networks Management Pack

Blog Author — Fri, 16 Apr 2010 04:19:37 GMT

With our latest release of the F5 Networks Management Pack, health visibility and configuration of failover conditions has never been better. The release includes updates to the existing failover health model which not only allows you to see what your failover peer is, but also allows for synchronization of configuration between the devices and the configuration of software triggered failover. What does this mean for you? Well now you have the ability to configure when a device fails over strictly based on the health of the device in question. Not only that, but with the availability of the peer address property on the device, users now have the flexibility to write their own custom scripts, alerts and failover scenarios based on the health of the failover peer as well.

Health-based failover gives users the ability to choose when a device will automatically fail over to its peer based on a specifically chosen statistic and threshold. For instance, say the hard drive on your Big-IP is filling up do to enormous log sizes. Place a threshold value at 15 GB and when the drive space usage breaches the 15 GB mark, the device will automatically fail over to its peer.

Another new feature in this release is better config-sync handling. By default, if device redundancy is detected, any configuration changes made on one device will set a warning flag on the peer device, letting you know that the devices may be out of sync. You can manually reset the state of this monitor or run one of the Synchronize Configuration tasks. Keep in mind that this only applies to changes made through Operations Manager, not changes made through the built-in Configuration Utility on the Big-IP itself. This monitor can be turned off if you wish not to receive the warning state for out-of-sync devices.

Here are some video links to demonstrate the new features:

Demonstration of Software Triggered Failover

Demonstration of Object Level Sync

Along with videos showing how to set up and configure the new features to work in your deployment:

Configuring the F5 Management Pack for Software Triggered Failover

Configuring the F5 Management Pack for Object Level Sync

Basic configuration and documentation of these features is also available on our Management Pack Wiki page.

iControl Development with BIG-IP LTM VE - Part 2

hamish@travellingkiwi.com — Mon, 12 Apr 2010 02:00:00 GMT

So now that we've got a vLTM installed and running, what can we do with it... There's a lot of examples out there of perl, python and Java. But not a lot of C...

I've also got an iPhone... Sadly we can't use my favourite SOAP programming language on the iPhone. In case you're wondering that's perl. Why perl? Well, mainly I'm a bit lazy, and perl does everything for you... Obviously you're more productive when that's the case, and the SOAP::Lite module has spoiled me for years I'm afraid, but it's time to do some hacking (And in the early days it gets quite ugly) in what I'd consider to be a rather obscure variant of C. Yes, it's Objective-C...

Like I've mentioned, the network map is a cool feature. One page showing your BigIP status. Virtual Servers, Default Pools, iRules, PoolMembers etc... Wouldn't that be nice on your iPhone? (Or better yet, your iPad). Obviously some things need to be altered a bit, the iPhone screen isn't exactly the best for viewing a lot of information at once, but it's nice to be able to pull it out of your pocket and quickly check the status when in a meeting...

But this isn't about iPhone programming. It's about getting your iPhone to talk to BigIP. And for that we use iControl. The F5 SOAP API to monitor and control an F5 appliance.

Opening a connection

First off. How to talk to the F5... Well, iControl is a SOAP interface. And this is based upon HTTP. Basically we open an HTTPS connection to the iControl server and send it a few pages of XML that tell it what we want it to do. Then it sends us several more pages of XML to tell us the 4 or so lines we wanted to know... (In case you get the opinion I think SOAP and XML are a wee bit excessive, you're probably right).

In objective-C we do something like this

-(XMLNode *)queryF5:(NSString *)operation namespace:(NSString *)namespace params:(NSMutableDictionary *)params {

NSString *targetURL = [NSString stringWithFormat: @"https://%@:%@@%@:443/iControl/iControlPortal.cgi", adminUser, adminPass, managementHost];

NSMutableURLRequest *req = [SOAPRequest requestForOperation:operation

atURL:]NSURL URLWithString: targetURL[

withNamespace:namespace

withParameters:params];

[req setTimeoutInterval:10];

NSURLResponse *resp;

NSData *respData = [NSURLConnection sendSynchronousRequest:req returningResponse:&resp error:nil];

NSString *debugString = []NSString alloc[ initWithData:respData encoding:NSUTF8StringEncoding];

if (respData == nil) {

// Error while sending SOAP request");

return nil;

}

// Successfully sent SOAP request, getting response

XMLNode *response = [SOAPResponse responseWi

return response;

}

What does this do? Well, ignoring the fact that some of the variables are seemingly appearing out of thin air (They're not really, they are class member variables. e.g. the adminUser, adminPass etc). First we setup the URL we want to talk to. That's the string

"https://%@:%@@%@:443/iControl/iControlPortal.cgi"

and we replace all those %@'s with the adminUser, adminPassword and ManagementHost respectively. I'll leave the discussion of what those variables actually are to their descriptive names. But basically we're accessing a URL using (semi) standard URL format. Note that the semi-standard nature of this format is due to the fact that the user:password@host is not actually a part of the HTTP standards. It's an extension made popular by usage only. And luckily the F5's recognise and respond to it.

Even though I said this wasn't about iPhone programming, in iPhone world, there is one extra piece of work we have to do. By default an F5 uses a self-signed cert for the HTTPS connection we're using for iControl. And by default an iPhone the iPhone wants to verify the cert trust. Which of course it can't. So we need to over-ride the check. That's done with a piece of code I found on the net...

@implementation NSURLRequest(DataController)

+ (BOOL)allowsAnyHTTPSCertificateForHost:(NSString *)host

{

return YES; // Should probably return YES only for a specific host

}

@end

Which over-rides the certificate check for the NSURLRequest class we're using to make a SOAP request.

Next Time

Well. Next time I'll show an actual request and response. And the work that goes into decoding an XML response from a SOAP server. I'd doit in perl, but that's so easy, it's only worth a few lines of code and saying go for it... Doing it in C (Or objective-C in this case) is a fairly surprising piece of work.

Other Random Notes

Static IP's... Normally you'd run an LTM with either a full-blown DHCP/DDNS architecture, or with a static IP. Most likely a static IP. However in the vLTM world, we're presented with a DHCP configuration on the vLTM and an environment (IN VMWare Fusion) that doesn't really do DDNS, and has a nasty habit of allocating a new IP everytime you suspend your laptop... There are plenty of reasons that I like to run static IP's on my LTM's, but vLTM seems remarkably resistant to allowing this (It may be a bug?). The result is to configure a static IP I had to resort to the command line config utility... I haven't used that since v4...

]

BIG-IP Logging and Reporting Toolkit – part four

Colin — Tue, 06 Apr 2010 15:45:00 GMT

So far we’ve covered the initial problem, the players involved and one in-depth analysis of one of the options (splunk). Next let’s dig into Q1labs’ Qradar offering. The first thing you’ll need to do, just like last time, is make sure your BIG-IP to pass syslog traffic off the box. Here’s a simple example of how you can get that done in your config file. These are the same as last time, so nothing shockingly new here, though this bit is important.

- Bigip v9

syslog {

remote server 10.10.200.31

}

- Bigip v10

syslog {

remote server {

   qradar {

      host 10.11.100.31

    }

}

}

This will send all syslog messages from the BIG-IP to the QRadar system; both BIG-IP system messages and any messages from iRules. If you’re interested in having iRules log to the QRadar system directly you can use the HSL statements or the log statements with a destination host defined. Ex) RULE_INIT has set ::QRadarHost “10.10.200.31” and then in the iRules event you’re interested in you assemble $log_message and then sent it to the log with log $::QRadarHost $log_message . A good practice would be to also record it locally on something like local0 incase the message doesn’t make it to the QRadar system. In my testing I used a single QRadar system running the log collector and event processor. If you’re using a more sophisticated deployment you’ll need to use the Deployment Manager to ensure that the QRadar log collectors are forwarding messages onto the Event Processor you’re going to work with. My QRadar system was already setup to receive syslog messages on port 514, so there wasn’t anything more to do to get messages flowing. The key to working with QRadar is defining regular expressions to extract the message data you’re interested in – once you have that done most things are done using the same process. In this section I’ll walk through all the tasks needed to extract custom data through build a report for the w3c case. Then I’ll show a summary using NEDS and dashboard data.

Here are my regexes for QRadar for w3c, NEDS and the dashboard data script:

message source

attribute name

regex

capture group

sample message

dashboard script

Compression Deflate uses

deflate\.out\.uses='(\d+)'

1

in dc post

dashboard script

Compression LZO uses

lzo\.out\.uses='(\d+)'

1

in dc post

dashboard script

Compression Null uses

null\.out\.uses='(\d+)'

1

in dc post

dashboard script

Dashboard-messageType

message_type='(.+?)'

1

in dc post

dashboard script

Dashboard-reportingSystem

HostName='(.+?)'

1

in dc post

dashboard script

Dashboard-routingEnabled

routing='(.+?)'

1

in dc post

NEDS iRule

NEDSv1-Flow-clientside-http

"(neds\.f5\.conn\.start\.v1)",(\"[\w\.resp\.v1]+\"\,)+\"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}\-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}@\d+\.\d+)\"

1

in NEDS Spec

NEDS iRule

NEDSv1-clientIPaddress

"(neds\.f5\.conn\.start\.v1","[\w\.]+)","(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})

2

in NEDS Spec

NEDS iRule

NEDSv1-clientPort

"(neds\.f5\.conn\.start\.v1","[\w\.]+)","(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:)(\d{1,5})

3

in NEDS Spec

NEDS iRule

NEDSv1-clientCloseBytesIn

(neds\.f5\.conn\.end\.v1)","([\w\.]+)","(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}@\d+\.\d+)",(\d+.\d+),(\d+),(\d+),(\d+)

7

in NEDS Spec

NEDS iRule

NEDSv1-clientCloseBytesOut

(neds\.f5\.conn\.end\.v1)","([\w\.]+)","(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}@\d+\.\d+)",(\d+.\d+),(\d+),(\d+),(\d+),(\d+)

8

in NEDS Spec

NEDS iRule

NEDSv1-clientClosePktsIn

(neds\.f5\.conn\.end\.v1)","([\w\.]+)","(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}@\d+\.\d+)",(\d+.\d+),(\d+),

5

in NEDS Spec

NEDS iRule

NEDSv1-clientClosePktsOut

(neds\.f5\.conn\.end\.v1)","([\w\.]+)","(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}@\d+\.\d+)",(\d+.\d+),(\d+),(\d+)

6

in NEDS Spec

NEDS iRule

NEDSv1-clientCloseTimestamp

(neds\.f5\.conn\.end\.v1)","([\w\.]+)","(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}@\d+\.\d+)",(\d+.\d+),

4

in NEDS Spec

NEDS iRule

NEDSv1-clientConnectionIngressVlan

(neds[\w\.]+start\.v1\",\"[\w\.]+",")(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}@\d+\.\d+)\",(\d+\.\d+)\,"(\w+)"

4

in NEDS Spec

NEDS iRule

NEDSv1-clientConnectionPolicyName

(neds[\w\.]+start\.v1\",\"[\w\.]+",")(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}@\d+\.\d+)\",(\d+\.\d+)\,"(\w+)"\,(\d+),(\d+),(\d+),\"([\w\.]+)\"

8

in NEDS Spec

NEDS iRule

NEDSv1-clientConnectionStartTimestamp

(neds[\w\.]+start\.v1\",\"[\w\.]+",")(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}@\d+\.\d+)\",(\d+\.\d+)

3

in NEDS Spec

NEDS iRule

NEDSv1-clientIPProtocol

(neds[\w\.]+start\.v1\",\"[\w\.]+",")(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}@\d+\.\d+)\",(\d+\.\d+)\,"(\w+)",(\d+),

5

in NEDS Spec

NEDS iRule

NEDSv1-httpRequestHost

(neds\.f5\.http\.req\.v1)",("[\w\.\"\:\-\@]+)","([\w\.\:\-\@]+)",(\d+\.\d+,\d+),"([\w\.\_\-]+)

5

in NEDS Spec

NEDS iRule

NEDSv1-httpRequestServerPort

(neds\.f5\.http\.resp\.v1)","([\w\.]+)","([\d\.:]+)-([\d\.]+):(\d{1,5})

5

in NEDS Spec

NEDS iRule

NEDSv1-httpRequestTCPReplyNumber

(neds\.f5\.http\.req\.v1)",("[\w\.\"\:\-\@]+)","([\w\.\:\-\@]+)",(\d+\.\d+),(\d+)

5

in NEDS Spec

NEDS iRule

NEDSv1-httpRequestUserAgent

(neds\.f5\.http\.req\.v1)",("[\w\.\"\,\:\-\@]+)","([\w/\._\%\@]+)",("[\w\@\.]*?"),"([\w/\.\s(;\-\:\)]+)

5

in NEDS Spec

NEDS iRule

NEDSv1-httpResponseContentLength

(neds\.f5\.http\.resp\.v1)","([\w\."\,\:\-\@]+)","([\w\/\;\s\=\-]+)","(\d+)

4

in NEDS Spec

NEDS iRule

NEDSv1-httpResponseContentType

(neds\.f5\.http\.resp\.v1)","([\w\."\,\:\-\@]+)","([\w\/\;\s\=\-]+)

3

in NEDS Spec

NEDS iRule

NEDSv1-httpResponseLBTarget

(neds\.f5\.http\.resp\.v1)","([\w\.\,:\-@/;\s\="]+),"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5})"

3

in NEDS Spec

NEDS iRule

NEDSv1-reportingSystem

(\"neds.+[\w]\.v1\"),\"([\w.]+)\"

2

in NEDS Spec

NEDS iRule

NEDSv1-responseHTTPContentLength

(neds[\w\.]+resp\.v1\",\"[\w\.]+",")(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}@\d+\.\d+)\",(\d+\.\d+),(\d+),"(\d{3})","([\w\/]+)","(\d+)"

7

in NEDS Spec

NEDS iRule

NEDSv1-responseHTTPServerResponseCode

(neds[\w\.]+resp\.v1\",\"[\w\.]+",")(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}@\d+\.\d+)\",(\d+\.\d+),(\d+),"(\d{3})"

5

in NEDS Spec

w3c iRule

W3C Client IP address

client_ip=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})

1

in dc post

w3c iRule

W3C Client Port

client_port=(\d{1,5})

1

in dc post

w3c iRule

W3C Client username

username=([\w]+)

1

in dc post

w3c iRule

W3C Content Length

content_length=(\d+)

1

in dc post

w3c iRule

W3C HTTP Request

request="(.*)"\ss

1

in dc post

w3c iRule

W3C HTTP version

HTTP/(\d\.\d)"

1

in dc post

w3c iRule

W3C Host header

host=(.+?)

1

in dc post

w3c iRule

W3C Member server

lb_server=(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}:\d{1,5})

1

in dc post

w3c iRule

W3C Server Response Code

server_status=(\d{3})

1

in dc post

w3c iRule

W3C User Agent

user_agent="(.*)"

1

in dc post

w3c iRule

W3C VIrtual Server name

virtual=(.*?)\s

1

in dc post

w3c iRule

w3c Server Port

lb_server=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:(\d{1,5})

1

in dc post

w3c iRule

w3c referer

referer=([\w\./\:\-]+)

1

in dc post

w3c iRule

w3c resp_time

resp_time=(\d+)

1

in dc post

W3C offload case

Now that BIG-IP is setup to send messages to the QRadar system double check to see that you’ve installed the w3c-client-logging iRule on a vip and we’ll see what it looks like when everything is put together. Login to your QRadar management console and navigate to the Events tab. You should see events streaming into QRadar if everything is configured correctly. If you can’t find what you’re looking for in the normalized events view change it to the raw view – I also opted to have my console autorefresh every minute.

There last message is the one I’m looking for. If you find a similar message and double click on it we can start to extract data from the message and build up some searches and reports. I’ve not fully customized my QRadar deployment, so I’m ignoring the fact that the Log Source for the iRule messages has been identified by the system’s FastIronDsm. After your screen is showing the Event Viewer click on the Extract Property button in the button bar. This will launch the Custom Event Property Definition tool, which will allow you to categorize event elements and write the regex for extracting the information you’re interested in. Right now, I’m interested in the HTTP server status codes.

For your custom field extractions, here’s a sample message from the W3C iRule:

Feb 9 14:23:21 tmm tmm[5088]: Rule w3c-client-logging : virtual=www.f5demo.com_http client_ip=65.197.145.92 client_port=37227 lb_server=10.10.200.1:80 host=www.f5demo.com username= request="GET /compression HTTP/1.0" server_status=301 content_length=322 resp_time=1 user_agent="check_http/1.96 (nagios-plugins 1.4.5)[" referer=

The regular expressions for key value pairings are pretty easy to create. In this window you can see that the regex has located the item in the log message I’m interested in – it’s highlighted in yellow. Save the regex extraction and you’ll be returned to the Event Viewer and look for the new property listed on the page.

Now the attribute shows up on the Event list page, down at the bottom. I’ve already entered several regular expressions for the NEDS data, and since I’ve assigned them all to the same Device Support Module (DSM) they’re showing up on this page; and this isn’t a NEDS message so they’re not applicable to this stream. With the extraction we just assigned to this DSM and log source we can return to the Event List and build a search. After the search is built it can be used to filter the events list, and we can build a report from it. In the Event Viewer Click on the Search button and select the New Event Search option.

I’ve also added extractions to my system for response time, and member server. This helps to further illustrate what’s happening in my environment – I can see what hosts are sending which response codes and get a rough idea on what the client and server performance is. Pick the fields you’re interested in including in the search and click on the ‘Filter’ button. QRadar composes the search and saves it to the list of defined searches. I could also add a regex to extract the BIG-IP name from the message and group by that attribute to get an idea of what’s happening across the various BIG-IPs in my environment. Now the search runs – and I find that in the last 6 hours there have been 147 HTTP 304 response codes recorded by the system. Here’s my search result:

I see that in the last 6 hours there have been 147 HTTP 304’s recorded by the system. To turn this search into a report or make it available to the dashboard, click on the Save Criteria button in the toolbar. I’ve found that it helps to group searches together, so I’ve created a group called BIG-IP for all my BIG-IP related searches. For this search to appear on your dashboard you’ll also need to click the “Add item…” button on the dashboard and locate your search.

To generate the report, click the Reports tab and find the Actions dropdown in the tool bar – select Create. I’m building a manual report for this step. My report uses a single frame and Events/Logs as the information source.

And here’s my report:

Accounting for the date format, there were a lot of 304’s returned to clients on March 5^th and I probably have data missing from March 10^th onwards because my BIG-IP was sending log messages somewhere else.

NEDS case

While the w3c offload case used an iRule with key/value tuples NEDS uses a comma delimited string to convey information in the message. I spent some time with the specification and wrote several regular expressions to extract the data. The process is identical to what’s outlined in the w3c case, so I’ll save the screen real estate and skip the screen shots of the process. You can find my regular expressions here – I’m fairly new to regular expressions, so I’m sure that there are improvements that can be made to make mine more efficient/maintainable. After defining the custom attributes here’s what I get when viewing a NEDS message in the Event Viewer. To syslog, A NEDS message looks like

Mar 30 10:44:59 tmm tmm]5088[: Rule networkEventDataStream <HTTP_RESPONSE>: "neds.f5.http.resp.v1","bigip9.f5demo.com","65.197.145.92:42709-65.197.145.93:80@1269971099.951082",1269971099.952527,1,"301","text/html; charset=iso-8859-1","322","10.10.200.1:80","65.197.145.92:42709-10.10.200.1:80"

Here’s the detailed view using the regular expressions for http response, client close, http request and client accepted messages.

HTTP Response

Client close

HTTP Request

Client Accepted

Here’s a select result of a search I composed for the connection close data. The clientCloseBytesOut is not N/A filters out all the non-client-close NEDS messages.

The process to generate a report for NEDS data mirrors the process for the W3C case:

1. Save the search

2. Create a report template

3. Add the search to the report template

4. Save the template

5. Run the report

Dashboard data case

To get the dashboard data streaming into QRadar, I had to modify my base script to send the messages via syslog, instead of just printing the string. In addition to the QRadar and BIG-IP systems, you’ll need another host with the requisite Perl modules installed to relay the data from the BIG-IP to the QRadar.

Here’s the script:

http://devcentral.f5.com/downloads/techtips/dashboard-syslog.pl

To use it you’ll need to:

1. On line 66 configure the username the script should use to access the dashboard interface

2. On line 67 configure the password for the username from step 1

3. On line 48 configure the IP address or name that for the QRadar event collector

4. Schedule the script to run periodically on your relay host – cron would do nicely.

Once you’ve got the data into QRadar you’ll need to:

1. Find the endpoint-isession-stat data log message and write your regular expressions for the data you’re interested in

2. Find the remote endpoint log message you’re interested in and write your regular expressions for the data you’re interested in

3. Build and save your searches

4. Build and run your report template

Here’s a sample endpoint-isession-stat data log message in syslog:

Mar 15 17:05:46 127.0.0.1 10.11.100.73: device_timestamp='Wed Mar 15 00:05:46 2010 GMT' HostName='bigip3900c.demo.f5demo.com' version.version='10.1.0' message_type='endpoint_isession_stat' name='_tunnel_ctrl_10.20.50.103' peer_ref='00:00:00:00:00:00:00:00:00:00:ff:ff:0a:14:32:67' null.in.uses='292670' null.in.errors='0' null.in.bytes_opt='31371493' null.in.bytes_raw='29030133' null.out.uses='292670' null.out.errors='0' null.out.bytes_opt='31371493' null.out.bytes_raw='29030133' lzo.in.uses='1250' lzo.in.errors='0' lzo.in.bytes_opt='139167' lzo.in.bytes_raw='124178' lzo.out.uses='1250' lzo.out.errors='0' lzo.out.bytes_opt='139166' lzo.out.bytes_raw='124177' deflate.in.uses='0' deflate.in.errors='0' deflate.in.bytes_opt='0' deflate.in.bytes_raw='0' deflate.out.uses='0' deflate.out.errors='0' deflate.out.bytes_opt='0' deflate.out.bytes_raw='0' dedup.in.uses='0' dedup.in.errors='0' dedup.in.bytes_opt='0' dedup.in.bytes_raw='0' dedup.out.uses='0' dedup.out.errors='0' dedup.out.bytes_opt='0' dedup.out.bytes_raw='0' dedup_in.hit_bytes='0' dedup_in.hits='0' dedup_in.hit_hist.bucket_1k='0' dedup_in.hit_hist.bucket_2k='0' dedup_in.hit_hist.bucket_4k='0' dedup_in.hit_hist.bucket_8k='0' dedup_in.hit_hist.bucket_16k='0' dedup_in.hit_hist.bucket_32k='0' dedup_in.hit_hist.bucket_64k='0' dedup_in.hit_hist.bucket_128k='0' dedup_in.hit_hist.bucket_256k='0' dedup_in.hit_hist.bucket_512k='0' dedup_in.hit_hist.bucket_1m='0' dedup_in.hit_hist.bucket_large='0' dedup_in.miss_bytes='0' dedup_in.misses='0' dedup_in.miss_hist.bucket_1k='0' dedup_in.miss_hist.bucket_2k='0' dedup_in.miss_hist.bucket_4k='0' dedup_in.miss_hist.bucket_8k='0' dedup_in.miss_hist.bucket_16k='0' dedup_in.miss_hist.bucket_32k='0' dedup_in.miss_hist.bucket_64k='0' dedup_in.miss_hist.bucket_128k='0' dedup_in.miss_hist.bucket_256k='0' dedup_in.miss_hist.bucket_512k='0' dedup_in.miss_hist.bucket_1m='0' dedup_in.miss_hist.bucket_large='0' dedup_out.hit_bytes='0' dedup_out.hits='0' dedup_out.hit_hist.bucket_1k='0' dedup_out.hit_hist.bucket_2k='0' dedup_out.hit_hist.bucket_4k='0' dedup_out.hit_hist.bucket_8k='0' dedup_out.hit_hist.bucket_16k='0' dedup_out.hit_hist.bucket_32k='0' dedup_out.hit_hist.bucket_64k='0' dedup_out.hit_hist.bucket_128k='0' dedup_out.hit_hist.bucket_256k='0' dedup_out.hit_hist.bucket_512k='0' dedup_out.hit_hist.bucket_1m='0' dedup_out.hit_hist.bucket_large='0' dedup_out.miss_bytes='0' dedup_out.misses='0' dedup_out.miss_hist.bucket_1k='0' dedup_out.miss_hist.bucket_2k='0' dedup_out.miss_hist.bucket_4k='0' dedup_out.miss_hist.bucket_8k='0' dedup_out.miss_hist.bucket_16k='0' dedup_out.miss_hist.bucket_32k='0' dedup_out.miss_hist.bucket_64k='0' dedup_out.miss_hist.bucket_128k='0' dedup_out.miss_hist.bucket_256k='0' dedup_out.miss_hist.bucket_512k='0' dedup_out.miss_hist.bucket_1m='0' dedup_out.miss_hist.bucket_large='0' outgoing.conns_idle_cur='0' outgoing.conns_idle_max='0' outgoing.conns_idle_tot='0' outgoing.conns_active_cur='2' outgoing.conns_active_max='3' outgoing.conns_active_tot='3' outgoing.conns_errors='0' outgoing.conns_passthru_tot='0' incoming.conns_idle_cur='0' incoming.conns_idle_max='0' incoming.conns_idle_tot='0' incoming.conns_active_cur='2' incoming.conns_active_max='6' incoming.conns_active_tot='127' incoming.conns_errors='0' incoming.conns_passthru_tot='0' dedup_status_array='cccc '

Here’s a sample remote endpoint log message in syslog:

Mar 15 17:05:50 127.0.0.1 10.11.100.73: device_timestamp='Wed Mar 15 00:05:46 2010 GMT' HostName='bigip3900c.demo.f5demo.com' version.version='10.1.0' message_type='woc_peer' peer_ref='10.20.50.103' name='bigip3900b.demo.f5demo.com' UUID='cd18:4840:f9e0:' mgmt_addr='10.11.100.72' version='10.1.0' dedup_cache='203588' dedup_action='DEDUP_ACTION_NONE' dedup_cache_refresh_flag='false' dedup_cache_refresh_count='0' state='WOC_PEER_STATE_READY' is_enabled='true' origin='MCP_ORIGIN_CONFIGURED' profile_serverssl='' tunnel_encrypt_data='true' tunnel_port='443' behind_nat='false' source_address='WOC_PEER_NAT_SOURCE_ADDRESS_NONE' config_status='none' routing='true' addr_list=''

And lastly if you’re an ASM or APM user there’s an F5 Networks DSM that will recognize your ASM logs; all you need to do is define the hostname or IP address of your QRadar system on your ASM logging profile.

Logging and Reporting Toolkit Series: Part One | Part Two | Part Three

]

pyControl v2 – Understanding the TypeFactory

citizen_elah — Thu, 01 Apr 2010 12:28:00 GMT

I admit it—I’ve been away from pyControl working on some other hot stove projects. When pyControl v2 launched in December, I ported some of the lighter scripts I’d written, but I put aside the scripts that required creating objects on the BIG-IP via iControl. After receiving a request to create a region in GTM via pyControl, I figured it was time to spend some energy learning the new techniques.

BIG-IP Logging and Reporting Toolkit – part three

Colin — Fri, 26 Mar 2010 14:38:00 GMT

In the first couple installments of this series we’ve talked about what we’re trying to accomplish, the vendor options we have available to us, their offerings, some strengths and weaknesses of each, etc. In this installment, we’re actually going to roll up our sleeves and get to getting. We’ll look at how to get things working in a couple different use cases including scripts, screenshots and config goodies. Before we get too far ahead of ourselves though, first things first – get the BIG-IP to send messages to syslog.

- Bigip v9

syslog {

remote server 10.10.200.30

}

- Bigip v10

syslog {

remote server {

    splunk {

      host 10.11.100.30

    }

}

}

This will send all syslog messages from the BIG-IP to the Splunk server; both BIG-IP system messages and any messages from iRules. If you’re interested in having iRules log to the Splunk server directly you can use the HSL statements or the log statements with a destination host defined. Ex) RULE_INIT has set ::SplunkHost “10.10.200.30” and then in the iRules event you’re interested in you assemble $log_message and then sent it to the log with log $::SplunkHost $log_message . A good practice would be to also record it locally on something like local0 incase the message doesn’t make it to the Splunk host.

For Splunk to receive the message you have to create a Data Input on udp:514 for the log statement. To cover HSL and log statements I’d recommend creating tcp:514 and udp:514 data inputs on Splunk. http://www.splunk.com/base/Documentation/4.0.2/Admin/Monitornetworkports covers this.

We’ll get to the scripts part in a bit, first…

W3C offload case

Now that BIG-IP is setup to send messages to Splunk and Splunk is setup to listen to them, let’s see what it looks like when put together. Open the Splunk Search application and enter ‘w3c-client-logging sourcetype=udp:514’ into the search bar. Here’s one of the things that makes Splunk really easy to work with: it recognized the key-value pairings in the log message without any configuration needed on my part. Next, I opened the Pick fields box and selected user_agent and added it to the list of fields I’m interested in; and now it shows up in alongside the log message and I can now build a report on it by clicking on the arrow.

The engineer in us wants to use technical terms to accurately convey the precise information we want to distribute. Splunk makes it easy to bridge the gap from technical terms to terms that are meaningful to non-engineers. So, for example a BIG-IP admin knows what this iRule is and what it’s called (in this case w3c-client-logging) – but those could be foreign concepts to folks in the Creative Service department that only want to know what browsers people are using to access a website. So, let’s employ some natural language too. The w3c-client-logging rule records a message when an HTTP transaction completes; a request and a response. So, let’s call it what it is.

On your Splunk system open up the $SPLUNKHOME/etc/system/local/eventtypes.conf file and add this:

[httpTransaction]

search = “Rule w3c-client-logging”

You might need to restart Splunk for this change to take effect.

Now, let’s go back to the search console and try out our new event type.

This is a basic usage of event types in Splunk, you can learn more here: http://www.splunk.com/base/Documentation/4.0.2/Admin/Eventtypesconf . With transforms.conf and props.conf you can also effectively rename the attributes so, lb_server could be called webServer instead.

Now that we have a custom event based off our search string, all we have to do is click the dropdown arrow next to userAgent (in this case) and select the report option from the dropdown. Here's the output we'd see:

Heh – lookit’ that; nagios is the most frequent visitor…

Network Event Data Stream Case

Now that we've seen the W3C example, let's take a look at another example that's much more rich (comma delineated). With no keys, just values, this changes things considerably. Let’s look at the Network Event Data Stream specification and see how it’s been implemented as an iRule.

iRule - http://devcentral.f5.com/wiki/default.aspx/iRules/NEDSRule.html

Doc – http://devcentral.f5.com/downloads/techtips/NedsF5v1.doc

Since this is an information rich data source, conveyed from the BIG-IP to the Splunk server using comma separated values it takes a few more simple steps for Splunk to be able to extract out the fields just like it did for the key-value pairs.

Open up $SPLUNKHOME/etc/system/local/transforms.conf and insert this:

[extract_neds.f5.conn.start.v1_csv]

DELIMS = ","

FIELDS = "EventID","Device","Flow","DateTimeSecs","IngressInterface","Protocol",”DiffServ","TTL","PolicyName","Direction"

[extract_neds.f5.conn.end.v1_csv]

DELIMS = ","

FIELDS = "EventID","Device","Flow","DateTimeSecs","PktsIn","PktsOut","BytesIn","BytesOut"

[extract_neds.f5.http.req.v1_csv]

DELIMS = ","

FIELDS = "EventID","Device","Flow","DateTimeSecs","Request","Host","URI","UserName","UserAgent"

[extract_neds.f5.http.resp.v1_csv]

DELIMS = ","

FIELDS = "EventID","Device","Flow","DateTimeSecs","Reply","ResponseCode","ContentType","ContentLength","LoadBalanceTarget","ServerFlow"

This names each list of information we’re interested in, indicates that the fields in the message are comma delimited and names the fields. You can name the fields for what’s appropriate for your environment.

Save the file. Next, open $SPLUNKHOME/etc/system/local/props.conf and insert this:

[eventtype::F5connectionStartEvent]

REPORT-extrac = extract_neds.f5.conn.start.v1_csv

[eventtype::F5connectionEndEvent]

REPORT-extrac = extract_neds.f5.conn.end.v1_csv

[eventtype::F5httpRequestEvent]

REPORT-extrac = extract_neds.f5.http.req.v1_csv

[eventtype::F5httpResponseEvent]

REPORT-extrac = extract_neds.f5.http.resp.v1_csv

This instructs the Splunk system to extract the information from the named fields.

Save the file. Next, open SPLUNKHOME/etc/system/local/eventtypes.conf and insert this (the ‘sourcetype=udp:514’ part is optional – set it up for your environment or omit the search term):

[F5connectionStartEvent]

search = neds.f5.conn.start.v1 sourcetype=udp:514

[F5connectionEndEvent]

search = neds.f5.conn.end.v1 sourcetype=udp:514

[F5httpRequestEvent]

search = neds.f5.http.req.v1 sourcetype=udp:514

[F5httpResponseEvent]

search = neds.f5.http.resp.v1 sourcetype=udp:514

Lastly, this defines the event to extract the data from.

Save the file, and restart Splunkd. There are a few processes you can restart to avoid a complete Splunkd restart, but my environment is a lab so I just restarted the whole thing. While Splunkd is restarting you should attach the NEDS iRule to a BIG-IP virtual server you want to receive data from and send some traffic though the VIP so your Splunk servers will get some data.

Now let’s navigate back to the Search app in the web ui. In the search bar, enter eventtype=F5connectionEndEvent . I opened the Pick fields box and selected BytesIn, BytesOut, Device, PktsIn and PktsOut .

As another way to use the Splunk search to report on traffic transiting a BIG-IP enter eventtype=F5connectionEndEvent |timechart avg(PktsOut) avg(BytesOut) into the search bar. This will generate a table for you listing the average number of packets transmitted from the VIP and the average number of bytes transmitted by the VIP for the default 10s time period.

I mentioned more to come about the script input at the top of the message. F5 recently added dashboards for WAN optimization and Access Policy management. One thing that I wish the dashboards provided is a historic view of the data so I can see how my infrastructure is changing over time as I upgrade applications and add more traffic to my network. Full disclosure: this BIG-IP interface isn’t a supported interface for anything other than the dashboard. Using BIG-IP 10.1 with a full WAN Optimization license, Perl and Splunk, here’s how I did it.

1) Place this script (http://devcentral.f5.com/downloads/techtips/text-dashboard-log.pl) somewhere on your Splunk system and mark it executable – I put mine in $SPLUNKHOME/bin/scripts

2) Ensure you have the proper Perl modules installed for the script to work

3) Add BIG-IP logon data to lines 58 and 59 – the user must be able to access the dashboard

4) Configure a data input for the Splunk server to get the dashboard data. My Splunk system retrieves a data set every 2 minutes. I’ve added in 2 collectors, one for each side of my WOM iSession tunnel.

After getting all this setup and letting it run for a while, if you navigate back to your Search console you should see a new Sourcetype show up called BIG-IP_Acceleration_Dashboard.

Clicking on the BIG-IP_Acceleration_Dashboard sourcetype displays the log entries sent to the Splunk system. Splunk recognizes the key-value pairings and has automatically extracted the data and created entries in the Pick fields list.

That’s a lot of data! Basically it’s the contents of the endpoint_isession_stat table and the endpoint data – you can get this on the CLI via ‘tmctl endpoint_isession_stat’ and ‘b endpoint remote’ . Now I can easily see that from basically March 8 until now my WOM tunnels were only down for about 4 minutes. Another interesting report I’ve built from here from here is the efficacy of adaptive compression for the data transiting my simulated WAN by charting lzo_out_uses, deflate_out_uses and null_out_uses over time.

Last, but certainly not least – there’s the Splunk for F5 Networks application available via http://www.splunk.com/wiki/Apps:Splunk_for_F5. You should definitely install it if you’re an ASM or PSM user.

Logging and Reporting Toolkit Series:

Part One

|

Part Two

iControl Development with BIG-IP LTM VE – Part 1

hamish — Mon, 15 Mar 2010 08:06:00 GMT

The LTM VE is a lifesaver when it comes to developing applications that use the iControl interface, or for development of iRules. My main (personal) use of it so far is the development of applications that use the iControl interface. Although a real LTM appliance has until now been the obvious choice of development target, the sheer cost of dedicated hardware is a big barrier to independent developers. In this respect the LTM VE comes into its own as an easy to use anywhere device.

Configuring a multi-server Testing Environment with VMWare Teams and BIG-IP LTM VE

dmacvittie — Thu, 11 Mar 2010 21:23:00 GMT

Having LTM-VE is nice, but setting things up so that you can actually use it for something is a different matter. There is a lot of information floating around out there about configuration ranging from Joe's Tech Tip to Lori's Architecture Reference to deployment guides. This article focuses on setting up a VMWare team that includes your BIG-IP and some test servers.

Since I wanted this to be something you can re-create, all of the servers utilized in this Tech Tip are clones of a straight-forward CENT-OS install. I toyed with a large number of OSS operating systems, and decided of all the OS's I installed, this was the one most suited to enterprise use, so cloned it a couple of times and changed the IP configuration. All of this will be documented below, the goal is to set you up so you can easily copy this environment. If for some reason you cannot, drop me an email and we can arrange for some way to get the files to you.

One thing to remember throughout an install of LTM-VE is that in the end it is just a BIG-IP. Sure, it's in an odd environment that offers some challenges of its own, but from an administration perspective it is just another BIG-IP. In terms of throughput and configuration it is different because of the environment, but in terms of day-to-day administration, it has nodes, pools, virtuals, iRules, etc. that are all accessed and managed in the same manner as you would if this was a physical BIG-IP.

I placed my BIG-IP and the three server instances into a single VMWare team, meaning I have single-button turn-on/turn-off for the entire test environment. For my purposes, my desktop (the VMWare host) is sufficient for representing clients, but I'll point out how and where you would add clients below.

And last caveat, this entire Tech Tip is done in VMWare Workstation. If you are running one of the server versions, some dialogs/etc will be slightly different.

First off, download and open the BIG-IP vmx file. A link to the vmx can be found on the DevCentral VE page. Once you have LTM-VE opened, you're ready to start your team-building exercise. Download CENT-OS from the link above, or some other OS that you are familiar with. Windows works just fine, I only excluded it from this solution because I can't be offering to give you copies of an OS that charges per-copy, and I did say if you had problems to drop me a line.

You'll need to install CENT-OS into a new VM, but VMWare makes that so easy that even if you've never done it you should have no problems. Just choose New|VM and tell it where to find the ISO images you downloaded.

Once the BIG-IP is loaded - and you can start it and play with it if you like, the default management address will be in the output of an ifconfig on the BIG-IP command line as eth0-mgmt. From your host OS's web browser you can navigate to that address and start licensing and configuration as explained in the release notes.

Next you'll want to create the team though. From the VMWare menu, select File|New|Team like this...

Give it a name and check the directory it will save to...

Then add VMs to the team. Choosing "existing VM" will move a VM into the team, choosing "Clone of a VM" will copy a VM into the team. I basically configured one copy of CENT-OS and then added clones of that one to the VM. I used the same process with BIG-IP LTM VE.

Finally, add network segments to the team if you are going to model an existing network with your team.

In the case that we're exploring here, I did not add any network segments to the team, but rather relied upon VMWare's DHCP capability to assign me networks that I then utilized when configuring the servers to use static IPs (as they would in most environments).

So the first thing you need to do after you’ve built your team is set your servers to use static IPs. They will default to DHCP, but that’s not the greatest solution for a load-balancing environment, since any change to the IP addresses of these servers will have an impact on the BIG-IP’s ability to load-balance. Note, if you are terribly lazy you probably don’t have to give them static IPs, because you control what goes in and out of the network, if you don’t add any servers to a segment, you should get the same IPs each time you boot. But boy, I just cannot recommend assuming a given server will always get the same address, that’s asking for trouble.

One last thing to check, go into the Connections tab of the Team Settings dialog. We started with each of our three servers set to NAT, and eventually had to switch them to Host Only during config. If you can’t ping across the VLANs (there should be an implied route between them), this is something worth checking.

And that covers creating the team. In short, the benefits of a team in our case are these:

-          Single button power on/off

-          Contained environment with everything you need to test in one VM

-          The ability to add network segments to emulate your production environment

As I mentioned above, I put nothing but Virtuals directly on the external network because I planned on testing with just my host OS, but you may well want to add clients to the external interface.

Next, you’ll have to configure the BIG-IP. You need to create an internal VLAN with the internal network (the one the servers are on) in it, and an external network that faces the outside world. The BIG-IP Management interface on first boot will be DHCP’d on your local network, you can change that setting through the browser on your client OS (we made it static on the local network). All of this is detailed in the release notes availabe on the VE page of DevCentral.

After your VLANs are set, then you need to go into the client OS’s and set the route to point at the BIG-IP address on the internal VLAN. Bing! At that point you should be able to ping the world. If you can, then you can create the nodes, pools, virtuals in the BIG-IP and start routing traffic!

I won’t go into a ton of detail here, since this article is running long already, but here are some screenshots of BIG-IP LTM VE with everything configured.

Here’s the list of nodes as-we-added them…

And the shots of the pool, showing it using these nodes…

And the VS, showing it using the pool. All is set, the VS is listening on 192.168.42.110.

There’s more I’d like to show you, but I’m not only out of room, this is longer than I’d wanted. So go forth and play, and I’ll circle back with some fun stuff like setting up iRules and such.

BIG-IP Logging and Reporting Toolkit - part two

Colin — Wed, 10 Mar 2010 11:10:00 GMT

In this second offering from Joe Malek’s delve into some advanced configuration concepts, and more specifically the logging and reporting world, we take a look at the vendors that he investigated, what they offer, and how they integrate with F5 products. He discusses some of the capabilities of each, their strengths and weaknesses and some of the things you might use each for. If you’ve been wondering what your options are for more in-depth log analysis and reporting, take a look to see what his thoughts are on a couple of the leading solutions.

Vendor descriptions:

Splunk - http://www.splunk.com/

“IT Search” is Splunk’s self identified core functionality. Splunk’s software contains multiple ways to obtain data from IT systems, indexes the data and reports on the data using a web interface. Splunk has invested in creating a Splunk for F5 application containing dashboard style views into log data for F5 products. Currently included in the application are LTM, GTM, ASM, APM and FirePass. The application is able to consume log messages sent to Splunk servers via syslog – and by extension iRules using High Speed Logging. Splunk is deployed as software to be installed on a customer provided system. Windows, Mac OS, Linux, AIX, and BSD variants are all supported host operating systems. Splunk can receive messages via files, syslog, SNMP, SCP, SFTP, FTP, generic network ports, FIFO queues, directory crawling and scripting. Splunk has a very intuitive and “Google like” interface allowing users to easily navigate and report on data in the system. Users are able to define reports, indices, dashboards and applications to present data as an organization requires. Upon receipt of data, Splunk can process the data according to in-built training or according to a user constructed taxonomy.

Q1 Labs - http://www.q1labs.com/

Q1 Labs brings a product called QRadar to market. QRadar combines functionality commonly found in SIEM, log management and network behavior analysis products. Q1 products are able to consume event messages as well as record information on a network connection basis. QRadar is available as a pay-for appliance and a no-charge edition in a virtual machine. Differences between the two editions are the SIEM and advanced correlation functionality. The no-charge edition is a log management tool only.

QRadar can receive messages via syslog SNMP, JDBC connectors, SFTP, FTP, SCP, and SDEE. Additionally QRadar con obtain network flow information in a port mirror/span mode. Customizing data views and report building are based on regular expressions. Customers can create their own regular expressions and build upon pre-configured expressions for reporting. In the SIEM module, QRadar includes approximately 250 events that can be sequenced together into complex “Offenses” in a manner similar to building a rule in Microsoft Outlook. “Universal Device Support Modules” can be created and shared among Q1 Labs customers.

PresiNET – http://www.presinet.com/

Whereas tcpdump is like an x-ray for your network, Total View One is like an MRI. Total View One enables customers to maximize the use of infrastructure resources and network performance. Total View One sensors collect protocol state information by tracking connections through a network. This is commonly done out-of-line from traffic streams via port mirroring or network tap technologies. Currently PresiNET has implemented the NEDS specification which enables Total View One to receive messages from BIG-IP products to process them as if they’d come from a PresiNET sensor. This integration started with the NEDS iRule and specification and from this PresiNET created their own parser. PresiNET products are delivered as appliances in both a central unit and sensor unit mode. Optionally one may subscribe to PresiNET on a managed service basis.    After you install a Total View One product in your network you get access to extensive views of available state information – with little or no additional work. If the included reporting capabilities aren’t enough, you can export data from the system as a CSV file.

What’s Next?

Now that you know who the players are and what they can do, be sure to check back next week to look at how the F5 products generate logs, how these technologies deal with them, and some testing results. To give you more of an idea of what’s to come, I’ll leave you with a look at the facts that will be delivered to the reporting systems from the F5 device(s) to see how they’re handled:

Virtual server accessed, client IP address, client port, LB decision results, http host, http username, user-agent string, content encoding, requested URI, requested path, content type, content length, request time, server string, server port, status code, device identifier, referrer, host header, response time, VLAN id, IP protocol, IP type of service, connection end time, packets, bytes, anything sent to a dashboard, firewall messages, client source geography, extended application log data, health information for back end filers, audit logs, SNMP trap information, dedup efficacy, compression codec efficacy, wom error counters, link characteristics as known, system state

Logging and Reporting Toolkit Series: Part One | Part Three

Maintenance Mode Rules for the F5 Networks Management Pack

Blog Author — Fri, 05 Mar 2010 16:07:46 GMT

A few releases ago we included a couple of new rules and new tasks to go with them. These rules and tasks will help you keep your Pool Members (LTM and GTM) in sync with your applications. These new rules are disabled by default, but we highly recommend enabling and integrating them into your deployment and existing distributed applications. Here is a short tutorial video on how to turn these rules on:

Enabling Maintenance Mode

For reference, the rules are named F5.Rules.DCO.MaintenanceModeSyncLtmPoolMember and F5.Rules.DCO.MaintenanceModeSyncGtmPoolVirtualServer and can be located under the F5 Device Type in the Rules area of the Authoring section of Operations Manager. By enabling these rules, any time an LTM Pool Member or GTM Pool Member (Virtual Server) is put into maintenance mode, it will also disable the Pool Member within the interval specified in the overrides of the rule. This is very important if you want to perform maintenance on either the Big-IP or the web server instance and do not want traffic to be sent. It should also be noted that once you stop maintenance mode, the Pool Members will not be re-enabled automatically. There are some technical reasons for this, but it boils down to the fact that a Pool Member may have been disabled before maintenance mode was started and we do not want to enable it if it was taken offline for another reason.

Now with these two new rules, there are also two Tasks associated with the device. They are named Sync LTM Pool Members Maintenance Mode and Sync GTM Pool Virtual Servers Maintenance Mode. If you do not want to enable the rules to automatically disable the Pool Members, you can still manually disable any Pool Member in maintenance mode on a device by running the appropriate task.

The ultimate reason behind including these rules and tasks it to allow greater flexibility with your applications. By enabling these rules, you can be assured that any Distributed Applications you have created that reference GTM or LTM Pool Members will behave more efficiently. Perhaps you have an IIS Distributed App that includes the related Virtual Server. By putting the entire application into maintenance mode, you will also ensure that traffic is not being sent to these servers, potentially causing more headache than necessary. If you have any questions regarding this feature, feel free to visit the forums here.

Announcing the F5 Management Pack 1.7.0: iRules-based object health events

Fri, 05 Mar 2010 11:24:08 GMT

The F5 Management Pack team has just released another version of our core Management Pack, which includes a new iRules feature and a variety of product improvements.

Our newest feature allows you to connect your iRules to the F5 Management Pack, and influence object health (e.g. LTM Virtual Server, Pool Members, Nodes, etc.) via events. We’ve also included support for 4 new PowerShell CmdLets allowing you to upload and remove iRules to your F5 BIG-IP device, and register and unregister iRules with LTM Virtual Servers. We’re really excited about the potential for iRules integration, which should enable a host of interesting diagnostic, monitoring, and orchestration scenarios into and beyond L2-L4. Our team will be posting more information online, including samples and videos around the iRules integration.

This new release also includes some improvements to setup, licensing, and clustered RMS support.

I’d like to thank all of you, our customers and partners, for providing great feedback on the product, reporting issues with detailed info, and working with the new features as we release them.

You can always find the latest F5 Management Pack and extensions at our DevCentral Download site: http://devcentral.f5.com/Default.aspx?tabid=217

If you have questions, ideas for new features, or need support, we welcome you to join and post at our DevCentral Forums: http://devcentral.f5.com/Default.aspx?tabid=53&afgroup=26

Thank you,

Stephen Fisher

F5 Management Pack, Product Development Manager

iRules and the F5 Management Pack

Blog Author — Fri, 05 Mar 2010 09:49:41 GMT

Starting with v1.7.0.390 the F5 Management Pack (F5 MP) has some built-in diagnostic capabilities based on iRules. Messages logged within iRules could be sent directly to the F5 Monitoring Service that now has an asynchronous TCP listener specifically designed to receive and process iRule messages, and based on these messages the F5 MP can write events into the Microsoft's System Center Operations Manager (SCOM) event data source, against the F5 device monitoring objects discovered. Putting it simply, you can mark an F5 monitoring object state in SCOM, as up and down or in warning state, directly from iRules.

A typical use case scenario that I'll be presenting to describe this feature would be an iRule created for detecting a possible Distributed Denial of Service (DDoS) attack. Examples of such iRules can be found on F5 DevCentral iRules CodeShare. I'll just choose a simple one here, as a way to detect and avoid a location specific DDoS attack through HTTP requests. In this example we are trying to detect a Layer 7 DDoS attack and there's always a certain possibility for false positives, so we would like to notify the SCOM administrator by marking the state of the appropriate LTM Pool Member object to 'Warning' or 'Critical' while attempting to deflect the attack. This way further actions can be immediately taken by the network administrators to address the attack.

We'll slightly change the iRule to accommodate our use-case and log a well-formatted message that will be sent to the F5 MP, which would parse and write it into the SCOM event data source against the F5 LTM Pool Member object being affected. So basically, the attack is being detected in the HTTP_REQUEST iRule event and then in the LB_SELECTED event the log message is sent out.

when RULE_INIT {
# state tracking: 0=idle; 1=normal; 2=under attack;

set ::state 0
set ::health "Healthy"
set ::message "OK."
}

when LB_SELECTED {
switch $::state {
2 {
set ::health "Critical"
set ::message "Under attack."
}
default {
set ::health "Healthy"
set ::message "OK."
}
}

# when not idle, log the state
if { $::state != 0 } {
log local0. "
<config_event>
<time>[clock seconds]</time>
<type>pool_member</type>
<name>[LB::server pool]</name>
<ip>[LB::server addr]</ip>
<port>[LB::server port]</port>
<health>$::health</health>
<message>$::message</message>
</config_event>"
}
}

when HTTP_REQUEST {
if {[string tolower [HTTP::header "Accept-Language"]] contains "ru"} {
set ::state 2
} else {
# more evolved logic should go here...
...
set ::state 0
}
}

In our example we used the local0 facility to log to, and for this the following syslog-ng configuration (/etc/syslog-ng/syslog-ng.conf) has been created on the BIG-IP, for logging to a remote host (TCP listener), where the F5 Monitoring Service is listening:

# Log to a remote host
destination d_loghost {
tcp("172.27.155.129" port(3939));
};

filter f_iRules {
match("config_event");
};

log {
source(s_syslog_pipe);
filter(f_iRules);
destination(d_loghost);
};

By default the F5 Monitoring Service will listen on port 999 for incoming iRule messages. In our example we set the port to 3939. The F5 Monitoring Service configuration file (f5mpsvc.exe.config) should contain the following configuration entry within the <appSettings> section:

<appSettings>
...
<add key="iRulesCustomHealthConfigurationPort" value="3939" />
...
</appSettings>

In a real world deployment, it is recommended to use a different logging facility on the BIG-IP (rather then local0), to avoid sending too much traffic to the F5 Monitoring Service, or the usage of log filters on the BIG-IP (such as f_iRules in our configuration example above) which would also affect performance.

Using this implementation, when the DDoS condition occurs, the iRules message is sent to the F5 Monitoring Service, which will process the message and will mark the state of the LTM Pool Member object to "Critical". Keep in mind that the F5 MP will only accept well-known messages logged from iRules, which should have the following format:

<config_event>
<time></time>
<type></type>
<name></name>
<ip></ip>
<port></port>
<health></health>
<message></message>
</config_event>

Most of the elements in the XML snippet above would require a value to be specified and the possible values for the <health> element would be: Healthy, Critical, Warning. As soon as the F5 MP writes the related event into the SCOM event data source, the corresponding LTM Pool member specified in the log message (see the LB_SELECTED event in the iRule implementation) will change its state to Critical. SCOM administrator can configure alerts against the state-change event triggered by the iRule and get notified about the possibility of the DDoS attack.

Figure 1: Sending iRule messages from BIG-IP to the F5 MP and SCOM

Installing the F5 Management Pack in a SCOM 2007 RMS Cluster with Distributed Management Servers

Blog Author — Mon, 01 Mar 2010 13:00:48 GMT

In a previous article I've documented the way the F5 Management Pack should be installed and configured to run in a SCOM 2007 RMS Cluster environment. In this article I will extend the clustered deployment scenario with SCOM 2007 when additional management servers are being used.

Following Microsoft's best practices deployment guidance for a SCOM 2007 distributed management server (DMS) environment, when installed in such a deployment, the F5 Management Pack (F5 MP) will only have the F5 Monitoring Service enabled and running on the Management Servers, while being disabled on the Root Management Server (RMS). This would also apply in a clustered RMS configuration, when the F5 MP setup wizard will automatically disable the F5 Monitoring Service on the RMS cluster nodes. This means that under normal circumstances, there's no need to cluster the F5 Monitoring Service inside an RMS cluster with DMS, since the F5 device discovery and management tasks are being executed on the Management Servers rather then the RMS cluster nodes.

Still, the F5 Monitoring Service could be enabled on the RMS cluster nodes within a DMS environment with RMS-clustering, in which case the configuration steps described in the RMS clustering article would be followed. In the following section I'll be focusing on DMS-specific issues regarding the F5 MP deployment in an RMS-cluster, regardless of the fact that the F5 Monitoring service is enabled or disabled on the RMS cluster nodes.

First of all, there is no special requirement needed for deploying the F5 MP on a DMS, other than having the F5 MP being already installed on the RMS, or at least one of the RMS cluster nodes. Again, the installation of the F5 MP on the RMS cluster node will disable by default the F5 Monitoring Service, but will install the management pack and related data source components.

There is one verification step worth taking for good measure, prior to attempting to run F5 device discovery and management tasks from the (distributed) management server and this is related to one of the most common issues encountered with deploying the F5 MP in a SCOM DMS environment with RMS-clustering:

Making sure the Security Principal Name (SPN) is properly set up for the account running the SCOM SDK service. Although this is mostly a SCOM 2007 deployment related issue and has no specific ties into the F5 MP deployment, we've seen it occurring often in clustered RMS with DMS. One way to quickly check if the SPN settings are valid is to run the:

SetSPN -L <account>

command, where the <account> is domain account running the SCOM SDK service. The expected output should be similar to:

Registered ServicePrincipalNames for CN=<name>, CN=Users, DC=<domain> ..:
        MSOMSdkSvc/<rms-cluster-node1>
        MSOMSdkSvc/<rms-cluster-node1>.<domain>
        MSOMSdkSvc/<rms-cluster-node2>
        MSOMSdkSvc/<rms-cluster-node2>.<domain>
        MSOMSdkSvc/<mgmt-server>
        MSOMSdkSvc/<mgmt-server>.<domain>
        ...

Similarly, running the:

SetSPN -L <mgmt-server>

command on the Management Server (or RMS cluster node) should produce a similar output with:

Registered ServicePrincipalNames for CN=<mgmt-server>, CN=Computers, DC=<domain> ..:
        MSOMSdkSvc/<mgmt-server>
        MSOMSdkSvc/<mgmt-server>.<domain>
        MSOMHSvc/<mgmt-server>
        MSOMHSvc/<mgm_server>.<domain>
        ...

Basically we have to make sure the SPN entries are correctly mapped to the hosts running the SCOM SDK and Health services: on the RMS cluster nodes and distributed management servers involved. In our example above these are <rms-cluster-node1>, <rms-cluster-node2> and <mgmt-server>, the latter one being the actual [distributed] management server that we're considering for deploying the F5 MP on.

On a last note, when using multiple management servers for discovering multiple F5 devices, keep in mind that each F5 device discovered will be "owned" by (or linked to) the Health Service on the management server where the F5 device has been discovered on. This means that subsequent management tasks related to the F5 device (such as device rediscovery or removal) should be initiated on the same management server for the given device.

Citrix XenApp 5.0 Implementation Tips

naladar — Fri, 26 Feb 2010 14:13:00 GMT

I recently had the pleasure of working on a Citrix 5.0 implementation and I wanted to share a few things that I learned during that setup. As many of you know, there are two deployment guides that have been made available by F5 Networks in regards to setting up Citrix Presentation Server 4.5 in TMOS versions 9.x and 10.x. They are excellent guides and the best thing about them is that you can utilize those guides to assist you in deploying Citrix XenApp 5.0, with a few exceptions of course. Those exceptions are what I will be covering in this tech tip.

Both of the previously mentioned deployment guides discuss editing files on the Citrix farms Web Interface servers so that it looks for the client IP address in the X-Forwarded-For HTTP header. Otherwise, every connection will appear to be originating from the BIG-IP LTM and not from its true IP. After reading both guides and looking at my current environment I was dismayed to find that the files and locations mentioned were no longer valid. I then turned to my top three resources on the web in the search for an answer: AskF5, DevCentral and Google.

I struck out on the first two (which seldom happens) but my Google search did turn up some interesting results on the Citrix Forums. I finally found some code posted by Sam Jacobs back in August 2009 that modifies the way the Citrix farm looks up the client IP address. His method allows for the use of the X-Forwarded-For header.

The first file that you will want to find and edit is the Include.java file. You will want to locate and change this file on every Web Interface XenApp server in the farm. Speaking from experience, save a copy of the original file to a safe location such as your desktop or flash drive. DO NOT copy the file and rename the original to Include.old and leave it on the server. It may sound crazy, but doing that will not work. I’m not a programmer, so I cannot tell you why that will not work, but I can tell you I know for a fact it will not. That being said, here is the file path for the Include.java file:

“\Inetpub\wwwroot\Citrix\XenApp\app_code\PagesJava\com\citrix\wi\pageutils\Include.java”

Now that you have found the file, open it up with a text editor (I use Textpad) and find the Java routine named “getClientAddress”. Replace the code for that routine with the code listed below.

public static String getClientAddress(WIContext wiContext) {
String ageClientAddress = AGEUtilities.getAGEClientIPAddress(wiContext);
String userIPAddress = wiContext.getWebAbstraction().getRequestHeader("X-FORWARDED-FOR");
if (userIPAddress == null) {
userIPAddress = wiContext.getWebAbstraction().getUserHostAddress();
}
return (ageClientAddress != null ? ageClientAddress : userIPAddress);
}

Save the file and wash/rinse/repeat this step on every Web Interface server in the farm. The next thing that you will want to do is to modify the login page so that it displays the client IP address being obtained from the X-Forwarded-For header. The file you will want to edit is called “loginView.ascx” and can be found in the following file path on your Web Interface Servers:

”\inetpub\wwwroot\Citrix\XenApp\app_data\include\loginView.ascx”

The code you will want to add is:

Client IP: <%= com.citrix.wi.pageutils.Include.getClientAddress(wiContext) %>

I added the code directly below the LoginPageControl viewControl line and it works well for me. Save the file and repeat this step on every Web Interface server in the farm and reboot each Web Interface Server after you are done.

That’s it! Well, you do have to complete the other setup steps listed in the deployment guide that you are using, but after that your farm will be ready for business! I am aiming to develop some custom monitors for the Web Interface Server and for the XML Broker Servers over the next few weeks. Once I have those done I will put them out in the forums for the community enjoy.

-naladar

BIG-IP Logging and Reporting Toolkit - part one

Colin — Thu, 25 Feb 2010 12:01:00 GMT

Joe Malek, one of the many awesome engineers here at F5, took it upon himself to delve deeply into a very interesting but often unsung part of the BIG-IP advanced configuration world: logging and reporting. It’s my great pleasure to get to share with you his awesome study and the findings therein, along with (eventually) a toolkit to help you get started in the world of custom log manipulation. If you’ve ever questioned or been curious about your options when it comes to information gathering and reporting, this is definitely something you should read. There will be multiple parts, so stay tuned. This one is just the intro.

Description

F5 products occupy critical positions in application delivery infrastructure. They serve as gateways, proxies, accelerators and traffic flow arbiters. In these roles customer expectations vary for the degree and amount of event information recorded. Several opportunities exist within our current product capabilities for our customers and partners to produce and consume log messages from and via F5 products. Efforts to date include generating W3C style log messages on LTM via iRules, close integration with leading vendors and ASM (requires askf5 login), and creating relationships with leading vendors to best serve our customers. Significant capabilities exist for customers and partners to create their own logging and reporting solutions.

Problems and opportunity

In the many products offered by F5, there exists a variety of logging structures. The common log protocols used to emit messages by F5 products are Syslog (requires askf5 login) and SNMP (requires askf5 login), along with built-in iRules capabilities. Though syslog-ng is commonplace, software components tend to vary in transport, verbosity, message formatting and sometimes syslog facility. This can result in a high degree of data density in our logs, and messages our systems emit can vary from version to version.[i] The combination of these factors results in a challenge that requires a coordinated solution for customers who are compelled by regulation, industry practice, or by business process, to maintain log management infrastructure that consumes messages from F5 devices.[ii]

By utilizing the unique product architecture TMOS employs by sharing its knowledge about networks and applications as well as capabilities built into iRules, TMOS can provide much of this information to log management infrastructure in a simple and knowledgeable manner. In effect, we can emit messages about appliance state and offload many message logging tasks from application servers. Based on our connection knowledge we can also improve the utility and value of information obtained from vendor provided log management infrastructure.[iii]

Objectives and success criteria

The success criteria for including an item in the toolkit is:

1. A capability to deliver reports on select items using the leading platforms without requiring core development work on an F5 product.

2. An identified extensibility capability for future customization and report building.

Assumptions and dependencies

Vendors to include in the toolkit are Splunk, Q1Labs and PresiNET

ASM logging and reporting is sufficient and does not need further explanation

Information to be included in sample reports should begin to assist in diagnostic activities, demonstrate ROI by including ROI in an infrastructure and advise on when F5 devices are nearing capacity

Vendor products must be able to accept event data emitted by F5 products. This means that some vendors might have more comprehensive support than others.

Products currently supported but not in active development are not eligible for inclusion in the toolkit. Examples are older versions of BIG-IP and FirePass, and all WANJet releases.

Some vendor products will require code modifications on the vendor’s side to understand the data F5 products send them.

[i] As a piece of customer evidence, Microsoft implemented several logging practices around version 9.1. When they upgraded to version 9.4 their log volume increased several-fold because F5 added log messages and changed existing messages. As a result existing message taxonomy needed to be deprecated and we caused them to need to redesign filters, reports and create a new set of logging practices.

[ii] Regulations such as the Sarbanes-Oxley Act, Gramm Leach Blyley Act, Federal Information Security Management Act, PCI DSS, and HIPPA.

[iii] It is common for F5 products to manipulate connections via OneConnect, NATs and SNATs. These operations are unknown to external log collectors, and pose a challenge when assembling a complete view of the network connections between a client and a server via an F5 device for a single application transaction.

What’s Next?

In the next installment we’ll get into the details of the different vendors in question, their offerings, how they work and integrate with BIG-IP, and more.

Logging and Reporting Toolkit Series: Part Two | Part Three

Laptop Load Balancing For The Developer With BIG-IP LTM VE

Joe — Wed, 24 Feb 2010 15:20:00 GMT

Introduction

So, you've probably read all about the BIG-IP LTM VE. You've gone through the download process and got the thing configured and running in VMWare. You've even gone through the initial configuration steps to get the management port configured and got the thing licensed up. Now what? Well, we've got a bunch of examples of firing up separate VM instances of host operating systems so that you can test creating pools of servers with those other VM instances. The problem there is that running those VM's is costly in RAM and CPU.

Being a developer, my first thought was how I could make use of my existing development environment with Visual Studio and IIS on the laptop that is hosting the LTM VE virtual machine. Why should I have to create a new VM instance of Windows and incur the overhead of running that extra VM? Well, you don't!

In this article, I'm going to show you how to build a full client-server environment utilizing only the laptop's host operating system and a single LTM VE instance.

Configuring IIS

The first step is to configure a web application. You could use any web server you desire (apache, tomcat, etc) but I'll use IIS since I'm already using it to debug applications I've written with Microsoft Visual Studio .NET. Below is the IIS 7 admin console running under Windows 7. The main trick here, is to make sure that the site bindings for your app will accept all ip addresses. That way requests coming from the internal VMWare interface will be able to handle the requests.

The Web Application

For this example, I'm going to go old school and write a simple .asp (HTML) page with no server side processing. Requests to the Default Web Site in IIS will return the contents of this file.

Creating your VM

I'm going to assume for this article, that you already have VMWare Workstation or Server installed and that you have got your BIG-IP LTM-VE fired up and licensed. If you don't, there are several walkthroughs here on DevCentral that will help you get going.

VMWare sets up several network adapters. We are going to make use of two of those interfaces to act as the external and internal networks for the LTM. By configuring IIS to handle requests from all addresses, this means we can create a pool member pointing to the internal network here. In this example 192.168.239.1 is my external network and 192.168.41.1 is my internal network address that will act as a pool member.

Provisioning your LTM-VE

Now, I could go and document all of the steps needed to create the objects on the LTM needed to successfully deploy the application but that wouldn't be any fun, plus it wouldn't let me show off my iControl PowerShell integration.

I've written a deployment iControl PowerShell script that will go through the process of configuring VLANs and SelfIPs, creating a pool with the internal host NIC as a pool member with it's associated health monitors, creating the Virtual Server, and, just for fun, an iRule that will modify the response content from IIS back to the browser.

Here's the output of the script. I'll describe the pieces below.

Create VLANs

A default LTM-VE configuration has no objects created so the first step is configuring VLAN for the external and internal networks. The code below can be used to create the "external" and "internal" VLANs. In my example the external VLAN is configured with the 1.1 interface as untagged (corresponding to the external VMWare NIC) and the internal VLAN using interface 1.2 (corresponding to the internal VMWare NIC).

1: function Create-VLAN()

2: {

3: param([string]$name, [long]$id, [string]$member_name);

4:

5: # Create VLAN

6: $vlans = (, $name);

7: $vlan_ids = (, $id);

8: $member = New-Object -TypeName iControl.NetworkingVLANMemberEntry;

9: $member.member_name = $member_name

10: $member.member_type = "MEMBER_INTERFACE";

11: $member.tag_state = "MEMBER_UNTAGGED"

12: $memberA = (, $member);

13: $memberAofA = (, $memberA);

14: $failsafe_states = (, "STATE_DISABLED");

15: $timeouts = (, 1500);

16: $mac_masquerade = (, "");

17:

18: (Get-F5.iControl).NetworkingVLAN.create(

19: $vlans, $vlan_ids, $memberAofA,

20: $failsafe_states, $timeouts, $mac_masquerade);

21: }

Looking in the BIG-IP GUI, the following VLANs were created.

Create SelfIPs

Now that the VLANs have been created, Self IP Addresses must be configured for those VLANs so that the LTM can communicate with the laptop network interfaces. The following code is used to create two SelfIPs (192.168.42.2 for the internal VLAN, and 192.168.239.2 for the external VLAN).

1: function Create-SelfIP()

2: {

3: param([string]$address, [string]$vlan);

4:

5: $self_ips = (, $address);

6: $vlan_names = (, $vlan);

7: $netmasks = (, "255.255.255.0");

8: $unit_ids = (, 0);

9: $floating_states = (, "STATE_DISABLED");

10: (Get-F5.iControl).NetworkingSelfIP.create(

11: $self_ips, $vlan_names, $netmasks,

12: $unit_ids, $floating_states

13: );

14: }

And again, the BIG-IP GUI shows the two Self IPs for the two VLANs.

Create Pool

The previous two steps were to ensure the lower networking layer was in place to allow connections from the upcoming generated Virtual Server as well as the defined pool members. The next step is to create a pool with a pool member of the internal VMWare NIC address (192.168.41.1). In this code, the pool is created and I've associated two health check monitors on the pool member. In this case I'm using the standard http and gateway_icmp monitors. You can modify these to suit your taste.

1: function Create-Pool()

2: {

3: param([string]$name, [string]$member_ip, [long]$member_port);

4:

5: $pool_list = (Get-F5.iControl).LocalLBPool.get_list();

6: if ( -not (Is-InList -search $name -list $pool_list) )

7: {

8: $pool_names = (, $name);

9: $lb_methods = (, "LB_METHOD_ROUND_ROBIN");

10: $member = New-Object -TypeName iControl.CommonIPPortDefinition;

11: $member.address = $member_ip;

12: $member.port = $member_port;

13: $memberA = (, $member);

14: $memberAofA = (, $memberA);

15:

16: Write-Host "Creating Pool `"$name`"...";

17: (Get-F5.iControl).LocalLBPool.create($pool_names, $lb_methods, $memberAofA);

18:

19: $monitor_association = New-Object -TypeName iControl.LocalLBPoolMonitorAssociation;

20: $monitor_association.pool_name = $name;

21: $monitor_association.monitor_rule = New-Object -TypeName iControl.LocalLBMonitorRule;

22: $monitor_association.monitor_rule.type = "MONITOR_RULE_TYPE_AND_LIST";

23: $monitor_association.monitor_rule.quorum = 1;

24: $monitor_association.monitor_rule.monitor_templates = ("http", "gateway_icmp");

25: $monitor_associations = (, $monitor_association);

26:

27: Write-Host "Assigning monitors to Pool `"$name`"...";

28: (Get-F5.iControl).LocalLBPool.set_monitor_association($monitor_associations);

29: }

30: }

And again, the BIG-IP GUI shows the generated Pool with the single defined member. Oh, and as a bonus, we have a Green Circle status which means the health checks succeeded and we're ready to go!

Create Virtual Server

Now that the application pool is created, the virtual server is configured to give an IP address you can use to test the IIS application. The Virtual Server creation process involves specifying virtual server configuration as well as a list of profiles to associate with the virtual. In this example, I'm using the stock "http" profile and as an added bonus, I've added in the "stream" profile so that my upcoming iRule will be able to do some content replacement.

1: function Create-Virtual()

2: {

3: param([string]$name, [string]$ip, [long]$port, [string]$pool);

4: $vs_list = (Get-F5.iControl).LocalLBVirtualServer.get_list();

5: if ( -not (Is-InList -search $name -list $vs_list) )

6: {

7: $definition = New-Object -TypeName iControl.CommonVirtualServerDefinition;

8: $definition.name = $name;

9: $definition.address = $ip;

10: $definition.port = $port;

11: $definition.protocol = "PROTOCOL_TCP";

12: $definitions = (, $definition);

13: $wildmasks = (, "255.255.255.255");

14: $resource = New-Object -TypeName iControl.LocalLBVirtualServerVirtualServerResource;

15: $resource.type = "RESOURCE_TYPE_POOL";

16: $resource.default_pool_name = $pool;

17: $resources = (, $resource);

18: $profile = New-Object -TypeName iControl.LocalLBVirtualServerVirtualServerProfile;

19: $profile.profile_context = "PROFILE_CONTEXT_TYPE_ALL";

20: $profile.profile_name = "http";

21: $profile2 = New-Object -TypeName iControl.LocalLBVirtualServerVirtualServerProfile;

22: $profile2.profile_context = "PROFILE_CONTEXT_TYPE_ALL";

23: $profile2.profile_name = "stream";

24: $profileA = ($profile2, $profile);

25: $profileAofA = (, $profileA);

26:

27: Write-Host "Creating Virtual Server `"$name`"...";

28:

29: (Get-F5.iControl).LocalLBVirtualServer.create($definitions, $wildmasks, $resources, $profileAofA);

30:

31: Write-Host "Enabling SNAT Automap on Virtual Server `"$name`"...";

32: (Get-F5.iControl).LocalLBVirtualServer.set_snat_automap( (, $name) );

33: }

34: }

You'll see the Virtual Server has been created with a destination address of 192.168.239.3 which coincides with the "external" VLAN and external VMWare NIC.

Create iRules

Now, this isn't necessarily needed for this deployment but I figured I'd throw something in to prove that traffic is actually going through the LTM and get you an iRule to start with. The follow code will create an iRule that does a stream replacement in the HTTP response from IIS. It converts all occurrences of "Test" with "Test (Modified by iRule)". This way we can tell from our client testing that the iRule is getting processed.

After the iRule is created, it is assigned to the Virtual Server with the LocalLB.VirtualServer.add_rule method.

1: function Create-iRule()

2: {

3: param([string]$virtual, [string]$rule_name);

4:

5: $rule_list = (Get-F5.iControl).LocalLBRule.get_list();

6: if ( -not (Is-InList -search $rule_name -list $rule_list) )

7: {

8: $rule = New-Object -TypeName iControl.LocalLBRuleRuleDefinition;

9: $rule.rule_name = $rule_name;

10: $rule.rule_definition = @"

11: when HTTP_REQUEST {

12: log local0. "Request for URI: [HTTP::uri]";

13: }

14: when HTTP_RESPONSE {

15: log local0. "Response [HTTP::status]";

16: STREAM::expression "@Test@Test (Modified by iRule)@";

17: STREAM::enable;

18: }

19: "@;

20: Write-Host "Creating iRule `"$rule_name`"...";

21: (Get-F5.iControl).LocalLBRule.create( (, $rule) );

22: }

23:

24: # Assign iRule to Virtual Server

25: $vs_rules = (Get-F5.iControl).LocalLBVirtualServer.get_rule( (, $virtual) );

26: $found = $false;

27: foreach ($vs_rule in $vs_rules[0])

28: {

29: if ( $rule_name.Equals($vs_rule.rule_name) ) { $found = $true; }

30: }

31: if ( -not $found )

32: {

33: $virtual_servers = (, $virtual);

34: $rule = New-Object -TypeName iControl.LocalLBVirtualServerVirtualServerRule;

35: $rule.rule_name = $rule_name;

36: $rule.priority = 500;

37: $rules = (, $rule);

38: Write-Host "Assigning iRule `"$rule_name`" to virtual `"$virtual`"...";

39: (Get-F5.iControl).LocalLBVirtualServer.add_rule($virtual_servers, $rules);

40: }

41: }

And there is the iRule in the BIG-IP GUI.

Testing Your Application

Now that the VLANs, SelfIPs, Pool, and VirtualServer are configured and functioning, it's time to test out the application. Here, I opened up a browser and typed in the address of the new Virtual Server (192.168.239.3) and you'll see the response came back. And, you'll also notice that the iRule did it's magic and modified the response content from the IIS application.

If a browser is too much work for you, curl works just as well.

Conclusion

I've shown in this example how you can build and test a fully functional web application with a single host operating system (in my case Windows 7) and a single instance of BIG-IP LTM VE. This will greatly reduce the complexity for developers who want to test applications they are currently debugging in the host operating system.

Downloads

The provisioning script used in this article can be found in the iControl CodeShare under PsProvisionVEForLocalDev

Whaddya Mean LTM VE Only Has Two Interfaces?

citizen_elah — Wed, 24 Feb 2010 02:00:00 GMT

Last week we covered the basics on network configuration for BIG-IP LTM VE on the VMware Workstation/player hypervisors and the ESX/ESXi hypervisors. This week we’ll cover a couple more options at your disposal.

Announcing the F5 Management Pack 1.6.2 and a few others

Sat, 20 Feb 2010 19:52:07 GMT

F5 Networks has been releasing versions of our Management Pack to DevCentral monthly for quite awhile, and I’d like to provide a bit more context around our last few releases (1.6.0, 1.6.1, and 1.6.2). If you’re always up to date with the earlier releases, feel free to skip right to the 1.6.2 release at the bottom of this post.

Or if you’d like to obtain the latest release, you can jump straight to our F5 Management Pack DevCentral download site.

Versioning notes

In general, we increment the minor version when we do health model or major feature updates, while bug fixes and maintenance releases result in the revision number be incremented.

Version 1.6.0

The 1.6.0 version introduced support for a data center orchestration scenario: virtualized app migration. Specifically, the extension management pack enables migrating a virtualized application (hosted inside Microsoft Virtual Machine Manager) to a different hardware host with the LTM configuration being updated in parallel and transparently. The administrator is only required to specify the virtual machine instance identifiers, the source host address and device, and the target host address and device. The F5 Management Pack will then make the appropriate updates to the BIG-IP, including creation of pool members, deletion of old pool members, and draining connections from the old server, if necessary. The F5 MP will also use the fastest available virtual machine migration algorithm available, whether that is a Hyper-V live migration [including network stack and live memory] or quick-migration. You can download the app migration extension at the F5 Management Pack Extensions download site as “Virtual App Migration Pack”.

In the 1.6.0 version, we also provided tasks and PowerShell APIs to enable and disable GTM Pool Virtual Servers for distributed application and wide-area configuration changes.

Version 1.6.1

Our next release was primarily focused around maintenance fixes and targeted customer issues.

We made some performance improvements to discovery which should show an order of magnitude improved to configuration discovery and updates. In the past, we’ve seen BIG-IP configurations with about 500-1,000 objects take anywhere from 5-10 minutes, depending on hardware and database configuration. With the performance improvements, we have simulated BIG-IP configurations with 1,000-15,000 objects being discovered in the same amount of time. I should note that a variety of factors affect configuration discovery, update and counter throughput, beyond simply duration. We’ll be posting more information on hardware, database, and System Center configuration guidance in the near future.

We also made some fixes to support case-sensitive collisions between BIG-IP configuration objects. For example, prior to version 1.6.1 if your BIG-IP had a pool member named “Foo” and another named “foo”, the F5 Management Pack may have encountered issues reporting the correct health of those objects, or updates to their configuration.

Version 1.6.1 also added support for more frequent device performance counter collection. While the interface to this work will be mostly transparent to Management Pack users, since the changes were made to the device network protocol, we have written an article that helps explain some scenarios and methods to increase F5 BIG-IP performance counter frequency.

Finally, we added support for clustered System Center Management Servers, which allows for pairing Root Management Servers, the F5 Management Pack monitoring service, and improving reliability for your network and application monitoring framework. Julian Balog, the F5 Networks engineer who implemented the majority of our clustering support, wrote a detailed article on F5 Management Pack clustering configuration and support.

Version 1.6.2

In our latest release, we have added support for BIG-IP device partitions. Notably, we have worked to enable our existing PowerShell BIG-IP device configuration APIs to continue working without needing to modify your existing PowerShell scripts.

We have also added support for synchronizing System Center Operations Manager “maintenance mode” with the state of the F5 BIG-IP LTM pool member and GTM Pool Virtual Server objects. For example, if you create a distributed application in System Center and put the application or portions of the infrastructure into “maintenance mode”, then you can enable “maintenance mode” sync to simultaneously disable the pool member or virtual server. We intentionally only support a one-way sync with maintenance mode, since our belief is that it should be an explicit action to re-add the application for traffic load balancing. For example, if a network application or pool member is disabled, but it isn’t officially in “maintenance mode” ~ e.g. the app server has crashed, we want to ensure the load balancer does not restart sending traffic to the crashed server.

We have also made a number of less visible reliability and user experience improvements over the last couple months. We have some really exciting features coming in our next few monthly releases, and I’ll post more information as they release.

And as always, if you have questions, ideas, feedback, or need support for the F5 Management Pack, we encourage you to visit our F5 Management Pack DevCentral Forums. We review every post, and these Forums are your most efficient avenue to support and feedback.\

Thank you,

Stephen Fisher

Product Development Manager - F5 Management Pack

New Geolocation Capabilities in v10.1

citizen_elah — Fri, 19 Feb 2010 11:00:00 GMT

With the BIG-IP GTM and 3-DNS products, location-based service has existed in the form of topology-based load balancing. The possibilities grow exponentially in v10.1, as you now have the capabilities in GTM and LTM. In this tech tip, we’ll discuss the iRules command access, the update procedures for keeping your data current, and some use cases for consideration.

BIG-IP LTM VE: What is it? Where do I start? Why is everyone so excited?

Colin — Wed, 17 Feb 2010 08:45:00 GMT

The already impressive flexibility and power of the BIG-IP takes a giant leap forward with the announcement of the BIG-IP LTM VE. This Virtual Edition of LTM not only allows for extremely easy trial of the technology and an easily attainable resource for local development or testing, but it also shows that no matter how many tricks LTM might have, there's likely one more around the corner.

If you're like me then you're already an F5 user. I use F5 gear just about every day while writing code and content for DevCentral, which I'm sure isn't a shock to anyone. If you're like me then you're also someone that's been looking for a few things for a while now. You've been looking for a way to edit your iRules on the road. You've been looking for a way to test configuration changes & profile tweaks without disrupting your current production (or staging, or test) environment(s). You've been wondering how to try out 10.1 without having to do a full upgrade on your currently running system. If you're like me, you're really, really excited about BIG-IP LTM VE. If any of these scenarios sound familiar, and you're not excited yet, then you should be.

The LTM VE release is a fully functioning BIG-IP LTM that you can run from VMware. All you need is a system running VMware and some free disk space, and you're set. Grab a copy of the files, unpack them, get a free trial license from f5.com, and in just a few minutes you'll be up and running with a fully functional LTM to poke at until your heart's content. It's rate limited to 1Mb/s but that's more than enough for just about any test case I could come up with short of full load traffic shaping tests and the like.

We've had so many requests over the years for exactly this kind of product that I'm extremely excited to be able to talk about it and share just how awesome it is. The PD guys have done a fantastic job making this easy to get started using and approachable. There are good docs to get you started, and even a step by step video walkthrough if you want to follow along from download to passing traffic. While this is some very cool news for all of you current F5 users that have been looking for a new test or dev setup, I'm equally as excited to see what everyone else is going to do with this. This is the first time that anyone that wants it will be able to have access to a BIG-IP device to see just how cool these things are, and I think that's something that shouldn't be overlooked. Suddenly everyone that wants to be able to dig in and see what this technology is all about, what we mean when we talk about ADCs and how powerful they are, how they can help shape your deployments with some pretty incredible business logic on the network, will be able to see it for themselves first hand, with only a small investment in time to do some setup and testing.

All of this is just the beginning, though. There is already talk of more to come. A trial is a fantastic first step but already the masses are clamoring for a more permanent solution. They want to be able to set up their test environments permanently and with less restrictions. Well that just might be coming too. I can't say when or give any details, but rest assured that the BIG-IP LTM VE concept will go far beyond just a free trial for people to play with. While a virtual edition of LTM doesn't offer nearly the same performance or some of the perks of having a full-blown hardware system, there are definitely places where a virtual LTM makes sense, and plans are already in the works to make sure that those needs can be met at some point down the road. Now is the time to get your hands on the LTM VE and get started.

So you're saying, "That's fantastic, but where do I start?". Here's what you'll need to do to get yourself going:

First, download the trial. - https://www.f5.com/trial/
This will require you to log in, so get registered if you're not already. Once you log in you'll be able to select the trial version you want (there's only one for now) and download it.

Next, generate a license. - https://www.f5.com/trial/secure/generate-eval-key.php
You won't be able to do anything with the shiny new image you just downloaded until you get it licensed, so make sure you go get yourself a trial license so you can make the blinky lights blink and the bits flow. This license will get you access to the core LTM functionality in the trial, so you'll have access to all sorts of cool toys like iControl and iRules.

After that, you'll need to get your new BIG-IP LTM VE system configured just like you normally would configure a BIG-IP. You'll need self IPs and VLANs and all that. If you're not familiar with setting up BIG-IP then check out one or both of these:

- Randy Reichenbach's (F5 PD) easy to follow "Getting the LTM VE Trial Running" walkthrough video.
In this video Randy walks you through the setup process all the way from downloading the image to licensing to configuring the BIG-IP itself to passing traffic. This is a great place to start for anyone, whether you're a beginner or a pro looking for a refresher. Things move kind of fast so if you have any questions or don't catch something in the video, the best place to go first is the release notes.

- The BIG-IP Virtual Edition Trial Release Notes.
The release notes have just about all the information you could want about the VE. What platforms it runs on, what the restrictions of the trial version are, what to do to get it configured, known issues, etc. This is a one stop shop of useful information that everyone downloading the trial should check out. If you've got questions about how to get started, what this thing can do or just about anything having to do with the trial, I'd check here first.

Once you've got things up and running, you've got your own personal BIG-IP playground. Now it's time to see what it can do, and I can't recommend a better place for finding cool things to try out than DevCentral. Test out some iRules, play with a custom monitor or two, write some TMSH scripts...whatever it is you've been eager to play with but were afraid to try on your production system (or maybe you didn't have a system at all), now's the time.

Be sure to check back with the BIG-IP LTM VE section on DevCentral often to keep up with updates. There's a forum here for discussing and sharing ideas, tips or problems with the trial. It will hopefully be a great place to see what other people are up to and figure out issues if you're having them, or even just to share feedback of your LTM VE experience. Heck, there has already been some interesting info from some users there talking about how, even though it's not technically supported, they've gotten the LTM VE running in virtualbox, VMware Player, fusion, etc. So there's already some interesting talk there that you won't want to miss out on. There will also be a steady stream of content showing off some of the things you can do with the trial, both from the DevCentral team and other folks inside F5 who are eager to show off what they're up to with this exciting new release. Be sure to let us know what you think about the trial VE or what you're up to with it, the more we hear back from the people using it the better we can make the next one.

Increasing the F5 Management Pack's Responsiveness for Statistics Collection

Blog Author — Tue, 19 Jan 2010 12:14:46 GMT

With the recent release (v1.6.1.212), the F5 Management Pack has an improved responsiveness to device statistics collection. By default, the timer interval for collecting stats is set to 30 seconds, but this value now can be easily overwritten in the (SCOM 2007) F5 Management Pack, as a rule override. The lowest value the timer interval could be set, theoretically, is 3 seconds. There is an internal safeguard in the F5 device (Big-IP) against lowering this threshold and putting unnecessary load on the device CPU, for collecting stats. In the real world stats would come, on average, every 5-10 seconds, even with the 3 second interval setting.

You may probably ask why would we even want to go that low timer interval for collecting statistics, normally, from an F5 device. One possible answer would come from the perspective of a relatively recent data center orchestration (DCO) scenario, that F5 Networks performed, of migrating a live application (web server) in a distributed Big-IP GTM/LTM environment. A simplified view of the scenario would be the following: Big-IP/GTM "A" handles high traffic in the middle of a hot summer in TX, where power plan / consumption / usage is extremely taxing. During these hot days, Data Center needs to migrate live applications to Big-IP/GTM "B" across geographical zone, to a better power plan. Big-IP-B will spin-up real-time virtual pool members to host the live app and handle/persist client connections, while Big-IP-A monitors existing client connections (for bleed-out) and retires the pool members ASAP, for reducing power consumption. The counter responsiveness improvement would play a big role in the bleed-out monitoring scenario, when the triggers for disabling pool members on Big-IP-A as soon as possible, should act immediately.

So, here are the steps for creating the rule overrides for the statistics collection interval, in the F5 Management Pack (we will use the "Device CPU Temperature" collection rule as an example):

Stage One: Find the collection rule we need to collect:

In the SCOM 2007 Management Console enter the Authoring view by selecting the Authoring link.

Select Authoring > Management Pack Objects > Rules.

In the Look for text box, enter “F5” and click Find Now: the F5 rules installed by the F5 Management Pack will appear, where we can further search for the rule we need to override.

To search specifically for the Device CPU Temperature rule, in the Look for text box enter a more specific search expression, such as "CPU Temp" (if the Clear button has not been pushed yet, the search would further on narrow the scope of the items looked for, compared to previous searches).

Stage Two: Enable the collection rule (if it's not enabled by default):

Select the Device CPU Temperature rule, right-click and select Overrides > Override the Rule > For all objects of type: F5 Management Pack Monitoring Service.

In the Override Properties window choose to create a new Management Pack for targeting the overrides (if not already created), by clicking the New button near the bottom of the screen. The Create a Management Pack window will appear.

Enter the name “F5.Override.ManagementPack”, click Next, then Create to finish creating the new management pack for the overrides.

In the Override Properties window examine the parameters listed. We have two override parameters. Check the Override property for the Enabled parameter and make sure its Override Setting value is set to True (compare it against the Effective Value, which may be False).

Under Select the destination management pack, choose the management pack you just created (i.e. F5.Override.ManagementPack).

Click OK.

Stage Three: Overriding the Interval parameter for the collection rule:

Select the Device CPU Temperature rule, right-click and select Overrides > Override the Rule > For all objects of another type....

In the Select a Target Type window, select the View all targets radio-button and in the Look for text box type "F5" to have all F5-related target entity types listed and from the target list choose F5 Device CPU, for example (or we could use any other F5 entity that is derived from the F5 Object type, such as F5 Device). Click OK.

In the Override Properties window choose to create a new Management Pack for targeting the overrides (if not already created), by clicking the New button near the bottom of the screen. The Create a Management Pack window will appear.

Enter the name “F5.Override.ManagementPack”, click Next, then Create to finish creating the new management pack for the overrides.

In the Override Properties window examine the parameters listed. We have two override parameters. Leave the Override property for the Enabled parameter unchecked. Check the "Override" property for the Interval parameter and set the Override Setting property to 10 seconds (this will have the collection of the device CPU temperature running every 10 seconds).

Under Select the destination management pack, choose the management pack you just created (i.e. F5.Override.ManagementPack).

Click OK.

Note:

It is generally not a good idea to save the rule overrides to the default management pack because it creates dependency problems during un-installation. As a good practice, make sure you do not apply any overrides to the default management pack. The Enforced checkbox in the Override Properties window is used partially to raise the priority of an override rule over other similar override rules.

We can review, edit, or delete the override for the rule. Right-click the rule in the Authoring view and select Overrides Summary, then Edit.

Once the rule overrides have been created the F5 Management Pack will propagate them to the F5 device in a matter of minutes. The F5 Management Pack crawls for changes in the rule overrides configuration every 5 minutes. To speed up the counters subscription, we can force the F5 Management Pack to recalculate them and force the rules to the target F5 devices (Big-IP). To do either of the following procedures:

1. Using the command prompt

Open up a command prompt and type the following:

cd "%ProgramFiles%\F5 Networks\Management Pack"

Then force the F5 Management Pack to refresh the rules:

f5mpcmd.exe /RefreshCollectionRules

2. Running the Refresh Collection Rules action

In the SCOM 2007 Management Console navigate to Monitoring > F5 Networks > F5 Actions. In the Actions pane on the right, click on Refresh F5 Device Collection Rules action and run the related task.

Clustering the F5 Management Pack with a SCOM 2007 RMS Cluster

Blog Author — Mon, 18 Jan 2010 11:50:09 GMT

Starting with version 1.6.1.212, the F5 Management Pack supports clustered Root Management Server (RMS) configurations with SCOM 2007 (including SCOM 2007 R2), This article will describe the step-by-step procedure of installing and configuring the F5 Management Pack with a SCOM 2007 R2 RMS cluster, running on a Windows Server 2008 R2 platform with Microsoft Cluster Service (MSCS).
For the purpose of this article we'll assume the following system configuration ready, for integrating the F5 Management Pack with the SCOM 2007 RMS cluster:

Windows Server 2008 R2 fail-over cluster with 2 cluster nodes

System Center Operations Manager 2007 R2, configured as RMS cluster

Cluster shared storage with at least one active (online) NTFS partition

1. Installing the F5 Management Pack in a SCOM 2007 RMS Cluster

Figure 1 illustrates the SCOM 2007 R2 RMS fail-over cluster on Windows Server 2008 R2, prior to configuring the F5 Management Pack to integrate with the RMS cluster:

MP-LAB-CLUSTER: the fail-over cluster server

MP-LAB-RMS: the SCOM 2007 R2 RMS cluster group

MP-LAB-CN1, MP-LAB-CN2: the fail-over cluster nodes, each running SCOM 2007 R2 as Root Management Servers.

Figure 1: SCOM 2007 R2 RMS fail-over cluster on Windows Server 2008 R2

Keep in mind that for each cluster node in the RMS cluster, where you plan to deploy the F5 Management Pack, you will need a separate F5 Management Pack license.

Here are the steps of installing and configuring the F5 Management Pack with the SCOM 2007 RMS cluster:

Install the F5 Management Pack on the primary cluster node (the one owning the SCOM 2007 RMS cluster group), in our case MP-LAB-CN1 (see Figure 1).

Install the F5 Management Pack on the secondary cluster nodes (there's no need to fail-over to the other nodes when installing the F5 Management Pack).

Using the Failover Cluster Manager management console, add the F5 Monitoring Service to the SCOM 2007 RMS cluster group, as a new cluster resource (generic service): right click on the SCOM 2007 RMS cluster group (MP-LAB-RMS) > Add a resource > 4 - Generic Service > and select the F5 Monitoring Service.

Make sure the F5 Monitoring Service shows up "Offline" in the "Other Resources" section of the SCOM 2007 RMS cluster group (see Figure 2).

Set the cluster resource dependencies for the F5 Monitoring Service (see Figure 3): right click on the F5 Monitoring Service cluster resource > Properties > Dependencies > set the dependency on the SCOM 2007 RMS cluster group AND the "System Center Management" service (the SCOM 2007 R2 Health Service).

Set the cluster node owners for the F5 Monitoring Service cluster resource (see Figure 4): right click on the F5 Monitoring Service cluster resource > Properties > Advanced Policies > check the cluster nodes owning the F5 Monitoring Service (basically the cluster nodes that have the F5 Monitoring Service installed, through Step 1 and 2).

Bring the F5 Monitoring Service cluster resource online: right click on the F5 Monitoring Service cluster resource > Bring this resource online.

Figure 2: Adding the F5 Monitoring Service as a generic service cluster resource to the SCOM 2007 RMS cluster group

Figure 3: Setting the dependencies for the F5 Monitoring Service cluster resource

Figure 4: Setting the owner cluster nodes for the F5 Monitoring Service cluster resource

Figure 5: The F5 Monitoring Service running within the SCOM 2007 RMS cluster group

2. Managing the F5 Management Pack in a SCOM 2007 RMS Cluster

Once the F5 Management Pack is installed and configured within the SCOM 2007 RMS cluster, discovering and managing F5 devices through the F5 Management Pack works exactly the same way, as in a standalone SCOM 2007 RMS environment. Always use the active cluster node (the one owning the RMS cluster group) to perform F5 Management Pack specific tasks. Needless to say, the SCOM 2007 Management Console should be configured to connect to the SCOM 2007 RMS cluster group, rather than the individual cluster nodes (Figure 6).

Figure 6: Connecting the SCOM 2007 Management Console to the RMS cluster group

Operations Manager Debugging Part II: Workflows

Blog Author — Fri, 13 Nov 2009 14:38:29 GMT

Workflows are the fundamental mechanism by which Operations Manager gets things done. Workflows, in essence, are individual pieces of code (modules) that are chained together. For a primer on workflows and modules, I recommend the AuthorMPs site. Workflows remind me of Powershell pipelines or to a lesser extent, linux command piping. Individual commandlets (the equivalent of OpsMgr modules) can be mixed together in different ways to accomplish a more complex task (the equivalent of an OpsMgr workflow) by piping output from one to another. As long as the output from one commandlet can be processed by the next commandlet in the pipeline, you can chain them together.

One big difference between developing pipeline-aware commandlets and developing OpsMgr modules and workflows is the potential for debugging. With Powershell, you can use the full debugging power of Visual Studio (or whatever editor you use) by simply setting breakpoints in your commandlet and attaching to the host running powershell (usually powershell.exe). Workflow debugging is certainly possible but not nearly as easy. There are several reasons for this:

If you’re not using custom-made modules, the running code is a black box. If the module does not already have the ability to output tracing information, there is no way to add it.

You can’t just run a workflow from a debugger. OpsMgr is complete control of when it starts and ends, so you must do this indirectly.

If you are using custom-made modules, you need to find the OpsMgr process that they get loaded into in order to debug them.

One of the most common problems in developing workflows is trying to figure why (or why not) a data item is making it through certain parts of the workflow such as condition detection modules. In order to avoid “trial and error” development, you need some way to debug the workflows. I’ll outline a few options below for troubleshooting this kind of scenario below.

Option 1: Attach a Debugger

If you’ve developed a custom module, you can step through the code by doing the following:

Get OpsMgr to start running your workflow. For example if you are debugging a monitor, make sure the monitor is enabled and it’s datasource is receiving data.

Open the code in your debugger (Visual Studio in this example)

Attach to the OpsMgr process running your workflow. This is the tricky step. The process is MonitoringHost.exe, but there may be several. You want the one that is running managed code (see the Type column in screenshot below) with the credentials of the Operations Manager action account. If it is still not obvious which process to attach to, you’ll have to attach to each one and see if your DLL module is loaded. Use the module window (Ctrl+Alt+U) to list the DLLs once you attach to the process. Once you see your DLL, you’re read to debug!

On a side note, I (or one of our team members) may do a blog post in the future on creating managed code modules.

Option 2: Use the Operations Manager Trace Files

Honestly I would leave this as a last resort, but the trace files can be useful beyond workflow debugging (such as script debugging). See #8 on my top 10 list for instructions and comments on this.

Option 3: Use the F5 Workflow Tracer

Inspired by the workflow tracer that was #6 in my top 10 list, our team has built a debugging module that can be used to trace the output of any module in your workflows. We used the same configuration elements as the workflow tracer and added support for data item batching, robust acknowledgements, and other minor improvements.

The module can be placed after any module in your workflows, and reports on the data item that come out of that module. Here is an example from the F5 Monitoring Log:

Element: F5.PRO.PerformanceMonitorType (note: this tells me which workflow is being traced)
Stage: After COUNTER TYPE detection (note: this tells me which module is being traced)
Target: {4447e45e-9f65-e01d-2c17-24dee61fb1d9}

<?xml version="1.0" encoding="utf-16"?>
<DataItem type="System.Performance.LinkedData" time="2009-11-12T12:32:46Z" sourceHealthServiceId="fa056941-2b22-7d8e-a34d-7604e1347f99">
<ObjectName>LTM Pool Member</ObjectName>
<CounterName>Server - Current Connections</CounterName>
<InstanceName />
<IsNull>false</IsNull>
<Value>0</Value>
<ManagedEntityId>{29c7fc90-0f44-b70c-9f88-5a7ee6d13a40}</ManagedEntityId>
<RuleId>{0e799c3d-b8d0-bce0-259b-f9025d507996}</RuleId>
</DataItem>

This module can be downloaded here. You’ll need to import the included MP into OpsMgr and place the included DLL in the Operations Manager program files folder. Instructions for installation and configuration are the same as the original workflow tracer, except with our MP and DLL.

Another barrier to debugging OpsMgr becomes apparent if you want to use this module to debug a sealed workflow (one that’s in a sealed MP and is not marked public). There is no way to simply insert a module of your own in such a workflow since you cannot change a sealed MP. The workaround goes like this:

Create an override for the workflow that disables it

Export the MP that the workflow is in to XML format (use the #1 tool on my top 10 list).

Get the XML for the workflow from the exported pack and create a copy of it in your own, new, MP. If the workflow references other elements marked “internal” to the MP, you will have to create copies of those as well.

Insert the debugging module wherever you’d like in your new workflow.

Import your new pack

Simple as that ! :)

To help understand this process, I’ve created an override pack for our own F5 PRO-Enabled Management Pack that replaces one of our monitors with one that includes the debugging module between each module in the workflow. I realize the pack looks strange as formatted in our wiki – just copy and paste from the <xml> element on down to </ManagementPack>.

Option 4: Use the Operations Manager Workflow Analyzer

I was both elated and saddened by the brand new workflow analyzer included in the new authoring console. Elated because this tool rocks, and saddened because it really would have helped to have it earlier :). This tool does the following things:

Shows you all the workflows registered in OpsMgr as well as their state

Technical note: Internally this is basically running the “Show Running Rules and Monitors” task that is already available via the UI

Allows you trace any of the workflows

Technical note: Internally this is overriding the TraceEnabled property on the workflow and hooking into OpsMgr’s built in tracing capabilities. I’ve looked for a way to override this property through the UI, and it does not seem possible.

Allows you do an ‘analysis’ of the workflow. This was not very interesting to me – just shows you some basic properties and running instances.

You can increase the amount of information printed out in the trace if you turn on verbose logging (see Option 2 above).

The main drawback of this tool is that it’s hooking into trace calls made by the OpsMgr modules so you’re not going to get this level of detail if you’re trying to debug a custom-built module. It may be possible to write to the same mechanism, but I have not looked into that yet.

Happy Debugging!

<< Back to Part I

Operations Manager Debugging Part I: Top 10 Tools for Developing and Debugging Management Packs

Blog Author — Wed, 04 Nov 2009 11:42:28 GMT

During that last year and a half that I’ve been on the F5 Management Pack team, we’ve had to do quite a bit of Systems Center Operations Manager debugging. I think it is certainly possible to develop management packs without needing to delve into Operations Manager internals at all, but much of what our pack does is outside the norm. For example:

We have a custom-built agent on the Big-IP LTM box that our management pack talks to using a proprietary protocol

We’ve built custom OpsMgr workflow modules in order to meet our performance objectives

We just released a PRO pack which is fairly cutting-edge technology so there is very little documentation to go on

In this series of posts, I’ll share some of the techniques we’ve used for debugging and developing management packs. So, without further adieu, I present part I – the top 10 most useful tools.

10. Runtime Health Explorer

Troubleshooting state change problems can be tricky. There are several ways to view health and monitor states of your discovered instances, but not all in one place. This tool does just that, and has a synchronization feature as well.

9. DIR

Yes, the lowly DIR command. Simple, yet powerful.

Try running this command:

c:\Program Files\System Center Operations Manager 2007\Health Service State\>dir *.PS1 /s /b

If you have scripts included in your management pack, they will be placed in a subfolder of this directory before they are executed. Looking at these scripts can help troubleshoot selectors and script caching. A great explanation of this technique can be found here.

8. Operations Manager Tracing Configuration Scripts

By turning verbose logging on, you can see an amazing amount of detail around what OpsMgr is doing. I have found this technique most useful when debugging scripts, especially ones using the new powershell module. For example, you may have an error in your script because you forgot to add a snapin and the normal task window will not show you what the problem is. The TracingGuidsScript.log file will show the error.

7. Module Explorer

It is critical to understand what modules are available to you when developing new workflows. This tool shows you all the modules available in your current OpsMgr environment.

6. Workflow Tracer

If you have a workflow with several ConditionDetection modules and you’re losing data items somewhere, this tool is extremely useful. This module can be inserted in any of your workflows (between each module in a workflow is the best for debugging) and it will log the data items that are passing through. This is also useful for understanding what the raw data items look like.

Part II of this series will be a further exploration and extension of this concept.

5. Lineage Explorer

Much thanks to SCOMNIVORE for three of the tools on this list, including this one. This tool is similar to the Module Explorer but allows you to view class and relationship type lineage as well as module lineage.

4. Reflector

Operations Manager is not open source, but there is an enormous amount of .NET code that can be decompiled and examined. One of the posts in this series (#3 I think) will cover this topic. However, I’ll leave you with this little snippet from Microsoft.EnterpriseManagement.HealthService.DLL:

public enum ModuleType { DataSource, Condition, ReadAction, WriteAction }

3. SQL Server Query Analyzer

A good understanding of the OperationsManager and OperationsManagerDW databases goes a long way when troubleshooting MP issues. There are also an enormous number useful OpsMgr SQL queries available out there. This guy covers a couple dozen in just one post and our own Julian Balog just posted a couple queries for troubleshooting statistics.

2. System Center Operations Manager 2007 R2 Authoring Resource Kit (Authoring Console)

I wasn’t incredibly impressed with the original authoring console, but the just-released new version looks great. It also includes several new tools for debugging. This is the best (only?) tool that targets the beginner to intermediate level MP author. I’ve used the console several times to author a module and then save it to XML to learn about the XML schema and how things are supposed to be put together.

1. MP to XML Dumper

This simple tool is #1 because the best way to learn and debug MPs in my experience is to look through other MPs, especially the built-in OpsMgr ones. This can also be done with a powershell one-liner from the OpsMgr powershell console:

Get-ManagementPack | Export-ManagementPack –Path “C:\packs\”

Honorable Mention:

MP Viewer: This didn’t make the list because the authoring console has all the same features and much more. The one nice thing about this tool however is that you can open a XML or MP file directly without needing to connect to OpsMgr, resolve references, etc.

If you have a great tool that is not on this list, please post a comment. I’ll make it “Top 11” if there’s something really good that I missed!

>> Part II: WorkFlows

Licensing Feature Added to the F5 Networks Management Pack

Blog Author — Mon, 02 Nov 2009 19:08:44 GMT

Licensing and registration has been officially added to the process for downloading and activating the F5 Networks Management Pack software package. While we realize the process may create a minor inconvenience for some people, it will go a long way in helping us support you, the loyal customer, in the future. The information we collect through the process will help give us a clearer picture of the Microsoft Systems Center (Operations Manager, Virtual Machine Manager, etc) environment in which the Management Pack is being installed.

Along with this release, there is new documentation available here: Licensing The F5 Management Pack. Along with this documentation is a tutorial video on how to proceed through registration and activation: Registration and Licensing of the F5 Networks Management Pack (temporary link).

I highly recommend you follow the instruction on that page and download the latest version of our Management Pack. If you have any questions, feel free to comment on this blog or visit our forums.

Authoring an F5 Management Pack PowerShell Agent Task in SCOM 2007 R2

Tue, 27 Oct 2009 18:46:51 GMT

1. Overview

This article describes a step-by-step procedure on authoring a custom PowerShell agent task targeting the F5 Management Pack in System Center Operations Manager (SCOM) 2007 R2. I will use the Authoring Console shipped with the System Center Operations Manager 2007 R2 Authoring Resource Kit (downloadable here) to create the override management pack defining the agent task. The procedure for authoring an agent task with SCOM 2007 SP1 would be similar, with the difference that the authoring tool would be the Authoring Console supporting SCOM 2007 SP1 (downloadable here) and some of the module types targeted by the agent task would be different.

The agent task that we’re going to create here for the F5 Management Pack as an example, will create an array of LTM Pool Member objects for an F5 device, using PowerShell. The same authoring concept can be extended to any generic scenario for creating a PowerShell agent task in SCOM 2007 R2.

2. Preliminary Steps

Make sure the F5 Management Pack is correctly installed on the SCOM 2007 R2 Root Management Server. In addition, you'll need to have the System Center Operations Manager 2007 R2 installed on the management server where you plan to author the override management pack described in this article, for implementing the custom agent task. For testing purposes it is also recommended to have at least one F5 device discovered, in order to run the custom agent task that we're creating.

Also, for a good understanding of the agent task implementation described in this article, some minimal SCOM Management Pack authoring skills are required as well as basic PowerShell programming skills. Since the agent task is targeting an F5 device and attempts to change the device configuration, by adding LTM Pool Members, make sure you have the necessary rights to perform this action on the F5 device.

For more information on F5 Management Pack related topics visit the F5 Management Pack Solution page here.

3. Authoring the Agent Task

At a high level here are the basic steps for creating the PowerShell agent task for adding an array of LTM Pool Members:

Create an override management pack (F5.Override.ManagementPack) referencing the F5 Management Pack. This override management pack will implement the custom PowerShell agent task.

Create a WriteAction Module Type (F5.WriteActionModuleType.AddLtmPoolMembers) containing the PowerShell script and override parameters needed for the agent task.

Create an Agent Task (F5.Task.AddLtmPoolMembers) as the actual task that would target the F5 LTM Pool object (for creating the LTM Pool Members in). This agent task will use (implement) the F5.WriteActionModuleType.AddLtmPoolMembers write-action module.

Save the F5.Override.ManagementPack and import it in SCOM 2007 R2.

The following sections will describe the step-by-step procedure for authoring the override management pack implementing the custom agent task for adding an array of LTM Pool Members.

3.1. Creating the Override Management Pack

Open the SCOM 2007 R2 Authoring Console and go to File > New... to create a new override management pack. Or you can open an existing override management pack if you already have one, referencing the F5 Management Pack. For the purpose of this article we'll just create a new override management pack.

In the New Management Pack window select Empty Management Pack and name the management pack F5.Override.ManagementPack, for example. Click Next.

Enter a display name (such as F5 Override Management Pack) and a description (optional) for the override management pack. Click Create.

In the SCOM 2007 R2 Authoring Console go to File > Management Pack Properties and choose the References tab. Click Add Reference... and browse to the location of the F5 Management Pack file(F5_ManagementPack.mp) located in the F5 Management Pack deployment folder (which by default is %Program Files%\F5 Networks\Management Pack). Select F5_ManagementPack.mp and click Open. Click OK in the override management pack window properties, when done.

Figure 1 shows the references required by the override management pack. All of them should be created by default except the F5_ManagementPack, which we just added in the previous step.

In the SCOM 2007 R2 Authoring Console save the changes by going to File > Save (or clicking the Save icon in the toolbar), and save the override management pack to the desired location (in our case we're going to save it in the same location where the F5_ManagementPack.mp file is located).

Figure 1 - The override management pack references

3.2. Creating the WriteAction Module Type for the Agent Task

In the SCOM 2007 R2 Authoring Console select the Type Library section and under Module Types select Write Actions.

In the Write Actions panel right click > New... > Composite Write Action... and choose a unique identifier for the Write Action. We'll name this F5.WriteActionModuleType.AddLtmPoolMember. Click OK.

In the General tab enter the name for the module type ("LTM Pool Member Write Action Module Type") and description (optional) - see Figure 2.

Figure 2 - The Id, Name and Description of the Write Action Module Type

In the Member Modules tab click Add... and in the Choose Module Type window's list-view locate and select the Microsoft.Windows.PowerShellWriteAction type. In the Module ID text box enter F5.WriteAction.AddLtmPoolMembers. Click OK.

In the Member Modules tab select the F5.WriteActionModuleType.AddLtmPoolMembers and click Edit.

In the Configuration window enter AddLtmPoolMembers.ps1 for the ScriptName value and 60 for the TimeoutSeconds value.

In the Configuration window select ScriptBody and click Edit... An external editor window will open (Notepad) and inside the <ScriptBody></ScriptBody> tags copy/paste the PowerShell code shown in Code Snippet 1. This is the PowerShell script referenced by AddLtmPoolMembers.ps1 and we’ll refer to this script in the following sections simply as AddLtmPoolMembers.ps1. This is the PowerShell script that eventually gets executed when the “Add LTM Pool Members” task will run.

param($mgmtAddress, $poolName, $targetAddresses);

# validate input parameters

if ([String]::IsNullOrEmpty($mgmtAddress)) { throw "Management address required." }

if ([String]::IsNullOrEmpty($poolName)) { throw "Pool name required." }

if ([String]::IsNullOrEmpty($targetAddresses)) { throw "Target addresses required." }

# trap and re-throw exceptions (if any), to correctly reflect task status on errors

trap [Exception] { throw $_; }

# get the F5 device connection info

$token = Get-F5.DeviceConfigurationToken $mgmtAddress;

# make sure we have a valid device configuration token

if ([String]::IsNullOrEmpty($token))

{

throw "Invalid device configuration token. Run the Set-F5.DeviceConfigurationToken task and try again."

}

# split the comma separated value string into an array of items

$targetAddresses = $targetAddresses.Split("`,");

foreach ($member in $targetAddresses)

{

Write-Host "Adding LTM Pool Member $member to LTM Pool $poolName...";

# add the LTM pool member

#Add-F5.LTMPoolMember -DeviceIP $mgmtAddress -Pool $poolName -Member $member -Token $token;

Write-Host "LTM Pool Member $member has been successfully added to the LTM Pool $poolName."

}

Code Snippet 1 - The PowerShell script within the <ScriptBody> tags of the write action module type (AddLtmPoolMembers.ps1 )

In the same external editor window (Notepad), immediately after the closing </ScriptBody> tag (the order is very important!) add the XML code describing the PowerShell snap-ins required by the script (see Code Snippet 2). These snap-ins are F5.Networks.ManagementPack and F5.Networks.iControl and are installed with the F5 Management Pack.

<SnapIns>

<SnapIn>F5.Networks.ManagementPack</SnapIn>

<SnapIn>F5.Networks.iControl</SnapIn>

</SnapIns>

Code Snippet 2 – The PowerShell snap-ins required by the AddLtmPoolMembers.ps1 script

in the same external editor window (Notepad), immediately after the closing </SnapIn> tag (the order is very important!) add the XML code describing the parameters needed by the script (see Code Snippet 3).

<Parameters>

<Parameter>

<Name>mgmtAddress</Name>

<Value>$Config/mgmtAddress$</Value>

</Parameter>

<Parameter>

<Name>poolName</Name>

<Value>$Config/poolName$</Value>

</Parameter>

<Parameter>

<Name>targetAddresses</Name>

<Value>$Config/targetAddresses$</Value>

</Parameter>

</Parameters>

Code Snippet 3 – The parameters required by the AddLtmPoolMembers.ps1 script

close the external editor window (Notepad) when done and save the content when prompted.

back in the Member Modules tab of the Write Action Module Types property window select the F5.WriteActionsModuleType.AddLtmPoolMembers and set the NextModule field to Module Output. The settings on the Member Modules tab should look similar to the ones shown in Figure 3.

Figure 3 - The Write Action Member Module settings

Go to the Configuration Schema tab and add the parameters of the Write Action Module Type, as shown in Figure 4.

Figure 4 - The parameters of the Write Action Module Type configuration schema

Go to the Overridable Parameters tab and add the overridable parameters of the Write Action Module Type, as shown in Figure 5.

Figure 5 - The overridable parameters of the Write Action Module Type configuration schema

Go to the Data Types tab, check the This module outputs data checkbox and set the Data Type to System.BaseData (see Figure 6).

Figure 6 - The Output Data type of the Write Action Module Type

Go to the Options tab and set the execution options for the Write Action Module Type as shown in Figure 7.

Figure 7 - Execution options for the Write Action Module Type

3.3. Creating the Agent Task

In the SCOM 2007 R2 Authoring Console select the Health Model section and under Tasks select Agent Tasks.

In the Agent Tasks pane right click > New > Custom Task... and choose a unique ID for the agent task, for example F5.Task.AddLtmPoolMembers.

In the Agent Task Properties window select the General tab and enter a name for the agent task ("Add LTM Pool Members") and a description (optional), see Figure 8). Set the Target of the Agent Task to F5.LTM.Pool (see Figure 9).

Figure 8 - Agent Task general properties

Figure 9 - Setting the Target for the Agent Task to F5.LTM.Pool

In the Configuration tab click on the Browse for type... hyperlink and in the Choose module type window look for the F5.WriteActionModuleType.AddLtmPoolMembers, defined in the previous section (see 3.2. Creating the WriteAction Module Type for the Agent Task). In the Module ID text box enter a module identifier, for example F5.WriteAction.AddLtmPoolMembers (see Figure 10). Click OK.

Figure 10 - Setting the Module Type for the Agent Task to F5.WriteActionModuleType.AddLtmPoolMembers (see section 3.2)

In the Configuration tab verify that the parameters defined for the F5.WriteActionModuleType.AddLtmPoolMembers are correctly shown. Set the following values for the parameters (see Figure 11).:

mgmtAddress = $Target/Host/Property[Type="F5_ManagementPack!F5.Device"]/MgmtAddress$

poolName = $Target/Property[Type="F5_ManagementPack!F5.LTM.Pool"]/PoolName$

timeoutSeconds = 60

Click OK to save the Agent Task and conclude the configuration.

At this point the custom agent task should be properly set up. Save the F5.Override.ManagementPack (File > Save in the SCOM 2007 R3 Authoring Console). The last step would be to import the F5.Override.ManagementPack in SCOM 2007 R2. Open the SCOM 2007 R2 Management Console, go to the Administration section, select Management Packs, right click > Import Management Packs > Add > Add from disk... and browse to the location where the override management pack is saved, select and click Install.

5. Running the Agent Task

When running F5 device related agent tasks in SCOM 2007, we need to have an appropriate device configuration token created (or cached) for the specific device, in order to have a successful task run. This behavior ties into the "Authorization Roles and Groups" model of the F5 Management Pack, i.e. supporting various authorization roles and groups via Microsoft SCOM 2007, Active Directory, and F5 BIG-IP’s user authentication model. The F5 Management Pack includes built-in roles for discovery and device configuration and Microsoft Windows and Active Directory accounts can be correlated with BIG-IP user accounts via a one time device authentication (performed through an F5 Management Pack agent task action).

Figure 11 - Running the Set F5 Device Configuration Token agent task

In other words, prior to run any configuration related agent task against an F5 device (such as Rediscovery, Enable/Disable LTM Virtual Server, Enable/Disable LTM Pool Member, etc.) we have to make sure we have a valid device configuration token. This is accomplished by running the "Set F5 Device Configuration Token" agent task. In the F5 Device diagram view, select the F5 device > right click > F5 Device Tasks > Set F5 Device Configuration Token), where we’ll have to provide the device authentication credentials for the appropriate overrides (see Figure 11). These credentials will be mapped to the account running the agent task(s), making up a 'device configuration token'. This is a one-time deal and we don't have to set the device configuration token with every agent-task run, assuming that we will use the same (Windows/Active Directory) account running the agent task(s). If the device configuration token has been set up successfully, we should get an output similar to Figure 12, at the end of the Set F5 Device Configuration Token task run.

Figure 12 - The output for a successful run of the Set F5 Device Configuration Token agent task

Once we have the device configuration token (credentials) cached, we can finally run the Add LTM Pool Members agent task: in the F5 Device diagram view, expand the F5 device hierarchy and navigate to the desired LTM Pool, right click > F5 LTM Pool Tasks > Add LTM Pool Members. In the agent task property window set the appropriate override values (see Figure 13). The target LTM Pool Member addresses should be entered as a comma-separated value string, e.g. similar to “10.10.10.1:80, 10.10.10.2:80”.

Figure 13 - Running the Add LTM Pool Member agent task

For a successful run of the Add LTM Pool Members task we should get an output similar to Figure 14.

Figure 14 - The output for a successful run of the Add LTM Pool Members agent task

6. Conclusion

Creating a custom PowerShell agent task in SCOM 2007 R2 is not a trivial procedure, but it’s definitely made possible by the SCOM 2007 R2 Authoring Console with an acceptable degree of flexibility. Scripting the agent task PowerShell script, targeting F5 Management Pack objects (such as LTM Pool Members, LTM Virtual Servers, etc.) would require a couple of PowerShell snap-ins shipped with the F5 Management Pack: F5.Networks.ManagementPack and F5.Networks.iControl. These snap-ins allow to some extent a native API interaction (wrapped around iControl), programming against the F5 Device objects (LTM Pool Members, LTM Virtual Servers, etc.).

You download the override management pack discurssed in this article here:

F5.Override.ManagementPack

A First Look at the F5 PRO-Enabled Management Pack for Microsoft Virtual Machine Manager 2008

Tue, 27 Oct 2009 00:05:51 GMT

The new PRO-Enabled management pack enables integration with Virtual Machine Manager (VMM) via PRO-Tips. This integration enables scenarios in which statistics and health events collected from Big-IPs by the core management pack are used to trigger actions by VMM.

One example of this kind of scenario is to use VMM to start up virtual machines when a traffic spike starts overwhelming your existing pool of servers. The same virtual machines can be suspended or stopped when the traffic load returns to normal. We’ve shown how the management pack can be used to enable these scenarios in a series of data orchestration videos. The new PRO-Enabled management gives the VMM administrator control and auditing of the actions that VMM is instructed to take.

When the PRO pack is installed, a new folder installed into Systems Center Operations Manager:

The new views available are:

F5 PRO Actions: Clicking on this view causes available PRO actions to show up in the actions pane. Currently the only action is “Discover PRO Pool Members”

F5 PRO Alerts: Any PRO alerts that you enable will be shown here. These alerts cause VMM to generate PRO tips.

F5 PRO Monitored Pool Members: This view shows all pool members that have PRO alerts enabled.

After the PRO Pack is installed, the PRO monitors need to be enabled and configured in order to start functioning. The configuration page for the “Add Server Resource” monitor is shown below:

By default, the configuration of the monitor is as follows:

The monitor is disabled

The Big-IP statistic that it monitors is LTM Pool Member Server Current Connections

The monitor is in a healthy state when current connections are less than or equal to zero

The monitor is in a critical state when the current connections are greater than zero

The monitor goes to a critical state when it detects incoming connections because the pool member that is being monitored has been set up on the Big-IP to receive connections only when the primary pool is overwhelmed. The data orchestration videos go into more detail on this setup.

When the monitor goes into a critical state it creates a PRO alert and causes VMM to generate a PRO Tip which can be viewed from the PRO Tip window:

The VMM administrator can then choose to implement the tip or dismiss the tip. Implementing the tip will cause new server resources to spin up. Some details of what the tip actually does can be configured from within Systems Center.

The VMM administrator can also set options that govern how all PRO tips are handled:

PRO tips can be enabled/disabled by severity level

Tips can be configured to automatically run by severity level

Once the action has taken place, the Jobs windows from within VMM can be used to view the audit trail of the actions that were taken by implementing the PRO tip:

Please visit the F5 Management Pack downloads page to download the new PRO pack. In the coming weeks we’ll have more in-depth coverage of what you can do with the PRO pack, so stay tuned!

Introducing the F5 PRO-Enabled Management Pack for Microsoft Virtual Machine Manager 2008

Mon, 26 Oct 2009 10:43:05 GMT

I am proud to announce we have just released our latest addition to the F5 Management Pack, the F5 PRO-Enabled Management Pack for Microsoft Virtual Machine Manager 2008 (MS VMM). If you haven’t heard MS VMM is a member of the Microsoft System Center family that enables a host of different technologies and scenarios including Hyper-V management, VMWare instance integration, optimization of virtualized resources (PRO), and monitoring integration with System Center Operations Manager.

Building on F5 Networks’ BIG-IP Management Pack for System Center Operations 2008 R2, we’re excited to offer a rapidly deployable virtualization solution that will help you dynamically increase and decrease your hardware and software resources, based on network health and activity.

If you’ve been following our blog posts over the last few months, we’ve provided a set of videos and documentation explaining how to use the F5 Management Pack to add and remove pool members based on traffic through a pool or virtual server.

With the F5 PRO-Enabled MP, these scenarios are built into the Virtual Machine Manager console, along with PRO-Tip authorization to implement the resolution policy.

We’re also updating our base Management Pack (version 1.5.0) to address a number of key customer fixes, and improve reliability.

You can find the latest F5 Management Pack and F5 PRO-Enabled Management Pack Extension at our DevCentral download portal: http://devcentral.f5.com/Default.aspx?tabid=217

For questions and support issues, we continue to host direct support via the DevCentral Forums at: http://devcentral.f5.com/Default.aspx?tabid=53&afgroup=26

Specifically, you can use the “Support Issues” Forum for bugs and issues, in any of our Management Pack products.

I’d love to hear your feedback on our data center orchestration products.

Thank you for your continued support,

Stephen Fisher

Manager, Product Development

F5 Networks

Windows Installer XML and Managed Custom Actions

Blog Author — Mon, 12 Oct 2009 01:03:22 GMT

For those of you who have every worked with any installer technologies, you probably know how cumbersome it can be to do anything non-conventional. A lot of the higher ticket installer packages like InstallShield or WISE offer a robust set of options, but still lack the granularity that some installations may require, aside from costing money. There are a number of decent free options available such as Nullsoft Scriptable Install System (NCSIS), Inno Setup, and Windows Installer XML (WiX). A lot of simple installations can rely on these free technologies, or even the built in Windows Installer support provided with Visual Studio. With the F5 Networks Management Pack there are a lot of post installation steps. This is where WiX came in to save the day. Without having to learn a new scripting language, or abide by the constraints of the Visual Studio MSI builder, WiX gives us the flexibility to perform all needed post installation steps, while staying in the comfortable setting of XML. WiX is also built on top of standard MSI technology, which is well documented and supported by the community, and that makes understanding exactly how everything works a lot easier.

I’m not going to go into the basics of WiX because there is already plenty of good documentation on how to get an installer off the ground. What really interests me ( and unfortunately has very little documentation) is the proper way to set up managed custom actions, and more so, how to cover all of your all installation scenarios and schedule custom actions in sequence with proper rollback. José Almeida has written a very useful blog, which will actually cover a lot of the same material I am going to cover, so my recommendation is to read through that first, as it is very useful and will cover a large percentage of custom action scenarios you may wish to implement. Now actually getting a managed Custom Action working in WiX is not too hard. In fact, it only requires a few lines of XML. That being said, there is a right way and a wrong way, so while it’s easy, it can also be very complex. If you plan to have a a large scale installer (20+ components), I’d recommend fragmenting your script at this point into several WiX files so that you can more easily manage your installer down the road. One such fragment should be your custom actions and as well as the InstallExecuteSequence.

Creating the Custom Actions

In Jose’s blog post, he uses the installUtilLib as the binary in which to run as the Custom Action. The reality is that you can use just about any binary. You may want to embed the binary or perhaps use an executable that is included with the installation package itself. For the sake of this post, I’m going to ignore a few unwritten rules and go ahead and show you how to perform a custom action using a freshly installed binary. If you do choose to use an executable, the best way to do this is to implement it as a Quiet Execution Custom Action. If you are going to use a Quiet CA, you MUST add the following line to your wxs file:

<Binary Id=”wixca” SourceFile=”wixca.dll” />

This line will add the WiX Custom Action binary library so that you can use it as an entry point for your Custom Actions because we want to run our binary quietly, otherwise we will get a cmd window popping up in the middle of install. So for the sake of this example, we are going to say that there is a binary file being dropped by the installer called ConfigurationManager.exe. By itself, this executable does not do anything, but with the proper command line arguments, it will actually run the custom actions. So for one complete custom action we will have something that looks like this:

<CustomAction Id="CAWriteConfigurationFile.Install" Return="check" Execute="deferred" BinaryKey="wixca" DllEntry="CAQuietExec" />
<CustomAction Id="CAWriteConfigurationFile.Install.Command" Property="CAWriteConfigurationFile.Install" Return="check" Value=’”[INSTALLDIR]ConfigurationManager.exe” Action=”WriteConfigFile” Type=”Install”’ />

<CustomAction Id="CAWriteConfigurationFile.Install.Rollback" Return="check" Execute="Rollback" BinaryKey="wixca" DllEntry="CAQuietExec" />
<CustomAction Id="CAWriteConfigurationFile.Install.Rollback.Command" Property="CAWriteConfigurationFile.Install" Return="check" Value=’”[INSTALLDIR]ConfigurationManager.exe” Action=”WriteConfigFile” Type=”Install.Rollback”’ />

<CustomAction Id="CAWriteConfigurationFile.Uninstall" Return="check" Execute="deferred" BinaryKey="wixca" DllEntry="CAQuietExec" />
<CustomAction Id="CAWriteConfigurationFile.Uninstall.Command" Property="CAWriteConfigurationFile.Uninstall" Return="check" Value=’”[INSTALLDIR]ConfigurationManager.exe” Action=”WriteConfigFile” Type=”Uninstall”’ />

<CustomAction Id="CAWriteConfigurationFile.Uninstall.Rollback" Return="check" Execute="Rollback" BinaryKey="wixca" DllEntry="CAQuietExec" />
<CustomAction Id="CAWriteConfigurationFile.Uninstall.Rollback.Command" Property="CAWriteConfigurationFile.Uninstall.Rollback" Return="check" Value=’”[INSTALLDIR]ConfigurationManager.exe” Action=”WriteConfigFile” Type=”Uninstall.Rollback”’ />

These are the very basic actions you need to implement in order to make this all work properly. Each pair above represents one piece of the whole custom action, but I will briefly dissect one pair to help give you a better understanding of what is going on. In the first line we have the Id of the Custom Action, which you should name something that makes reading it easier. Next we have Return, which is set to check so that if the action fails, we know to rollback. Then we have Execute, which is set to deferred, which I will explain in a moment. The last 2 pieces are the BinaryKey and DllEntry, which are explained on plenty of pages, but basically it’s telling the WiX compiler to use the WiX Custom Action library and call the function CAQuietExec, which will execute the next line’s Value property, which was also purposefully named with a .Command at the end. So in the second line, we have the actual command that CAQuietExec will run. The Property field MUST be named the same as the Id field of the previous line. This is important, as it tells the .Command which Custom Action to ‘link’ to. The last piece of the second line is the Value property, which contains the actual command to run. In this case, we are running ConfigurationManager.exe and passing it options for Action and Type, which tell it what to do. It’s useful to handle the executable this way, as it gives us a framework in which to properly implement the Custom Action in all scenarios. The last note to make is that the Execute property on the Rollback actions are set to Rollback. This ensures that these commands ONLY run when Rollback has been initiated.

Sequencing the Custom Actions

Moving forward from here, the most important thing to remember is that the actual Custom Actions are not wrapped in the same type of transaction as the rest of the installer, which means it is imperative that you take care of any rollback on your own. If you do not do this, you could leave the system in a potentially bad state. That being said, other scenarios you might want to consider are Upgrade and Reinstall/Repair. If your Install action is the same for upgrade and repair, you just need to adjust the conditional in the next section; however, if they are different, you will have to create a set for each one like the ones listed above. Once you have laid out the Custom Actions you wish to run, the next piece of the puzzle is to lay out the InstallExecuteSequence. I personally like to keep things in the proper sequence, as it is easier to read and maintain later, so we will actually start with the Uninstall Sequence. For the actions listed above, it will look something like this:

<!-- Uninstall Action Sequence –>
<Custom Action="CAWriteConfigurationFile.Uninstall.Rollback.Command" After=’InstallInitialize’>Installed and REMOVE="ALL"</Custom>
<Custom Action="CAWriteConfigurationFile.Uninstall.Rollback" After='CAWriteConfigurationFile.Uninstall.Rollback.Command'>Installed and REMOVE="ALL"</Custom>
<Custom Action="CAWriteConfigurationFile.Uninstall.Command" After='CAWriteConfigurationFile.Uninstall.Rollback'>Installed and REMOVE="ALL"</Custom>
<Custom Action="CAWriteConfigurationFile.Uninstall" After=’CAWriteConfigurationFile.Uninstall.Command’>Installed and REMOVE="ALL"</Custom>

In the section above, you will notice that the Rollback is scheduled before the actual command. This is very important, as it ensures that the Rollback action gets scheduled before we run the actual command. Otherwise, if the command fails, the Rollback action was never scheduled and any changes made by the command will not be undone. This can leave the system in an unstable state, so it is important you make sure to put these in the correct order. You will also notice in the first line that we are scheduling the Uninstall Sequence to basically start after InstallInitialize. We can actually schedule these actions to run after ProcessComponents, but it must be scheduled before UnpublishComponents if you plan to use an executable dropped by the installer, as UnpublishComponents will make the binary unavailable to use for the Custom Actions. So now that we have the Uninstall Sequence set up, we will look at the Install Sequence:

<!-- Install Action Sequence –>
<Custom Action="CAWriteConfigurationFile.Install.Rollback.Command" After=’PublishProduct’>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CAWriteConfigurationFile.Install.Rollback" After='CAWriteConfigurationFile.Install.Rollback.Command'>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CAWriteConfigurationFile.Install.Command" After='CAWriteConfigurationFile.Install.Rollback'>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CAWriteConfigurationFile.Install" After=’CAWriteConfigurationFile.Install.Command’>NOT Installed or REINSTALL="ALL"</Custom>

Just as in the Uninstall Sequence, we schedule the Rollback action directly before the command itself to ensure proper rollback. You will also notice that the first part of the Install Sequence happens after PublishProduct, which ensures that the binary we wish to use is available for the Custom Action.

Multiple Managed Custom Actions

The information before this point can actually be found in pieces around the internet, but what I’m about to show you is not. Assume for a moment that we had not one, but 3 or 4 Custom Actions we wish to run using our ConfigurationManager.exe tool. Not only that, but they require us to run them in a particular order. Now we have an interesting case in which to look at. Creating the Custom Actions themselves is simple enough. Following the pattern above will get you on the right path in a few minutes, but to ensuring proper scheduling can be tricky. Let’s say we have 4 actions called CAUpdateVirtualMachineDatabase, CARegisterNonGacComponents, CARegisterNewVirtualizationSnapIn, and CAImportCustomDataCenterManagementPack. Including the CAWriteConfigurationFile action, we have have five, which must run in this order: CARegisterNonGacComponents, CAWriteConfigurationFile, CAUpdateVirtualMachineDatabase, CARegisterNewVirtualizationSnapin, and finally CAImportCustomDataCenterManagementPack, which relies on each of the previous 4 actions in some way. The Install Action Sequence would look something like this:

<!-- Install Action Sequence –>
<Custom Action="CARegisterNonGacComponents.Install.Rollback.Command" After=’PublishProduct’>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CARegisterNonGacComponents.Install.Rollback" After=’CARegisterNonGacComponents.Install.Rollback.Command'>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CARegisterNonGacComponents.Install.Command" After=’CARegisterNonGacComponents.Install.Rollback'>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CARegisterNonGacComponents.Install" After=’CARegisterNonGacComponents.Install.Command’>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CAWriteConfigurationFile.Install.Rollback.Command" After=’CARegisterNonGacComponents.Install’>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CAWriteConfigurationFile.Install.Rollback" After=’CAWriteConfigurationFile.Install.Rollback.Command'>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CAWriteConfigurationFile.Install.Command" After=’CAWriteConfigurationFile.Install.Rollback'>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CAWriteConfigurationFile.Install" After=’CAWriteConfigurationFile.Install.Command’>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CAUpdateVirtualMachineDatabase.Install.Rollback.Command" After=’CAWriteConfigurationFile.Install’>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CAUpdateVirtualMachineDatabase.Install.Rollback" After=’CAUpdateVirtualMachineDatabase.Install.Rollback.Command'>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CAUpdateVirtualMachineDatabase.Install.Command" After=’CAUpdateVirtualMachineDatabase.Install.Rollback'>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CAUpdateVirtualMachineDatabase.Install" After=’CAUpdateVirtualMachineDatabase.Install.Command’>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CARegisterNewVirtualizationSnapin.Install.Rollback.Command" After=’CAUpdateVirtualMachineDatabase.Install’>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CARegisterNewVirtualizationSnapin.Install.Rollback" After=’CARegisterNewVirtualizationSnapin.Install.Rollback.Command'>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CARegisterNewVirtualizationSnapin.Install.Command" After=’CARegisterNewVirtualizationSnapin.Install.Rollback'>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CARegisterNewVirtualizationSnapin.Install" After=’CARegisterNewVirtualizationSnapin.Install.Command’>NOT Installed or REINSTALL="ALL"</Custom>

<Custom Action="CAImportCustomDataCenterManagementPack.Install.Rollback.Command" After=’CARegisterNewVirtualizationSnapin.Install’>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CAImportCustomDataCenterManagementPack.Install.Rollback" After=’CAImportCustomDataCenterManagementPack.Install.Rollback.Command'>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CAImportCustomDataCenterManagementPack.Install.Command" After=’CAImportCustomDataCenterManagementPack.Install.Rollback'>NOT Installed or REINSTALL="ALL"</Custom>
<Custom Action="CAImportCustomDataCenterManagementPack.Install" After=’CAImportCustomDataCenterManagementPack.Install.Command’>NOT Installed or REINSTALL="ALL"</Custom>

So after looking at the XML above, the problem might seem a little easier to handle. There is one other small issue we need to address: What happens when one of the above acts differently upon repair or upgrade. So let us look at the upgrade scenario, specifically the Database related custom action. If a database is updated, typically we do not want to destroy the data, so we will want to ‘upgrade’ the database with the new fields or tables, etc. The Custom Action blob itself will look exactly like one in the previous section. The above blob will actually look different. We’ll call the upgrade CA CAUpgradeVMDatabase. Here’s how I would fit it in the existing Install Action Sequence:

<Custom Action="CAUpgradeVMDatabase.Upgrade.Rollback.Command" After=’CAWriteConfigurationFile.Install’>Installed and REINSTALL="ALL" and UPGRADEPATH</Custom>
<Custom Action="CAUpgradeVMDatabase.Upgrade.Rollback" After=’CAUpgradeVMDatabase.Upgrade.Rollback.Command'>Installed and REINSTALL="ALL" and UPGRADEPATH</Custom>
<Custom Action="CAUpgradeVMDatabase.Upgrade.Command" After=’CAUpgradeVMDatabase.Upgrade.Rollback'>Installed and REINSTALL="ALL" and UPGRADEPATH</Custom>
<Custom Action="CAUpgradeVMDatabase.Upgrade" After=’CAUpgradeVMDatabase.Upgrade.Command’>Installed and REINSTALL="ALL" and UPGRADEPATH</Custom>
<Custom Action="CAUpdateVirtualMachineDatabase.Install.Rollback.Command" After=’CAUpgradeVMDatabase.Upgrade’>NOT Installed or REINSTALL="ALL" and NOT UPGRADEPATH</Custom>
<Custom Action="CAUpdateVirtualMachineDatabase.Install.Rollback" After=’CAUpdateVirtualMachineDatabase.Install.Rollback.Command'>NOT Installed or REINSTALL="ALL" and NOT UPGRADEPATH</Custom>
<Custom Action="CAUpdateVirtualMachineDatabase.Install.Command" After=’CAUpdateVirtualMachineDatabase.Install.Rollback'>NOT Installed or REINSTALL="ALL" and NOT UPGRADEPATH</Custom>
<Custom Action="CAUpdateVirtualMachineDatabase.Install" After=’CAUpdateVirtualMachineDatabase.Install.Command’>NOT Installed or REINSTALL="ALL" and NOT UPGRADEPATH</Custom>
Dissecting what we have above, you’ll notice that the big difference between the two is the conditions in which they run. The Upgrade CA requires the product to be installed and the REINSTALL and UPGRADEPATH flags to be set. We do this so that we can schedule this single CA in the middle of the installation sequence. Not only does this make it easier to read, but if we only have the one CA that is affected by a major upgrade it doesn’t make much sense to write 4 more Custom Actions for the rest of the install Custom Actions when we can ensure it gets run in the right sequence with the rest of them. On a side note, the UPGRADEPATH property is actually set by a piece of WiX that looks something like this:

<Upgrade Id='XXXXXXXX-XXX-XXXX-XXXX-XXXXXXXXXXXX'><UpgradeVersion OnlyDetect='no' Property='UPGRADEPATH' Minimum="5.6.1" IncludeMinimum='yes' Maximum='9.5.0' IncludeMaximum='no' /></Upgrade>

This will tell the Custom Action CAUpgradeVMDatabase to run when we are installing on top of a minimum version of 5.6.1. With that being said, you should have all the information you now need to build sequenced managed Custom Actions using WiX. If there is anything I missed, feel free to leave comments or criticisms. Also keep in mind that there are a handful of individuals out there who will recommend against doing what I just showed you, but if you follow the few guidelines I set up, and don’t forget your rollback/uninstall/upgrade/repair actions, you should be able to safely run your managed Custom Actions.

Overriding Alert (Monitor) Messages with SCOM 2007 and the F5 Management Pack

Blog Author — Wed, 07 Oct 2009 16:34:50 GMT

In this blog post I’ll describe a way to override alert messages (for health state monitors and alerts) using Microsoft System Center Operations Manager (SCOM) 2007 R2 and the F5 Management Pack. The topic covered in this article is rooted in several requests from the field, where there’s been a need for creating custom alert messages for monitors and alerts related to F5 devices.

I will focus on describing a common use case: monitoring an LTM Pool Member using the LTM Pool Member Aggregate Monitor defined in the F5 Management Pack. The code samples, XML authoring and implementation, and ultimately the tests have all been done using SCOM 2007 R2 and F5 Management Pack v.1.4.1.251, but they should be applicable to earlier and possibly future versions as well.

From the beginning I need to highlight the fact that we’re discussing more than a trivial override for health state alerts and monitors here. As you would probably guess, the alerts and monitors defined in the F5 Management Pack are public, so they can be accessed from other (or override) management packs, but certain properties on these objects cannot be overwritten (through overrides) using the SCOM 2007 Management Console user interface, and also there are other limitations on performing overrides, as the F5 Management Pack is a sealed management pack.

The alert and monitor messages / descriptions fall into this category. They simply cannot be overwritten using the SCOM UI. One way to override the alert messages is through XML authoring and override packs. The following implementation will assume that you are familiar with XML and SCOM management pack authoring, and also would require a tiny bit of SQL Server query skills.

And I will start by scoping out the major steps involved in creating the alert message override, for an LTM Pool Member Aggregate Monitor:

Create an override for the LTM Pool Member Aggregate Monitor targeting an override management pack (F5.Override.ManagementPack).

Export the override management pack into an XML file: F5.Override.ManagementPack.xml.

Edit the F5.Override.ManagementPack.xml file for adding the required string resources for the override alert message.

Re-import the updated override management pack (from F5.Override.ManagementPack.xml) (Phase 1).

Query the OperationsManager SQL database for the ID (GUID) of the override alert message string resource (added in step 3).

Edit the F5.Override.ManagementPack.xml file again to add the alert message related monitor property overrides referencing the ID found above.

Re-import again the updated F5.Override.ManagementPack.xml file (Phase 2).

I will describe in detail each of these steps in the following paragraphs.

Step 1: Override the LTM Pool Member Aggregate Monitor

I will briefly go through the steps to create an override for the LTM Pool Member Aggregate Monitor, targeting a certain pool member (in our example 172.27.155.29:80), and you can see the final results in Figure 1:

open the Health Explorer for the given pool member (you can right click in the diagram view of the F5 device > Health Explorer).

in the Health Explorer browse to Entity Health > Availability > Monitor State and right click.

in the Monitor State context menu choose Monitor Properties… and go to the Overrides tab.

in the Overrides window click the Override… button and select the For the object: <ip_address:port> menu item.

select a new management pack and name it to something like F5.Override.ManagementPack for the destination management pack.

set up the overrides to something like the ones shown in Figure 1.

Figure 1: The LTM Pool Member Aggregate Monitor Overrides

Step 2: Export the Override Management Pack (F5.Override.ManagementPack.xml)

in the SCOM Management Console navigate to Administration and choose Management Packs.

in the right panel select the override management pack (F5.Override.ManagementPack), right click > Export Management Pack…

save the management pack to a folder where you can access it.

Note:

If you named the override management pack as suggested here (F5.Override.ManagementPack), the export to the XML file would slightly change the name of the file to F.Override.ManagementPack.xml. You can stay with this name or you can change it to F5.Override.ManagementPack.xml, but then make sure that in the file you change the override management pack <Name> and <ID> attributes as well, to match F5.Override.ManagementPack (see Figure 2).

<Identity>
<ID>F5.OverrideManagementPack</ID>
<Version>1.0.0.0</Version>
</Identity>
<Name>F5.Override.ManagementPack</Name>

Figure 2: Make sure the override management pack file name and ID are matching.

Step 3: Add the Alert Message String Resources to the Override Management Pack

open the F5.Override.ManagementPack.xml file for editing and locate the <Presentation> section. Add the following XML code inside <Presentation> section:

<Presentation>

<StringResources>
<StringResource ID="F5.LTM.PoolMember.Monitor.AggregateMonitor_Override_AlertMessageResourceID" />
</StringResources>

</Presentation>

Figure 3: Adding the string resource for the allert message, to the override management pack

continue by locating the <LanguagePacks> section and then add the following XML code inside the <LanguagePack> section:

<DisplayString ElementID="F5.LTM.PoolMember.Monitor.AggregateMonitor_Override_AlertMessageResourceID">
<Name>Override Alert Message</Name>
<Description>{0}: {1}.</Description>
</DisplayString>

Figure 4: Adding the display string for the alert message

save the updated F5.Override.ManagementPack.xml file.

As you can see, the display string consists of two placeholders in our example: {0} and {1}. That’s because we aim for the following alert message: <pool_member>: <status>. The two placeholders correspond to the pool member name and status parameters, respectively, which will be loaded dynamically at runtime.

Step 4: Re-import the Updated Override Management Pack (Phase 1)

in the SCOM Management Console navigate to Administration and choose Management Packs.

delete the “old” override management pack first: in the right panel select the override management pack (F5.Override.ManagementPack), right click > Delete…

clear the health service state cache related to the override management pack: open Windows Explorer > Program Files\System Center Operations Manager 2007\Health Service State\Management Packs and delete the override management pack XML file (named as something like F5.OverrideManagementPack.{E0E8B7BC-E4B3-4E37-B1D5-67873CC877E6}.{743C1DA1-5328-7BE1-3681-482DE7E7133B}.xml)

import the updated override management pack: SCOM Management Console > Administration > Management Packs > right click > Import Management Packs > select the updated override management pack XML file (F5.Override.ManagementPack.xml).

Step 5: Query the SCOM Database for the override alert message resource ID

open SQL Management Studio and connect to the SQL Server hosting the SCOM 2007 OperationsManager database.

select the OperationsManager database (to run the query against)

open a new query window and type in the following SQL query:

SELECT * FROM StringResource
WHERE StringResourceName = 'F5.LTM.PoolMember.Monitor.AggregateMonitor_Override_AlertMessageResourceID'

Figure 5: Querying the SCOM database for the override alert message resource ID

execute the query and copy the ID (GUID) value returned by the query. In our example this value is: 7E58A9E2-87C8-409A-DF1E-034BAF832B0C, but this is a globally unique identifier and it’s different in your environment. Make sure you get the value returned by the query (don’t copy / paste the one given in this example).

Step 6: Update the Override Management Pack with the Monitor Property Overrides

open the F5.Override.ManagementPack.xml file and locate the <Override> section.

copy and paste the XML code snippet shown in Figure 6, preferably at the beginning of the <Override> section: make sure to put the appropriate value for the GUID of the alert message resource ID for the MonitorPropertyOverrideAlertMessage property:

<MonitorPropertyOverride ID="MonitorPropertyOverrideAlertMessage" Context="F5_ManagementPack!F5.LTM.PoolMember" Enforced="false" Monitor="F5_ManagementPack!F5.LTM.PoolMember.Monitor.AggregateMonitor" Property="AlertMessage">
<Value>7E58A9E2-87C8-409A-DF1E-034BAF832B0C</Value>
</MonitorPropertyOverride>
<MonitorPropertyOverride ID="MonitorPropertyOverrideAlertParameter1" Context="F5_ManagementPack!F5.LTM.PoolMember" Enforced="false" Monitor="F5_ManagementPack!F5.LTM.PoolMember.Monitor.AggregateMonitor" Property="AlertParameter1">
<Value>$Data/Context/DataItem/LoggingComputer$</Value>
</MonitorPropertyOverride>
<MonitorPropertyOverride ID="MonitorPropertyOverrideAlertParameter2" Context="F5_ManagementPack!F5.LTM.PoolMember" Enforced="false" Monitor="F5_ManagementPack!F5.LTM.PoolMember.Monitor.AggregateMonitor" Property="AlertParameter2">
<Value>$Data/Context/DataItem/EventDescription$</Value>
</MonitorPropertyOverride>

Figure 6: Adding the monitor property overrides to the override management pack

the two alert parameters featured in the code snippet above (MonitorPropertyOverrideAlertParameter1 and MonitorPropertyOverrideAlertParameter1) are the ones filling up the placeholders in the alert message (as described at the end of Step 3.

keep in mind that within the <Value> tags we can only get $Data value types ($Target value types would not render correctly); you can choose to have multiple alert parameters, which you can find inside the context of the aggregate monitor (see the code snippet in Figure 7)

< DataItem type =" System.Event.LinkedData " time =" 2009-10-08T08:11:13.6860686Z " sourceHealthServiceId =" ff2ffa3e-2b0c-803f-af44-328e9b517423 " >
< EventOriginId > {6bbefb8c-0b90-49d2-8e0c-2d2430a247a7} </ EventOriginId >
< PublisherId > {26bf7cd6-0137-5f29-b3f6-2e4000000000} </ PublisherId >
< PublisherName > F5 Networks </ PublisherName >
< EventSourceName > F5 Networks </ EventSourceName >
< Channel > F5 Networks </ Channel >
< LoggingComputer > 172.27.155.129:80 </ LoggingComputer >
< EventNumber > 311 </ EventNumber >
< EventCategory > 1 </ EventCategory >
< EventLevel > 1 </ EventLevel >
< UserName > F5 Networks </ UserName >
< RawDescription > Monitor associated with configuration object is down </ RawDescription >
< LCID > 127 </ LCID >
< CollectDescription > True </ CollectDescription >
< Params />
< EventData >
< Data />
</ EventData >
< EventDisplayNumber > 311 </ EventDisplayNumber >
< EventDescription > Monitor associated with configuration object is down </ EventDescription >
< ManagedEntityId > {ce5bf975-3a68-7283-ff41-2a7f0e7e40b4} </ ManagedEntityId >
< RuleId > {91a47fc4-a084-20c9-2c91-367e7f4a3524} </ RuleId >
</ DataItem >

Figure 7: The context data for the LTM Pool Member Aggregate Monitor

save the updated F5.Override.ManagementPack.xml file.

Note that the context data may serve for choosing the appropriate name for other custom parameters to be featured inside the override alert message.

Step 7: Re-import the Updated Override Management Pack (Phase 2). Done!

in the SCOM Management Console navigate to Administration and choose Management Packs.

delete the “old” override management pack first: in the right panel select the override management pack (F5.Override.ManagementPack), right click > Delete…

clear the health service state cache related to the override management pack: open Windows Explorer > Program Files\System Center Operations Manager 2007\Health Service State\Management Packs and delete the override management pack XML file (named as something like F5.OverrideManagementPack.{E0E8B7BC-E4B3-4E37-B1D5-67873CC877E6}.{743C1DA1-5328-7BE1-3681-482DE7E7133B}.xml)

import the updated override management pack: SCOM Management Console > Administration > Management Packs > right click > Import Management Packs > select the updated override management pack XML file (F5.Override.ManagementPack.xml).

If you test the LTM Pool Member Aggregate Monitor at this time (stopping the IIS service on it for example), you should get an alert message similar to the one shown in Figure 8.

Figure 8: The custom alert message for the LTM Pool Member Aggregate Monitor override

Conclusion

In this article we went through the non-trivial task of creating an alert override featuring a custom alert message. The example presented here can be easily adapted / extended to other monitors and alerts inside SCOM 2007 and the F5 Management Pack, or other management packs. I hope you’ll find the article useful, and if you have any questions or suggestions please feel free to post them here.

You can download the override management pack shown as an example, here: F5.OverrideManagementPack.xml

Installing the F5 Management Pack Using the Command Line

Blog Author — Wed, 07 Oct 2009 05:40:37 GMT

To start with, the F5 Networks Management Pack has been able to run from a command prompt since it’s first release. All of the options available through the pretty user interface are actually all available using command line switches/options. There are also a few extra options not available through the standard GUI. I will go a little more in depth with those options, as they will be new to most of you.

First of all, if you open a standard command prompt in Windows, make sure you are running as a local administrator. In Windows 2008, the easiest way to do this is by right-clicking on the “Command Prompt” icon and finding the little blue and yellow shield that says “Run as administrator” (note: you may be prompted for credentials). From here you will want to find and run the F5Networks.ManagementPack.Setup.exe file. My recommendation is to just drag-n-drop the downloaded file directly into the Command Prompt window, where it should automatically drop in the correct path and filename for you. Now you are ready to add your command line options.

You can find a list of all of our supported command line options by visiting our Wiki documentation. The ‘new’ options you should be most interested in are the /logfile, /debuglogging, /passive and /quiet switches. The rest of the options in the table are available through the standard user interface for the installer. Note that if you run setup from the command prompt without the /passive or /quiet flags set, the GUI will still pop up, but now all of the fields will be populated for you to review as you step through the wizard. The Wiki documentation also has a few examples for you to follow for best results. I personally recommend using the /passive option, as it will still give you a visual cue that installation is happening and notify you when it has completed, whereas the /quiet will run asynchronously on the system with no UI or messages of any kind.

The /logfile and /debuglogging options are very useful if you are having problems with setup and wish to troubleshoot why our software package will not install properly. We also would need these logs if we were to help you troubleshoot, so it is a good habit to get into. If you have any problems trying to run setup from a command prompt, feel free to drop by the F5 Management Pack Forums and ask for assistance.

F5 Management Pack v1.4.1.93 Released: Globalization Support!

Blog Author — Wed, 30 Sep 2009 07:23:03 GMT

Greetings everyone. I would like to announce that the latest release of the F5 Networks Management Pack is now available on the downloads page. Direct links are available below:

Download F5 Networks Management Pack v1.4.1.93 x64 Version Now

Download F5 Networks Management Pack v1.4.1.93 x86 Version Now

With this latest release of the Management Pack, we have added globalization support to our software, which allows us to support localized copies of Windows Server 2003 & 2008 R2, SQL Server 2005 SP2 & 2008, and Systems Center Operations Manager 2007 SP1 & R2. While our software support installation into these localized environments, we still do not have a language pack or localized resources. This just means that all dialogs and information will still be in English.

If you have any questions or complaints, feel free to visit our support forum here.

Statistics Collection and Management Pack Scalability

Fri, 18 Sep 2009 08:12:35 GMT

We’ve had several folks ask about how the Management Pack scales and how to determine what kind of hardware is needed when monitoring large numbers of Big-IPs. This article will outline how to tune statistics collection in the Management Pack to best suite your needs and available hardware.

Keep in mind that Operations Manager and SQL Server have countless ways to optimize performance and that those options should be reviewed first as they potentially will make much more of a difference that just tuning the F5 Management Pack. One example is that your Data Warehouse should be running on a separate server that is optimized for SQL Server (such as having data and logs on separate disks).

Once your general infrastructure is optimized, it’s time to consider which performance metrics you’d like to gather. By default only the global (device level) stats are collected but you can turn on collection for CPUs, Nodes, Pool Members, etc. Please see the Performance Collecting and Monitoring topic for exactly how to turn on statistics collections.

In our testing we’ve found that a single Operations Manager management server running on a Dell 2950 can handle up to 250 statistics/sec from monitoring Big-IPs. I would recommend trying to shoot for a lower number than that so that you have some CPU cycles left for monitoring other types of objects in your network and handling non-OpsMgr tasks.

So let’s assume that we don’t want to go above 150 stats/sec for each management server. You can use the following table to calculate how many stats/sec you will generate by turning on the various categories of statistics:

Type of Statistic # of Stats

Global (Device) 49

CPU 3

Chassis 1

Partition 5

Trunk 9

Interface 9

LtmNode 7

LtmVirtualServer 7

LtmPool 7

LtmPoolMember 7

GtmDataCenter 7

GtmServer 7

GtmVirtualServer 7

GtmWideIP 4

GtmPool 3

GtmPoolVirtualServer 3

As I mentioned only the Global stats are turned on by default and are collected every 2 minutes, so out-of-the-box stats/sec when monitoring 2 Big-IPs can be calculated as such:

((2 Big-IP * 49 stats) / 120 seconds) = .8 stats/sec

Now let’s suppose you want to turn on stats for Pool Members, and each Big-IP has 20 pool members. The additional stat load would be the following:

(2 Big-IP * 20 Pool Members Each * 7 stats per pool member) / 120 seconds = 23 stats/sec

We now would be collecting a grand total of 23.8 stats/sec – still lots of room to grow! Armed with this methodology for calculating load, you can now balance your need for each kind of statistic against the load it will put on the management server.

One way to determine for sure how many stats/sec are currently being generated is to view the various F5 counters that are installed in the Windows Performance Monitor.

For more detailed information on which stats are collected, please see the health model wiki page.

Create F5 Management Pack Rule Overrides with PowerShell

Blog Author — Mon, 14 Sep 2009 09:15:01 GMT

With this article we'll initiate a series of PowerShell automation topics focused on the F5 Management Pack and Microsoft System Center Operations Manager 2007.

1. Overview

In this blog I will write about creating rule overrides in System Center Operations Manager (SCOM) 2007, using Windows Powershell. Although the code samples specifically target creating rule overrides defined within the F5 Management Pack, you can easily adapt the implementation with minimal changes, for targeting rule overrides within any SCOM 2007 management pack. The code was implemented using PowerShell 1.0 and the samples have been tested with SCOM 2007 SP1 and R2.

Before we go into the details, make sure you get familiar with the way rule overrides are created within the F5 Management Pack, following the F5 DevCentral article about Performance Collection and Monitoring. Attached to this blog you’ll find a small PowerShell utility library which you’ll need to download in order to test with the code samples described in this article.

The code samples presented in this article assume that you have installed Microsoft System Center Operations Manager 2007 (SP1 or R2) and the F5 Management Pack on your system.

2. Implementation

Bellow is the source code of the PowerShell utility library implementing the functionality for creating rule overrides, described in this post. You can copy / paste this code "as-is" into a PowerShell library file. This file would be referenced in any script using the functions defined in this library. Let's name this file F5MpUtils.ps1for our reference here, in this article.

# ***********************************************************************************
# * F5 Management Pack PowerShell Utility Library 1.0
# *
# * System requirements:
# * SCOM 2007 SP1 or later
# * F5 Management Pack
# ***********************************************************************************
Add-PSSnapin "Microsoft.EnterpriseManagement.OperationsManager.Client" -ErrorAction SilentlyContinue
Add-PSSnapin "F5.Networks.ManagementPack" -ErrorAction SilentlyContinue
Add-PSSnapin "F5.Networks.iControl" -ErrorAction SilentlyContinue

#region Private Constants

# the ID of the F5 Management Pack in SCOM
$F5ManagementPackName = "F5_ManagementPack";

# F5 override management pack
$F5OverrideMpName = "F5_Override_ManagementPack";
$F5OverrideMpDisplayName = "F5 Override Management Pack";
$F5OverrideMpDescription = "Hosts various overrides for the F5 Management Pack"

# F5 Management Pack classes
$F5MonitoringServiceClass = "F5.MonitoringService";
$F5DeviceClass = "F5.Device";

# default language code used for the override management pack
$DefaultLanguageCode = "ENU";

#endregion

#region Private Fields

# the F5 Management Pack object
[Microsoft.EnterpriseManagement.Configuration.ManagementPack] $private:f5Mp;

# the SCOM Management Group
[Microsoft.EnterpriseManagement.ManagementGroup] $private:mgmtGroup;

# SCOM installation folder
[string] $private:scomInstallDir;

# SCOM SDK host
[string] $private:scomSdkHost;

#endregion

#region Private Helper Functions

# ***********************************************************************************
# Initializes various fields needed by this utility library.
# ***********************************************************************************
function Initialize()
{
# get the SCOM installation folder from the registry
$script:scomInstallDir = $(Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Setup").InstallDirectory;

# get the SCOM SDK host from the registry
$script:scomSdkHost = $(Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Microsoft Operations Manager\3.0\Machine Settings").DefaultSDKServiceMachine;

# get the Management Group Connection (if any) to the SDK service
$mgc = Get-ManagementGroupConnection | Where-Object {$_.ManagementServerName -eq $scomSdkHost};

# do we have a valide Management Group Connection?
if ($mgc -eq $null)
{
# create a new Management Group Connection
New-ManagementGroupConnection $scomSdkHost -ErrorAction SilentlyContinue -ErrorVariable Err;

# check for errors
if ($Err) { $(throw Write-Host $Err); }
}

# get the Management Group from the Management Group connection
$mg = $mgc.ManagementGroup;

# get the F5 Management Pack
$script:f5Mp = GetManagementPack($F5ManagementPackName);
}

# ***********************************************************************************
# Creates a SCOM Management Pack by name.
# $name = the name of the Management Pack.
# $displayName = the friendly name of the Management Pack.
# ***********************************************************************************
function CreateManagementPack([string] $name, [string] $displayName)
{
# attempt to get the Management Pack
$mp = GetManagementPack($name);

# check if the management pack exists
if ($mp -eq $null)
{
$mpStore = New-Object Microsoft.EnterpriseManagement.Configuration.IO.ManagementPackFileStore;
$mpStore.AddDirectory($scomInstallDir);

$mp = New-Object Microsoft.EnterpriseManagement.Configuration.ManagementPack($name, $displayName, (New-Object Version(1, 0, 0)), $mpStore);
$mgmtGroup.ImportManagementPack($mp);

$mpXmlWriter = New-Object Microsoft.EnterpriseManagement.Configuration.IO.ManagementPackXmlWriter($scomInstallDir);

# write the management pack xml file and make sure to clear any output before returning
$mpXmlWriter.WriteManagementPack($mp) | Out-Null;
}

$mp;
}

# ***********************************************************************************
# Gets a SCOM Management Pack by name.
# $name = the name of the Management Pack.
# ***********************************************************************************
function GetManagementPack([string] $name, [bool] $force = $false)
{
# make sure the Management Group is still connected
if (!$mgmtGroup.IsConnected)
{
$mgmtGroup.Reconnect();
}

$criteria = [string]::Format("Name = '{0}'", $name);
$mpCriteria = New-Object Microsoft.EnterpriseManagement.Configuration.ManagementPackCriteria($criteria);
$mgc = Get-ManagementGroupConnection;
$script:mgmtGroup = (Get-ManagementGroupConnection).ManagementGroup;

$mp = $mgmtGroup.GetManagementPacks($mpCriteria)[0];

return $mp;
}

# ***********************************************************************************
# * Creates a rule override.
# * $name = the name of the rule to override
# * $sourceClass = the monitoring class of to the rule to be overriden
# * $targetClass = the monitoring class / context of the rule override
# * $parameters = the parameters to override (hashtable)
# * $enforce = flags the forced state of the rule override
# * $languageCode = the language code used for the override management pack
# ***********************************************************************************
function CreateRuleOverride(
[string] $name,
[string] $sourceClass = $F5MonitoringServiceClass,
[string] $targetClass,
[hashtable] $parameters,
[bool] $enforce = $false,
[string] $languageCode = $DefaultLanguageCode)
{
# validate parameters
if (($parameters -eq $null) -or ($parameters.Count -eq 0))
{
throw "Enter a hashtable with the override rule parameters.";
}

# create the override management pack (if necessary)
$f5OverrideMp = CreateManagementPack -name $F5OverrideMpName -displayName $F5OverrideMpDisplayName;

# create the criterias to search for
$sourceClassCriteria = New-Object Microsoft.EnterpriseManagement.Configuration.MonitoringClassCriteria($("Name='{0}'" -f $sourceClass));
$targetClassCriteria = New-Object Microsoft.EnterpriseManagement.Configuration.MonitoringClassCriteria($("Name='{0}'" -f $targetClass));

# get the monitoring classes featured by the rule override
$sourceMonitoringClass = $mgmtGroup.GetMonitoringClasses($sourceClassCriteria)[0];
$targetMonitoringClass = $mgmtGroup.GetMonitoringClasses($targetClassCriteria)[0];

# create the rule criteria based on display name
$ruleCriteria = New-Object Microsoft.EnterpriseManagement.Configuration.MonitoringRuleCriteria($("DisplayName='{0}'" -f $name));

# get the rule to override
$rule = $sourceMonitoringClass.GetMonitoringRules($ruleCriteria)[0];

# cast some of the generic types needed by the rule override properties using reflection (::op_Implicit())
# (this is apparently the only way to perform valid casts on types implementing the implicit operator)
$rule = [Microsoft.EnterpriseManagement.Configuration.ManagementPackElementReference``1[Microsoft.EnterpriseManagement.Configuration.ManagementPackRule]]::op_Implicit($rule);
$sourceMonitoringClass = [Microsoft.EnterpriseManagement.Configuration.ManagementPackElementReference``1[Microsoft.EnterpriseManagement.Configuration.ManagementPackClass]]::op_Implicit($sourceMonitoringClass);
$targetMonitoringClass = [Microsoft.EnterpriseManagement.Configuration.ManagementPackElementReference``1[Microsoft.EnterpriseManagement.Configuration.ManagementPackClass]]::op_Implicit($targetMonitoringClass);

# look for the "Enabled" override parameter (if any)
$enabled = $parameters["Enabled"];

# for the "Enabled" override parameter we create a rule property override object
if (($enabled -ne $null) -and ($enabled.ToString().ToLower() -eq "true"))
{
# create the rule property override object
$ruleOverrideDisplayName = $("{0} Property Override" -f $name);
$ruleOverrideName = $ruleOverrideDisplayName.Replace(" ", [String]::Empty);
$ruleOverride = New-Object Microsoft.EnterpriseManagement.Configuration.ManagementPackRulePropertyOverride($f5OverrideMp, $ruleOverrideName);

# set the rule override parameters
$ruleOverride.Rule = $rule;
$ruleOverride.Property = [Microsoft.EnterpriseManagement.Configuration.ManagementPackWorkflowProperty]::Enabled;
$ruleOverride.Value = $enable.ToString().ToLower();
$ruleOverride.Context = $sourceMonitoringClass;
$ruleOverride.LanguageCode = $languageCode;
$ruleOverride.Enforced = $enforce.ToString().ToLower();
$ruleOverride.DisplayName = $ruleOverrideDisplayName;

# remove the "Enabled" parameter from our hashtable for further processing
$parameters.Remove("Enabled");
}

# for the rest of the rule override parameters create a rule configuration override object
if ($parameters.Count -gt 0)
{
# create the rule configuration override object
$ruleConfigOverrideDisplayName = $("{0} Configuration Override" -f $name);
$ruleConfigOverrideName = $ruleConfigOverrideDisplayName.Replace(" ", [String]::Empty);
$ruleConfigOverride = New-Object Microsoft.EnterpriseManagement.Configuration.ManagementPackRuleConfigurationOverride($f5OverrideMp, $ruleConfigOverrideName);

# set the rule configuration override parameters
$ruleConfigOverride.Rule = $rule;

# iterate through the parameter hashtable to set override parameters / values
foreach ($key in $parameters.Keys)
{
$ruleConfigOverride.Parameter = $key;
$ruleConfigOverride.Value = $parameters[$key];
}

$ruleConfigOverride.Module = "DataSource";
$ruleConfigOverride.Context = $targetMonitoringClass;
$ruleConfigOverride.LanguageCode = $languageCode;
$ruleConfigOverride.Enforced = $enforce.ToString().ToLower();
$ruleConfigOverride.DisplayName = $ruleConfigOverrideDisplayName;
}

# verify the changes in the override management pack
$f5OverrideMp.Verify();

# save the changes into the override management pack
$f5OverrideMp.AcceptChanges();
}

#endregion

#region Script Run

# initialize library
Initialize

#endregion

The function responsible for implementing the rule override functionality has the prototype shown in the code snippet bellow:

function CreateRuleOverride(
[string] $name,
[string] $sourceClass = $F5MonitoringServiceClass,
[string] $targetClass,
[hashtable] $parameters,
[bool] $enforce = $false,
[string] $languageCode = $DefaultLanguageCode)

As you can see, the rule override gets created based on:

the name of the rule to be overriden ($name);

the monitoring class associated with the rule to be overriden ($sourceClass);

the target monitoring class / context of the rule override ($targetClass);

the parameters of the rule to be overriden, as a hashtable ($parameters).

All of the F5 Management Pack rules are associated with the F5.MonitoringService monitoring class. Rule overrides for the F5 Management Pack would usually target the F5.Device monitoring class (as most of the objects defined within the F5 Management Pack are derived from F5.Device).

The implementation of the CreateRuleOverride() function is relatively simple, and goes by creating the override management pack (for hosting the rule overrides), if it's not already created. We named this override management pack F5_Override_ManagementPack, but you can name it any way you like (by changing the $F5OverrideMpName and $F5OverrideMpDisplayName constants in the library script).

# create the override management pack (if necessary)
$f5OverrideMp = CreateManagementPack -name $F5OverrideMpName -displayName $F5OverrideMpDisplayName;

The next step is to get the rule we need to override. This is done by creating a criteria based on the name of the rule ($ruleCriteria) and filter the monitoring classes on the local management group based on this criteria.

# get the rule to override
$rule = $sourceMonitoringClass.GetMonitoringRules($ruleCriteria)[0];

The only challenging part in implementing the PowerShell script for creating the rule override was to correctly cast the rule and rule override context objects, which are defined as generic types inside SCOM derived from the Management PackElementReference<T> object, which also has an implicit operator. The code snippet bellow illustrates the class definition for the ManagementPackElementReference<T> object in .NET (as C# code):

[Serializable]
public class ManagementPackElementReference<T> where T : Microsoft.EnterpriseManagement.Configuration.ManagementPackElement
{
public static implicit operator ManagementPackElementReference<T>(T t);

public Guid Id { get; }
public string Name { get; }

public override bool Equals(object obj);
public T GetElement();
public override int GetHashCode();
public override string ToString();
}

The difficulty using this type of object in PowerShell comes when tyring to perform either an implicit or explicit cast on the object, trying to match the exact type we're using it for, in particular when assigning the rule and context objects (follow the code snippets bellow). Because of the implicit operator, PowerShell is not able to find the appropriate cast, and throws an "invalid cast" exception.

The code snippet bellow uses .NET reflection to call the op_implicit() method (the equivalent of the implicit operator declaration) on the generic type we need to cast:

# cast some of the generic types needed by the rule override properties using reflection (::op_Implicit())
# (this is apparently the only way to perform valid casts on types implementing the implicit operator)
$rule = [Microsoft.EnterpriseManagement.Configuration.ManagementPackElementReference``1[Microsoft.EnterpriseManagement.Configuration.ManagementPackRule]]::op_Implicit($rule);
$sourceMonitoringClass = [Microsoft.EnterpriseManagement.Configuration.ManagementPackElementReference``1[Microsoft.EnterpriseManagement.Configuration.ManagementPackClass]]::op_Implicit($sourceMonitoringClass);
$targetMonitoringClass = [Microsoft.EnterpriseManagement.Configuration.ManagementPackElementReference``1[Microsoft.EnterpriseManagement.Configuration.ManagementPackClass]]::op_Implicit($targetMonitoringClass);

This way assigning the override rule properties to the appropriate type would succeed:

# set the rule override parameters
$ruleOverride.Rule = $rule;
$ruleOverride.Property = [Microsoft.EnterpriseManagement.Configuration.ManagementPackWorkflowProperty]::Enabled;
$ruleOverride.Value = $enable.ToString().ToLower();
$ruleOverride.Context = $sourceMonitoringClass;
$ruleOverride.LanguageCode = $languageCode;
$ruleOverride.Enforced = $enforce.ToString().ToLower();
$ruleOverride.DisplayName = $ruleOverrideDisplayName;

Finally, we verify and seal the changes made in the override management pack:

# verify the changes in the override management pack
$f5OverrideMp.Verify();

# save the changes into the override management pack
$f5OverrideMp.AcceptChanges();

Keep in mind that the implementation of creating the rule override, closely follows the article about the rule overrides mentioned in Performance Collection and Monitoring.
3. Usage
In the following example we create a rule override for the "LTM Virtual Server Client Current Connections" rule in the F5 Management Pack. This rule is not enabled by default and it represents a statistics collection rule. The rule override will enable this rule and will set the collection interval to 120 seconds.

# add a reference to the utility library
. C:\Samples\F5MpUtils.ps1

CreateRuleOverride -name "LTM Virtual Server Client Current Connections" -targetClass "F5.Device" -parameters @{"Enabled" = "true"; "Interval" = "120"};

4. Conclusion

The F5 Management Pack supports full automation using PowerShell for managing F5 devices using Microsoft System Center Operations Manager (SCOM) 2007. Most of the functionality exposed through the SCOM SDK can be mirrored through PowerShell when implementing various administrative and maintenance tasks. Some challenge in implementing / translating the various APIs in PowerShell may rise when dealing with special types (such as generics, or types defined by implicit operators), but .NET reflection would come to the rescue.

We'll be continuing our series with posting articles on F5 Management Pack and SCOM 2007 integration with PowerShell and any feedback related to this topic is greatly appreciated.

A special thanks goes to Marc Cauchy who assisted with some fine details in authoring the PowerShell scripts presented in this post.

Datacenter Orchestration Scenarios Now Available

Blog Author — Fri, 28 Aug 2009 07:33:01 GMT

For the past couple of months, fellow developer Joel Hendrickson has been working on some fairly impressive data center orchestration demonstration scenarios. These automated demos really show the true potential of the F5 Networks Management Pack when combined with Systems Center Operations Manager (SCOM) and Virtual Machine Manager (VMM).

Joel has combined the technologies above into a series of scripts and recovery actions that automate the process of spinning up and down resources such as web servers, disabling and enabling pool members when put into maintenance mode inside of Ops Manager, and failing over the active member of a redundant F5 pair based on health or statistics, as opposed to network or hardware failure.

Here are the scenarios:

Add / Remove Resource:

Documentation | Add Video | Remove Video

How-To | How-To Video

Maintenance Mode:

Documentation | Video

How-To

Fail-over:

Documentation | Video

How To

Documentation on the scenarios themselves, along with videos and ‘How-To’ tutorials are now available and will continue to be expanded upon in the documentation section of the F5 Networks Management Pack area.

Announcing F5 Management Pack 1.4 for Microsoft System Center Operations Manager 2007

Tue, 25 Aug 2009 04:44:35 GMT

The Management Pack product team is proud to announce the immediate release of version 1.4, which includes cumulative support for new platforms, features related to data center orchestration, and a numerous stability and user experience improvements.

This new version includes support for the following platforms:

All versions of F5 BIG-IP LTM and F5 BIG-IP GTM starting with 9.2, including the latest hardware release Viprion and 10.0.1

Microsoft Windows Server 2003 x86 and x64, Windows Server 2008 x64

Microsoft SQL Server 2003 and 2008

Microsoft System Center Operations Manager 2007 and R2

We have also worked to include key technology pieces to enable custom data center orchestration of network applications utilizing both F5 BIG-IP.

Distributed Data Centers:

Support for Distributed Management Servers (SCOM gateway servers), allowing for monitoring and managing up to 20 F5 devices, in a distributed global environment

Authorization

Support for Authorization roles and groups via Microsoft SCOM, Active Directory, and F5 BIG-IP’s user authentication model

The Management Pack includes built-in roles for discovery and device configuration

Microsoft Windows and Active Directory accounts can be correlated with BIG-IP user accounts via a one time device authentication Management Pack action

Device Configuration

Support for key network and data center administrator device configuration tasks, including device failover, virtual server and pool member configuration

Fully PowerShell scriptable device configuration API, integrated directly into the SCOM configuration hierarchy

We know your network applications are high-availability, business-critical, and distributed across your company. With these new features, we hope to dramatically increase the efficiency for your organization to quickly and easily adapt to changing network conditions.

Finally, we have heard you loud and clear about the pain points around upgrading between releases of the Management Pack, and starting with 1.4, the setup utility includes built-in upgrade. We will support upgrading from 1.3 to 1.4 with this release. Also, all future releases of the Management Pack are being designed to ensure the best upgrade experience. That means persistence of your BIG-IP device database, Management Pack overrides (for perf counter and threshold rules), and all compatible BIG-IP configuration data.

You can find the software download on F5’s DevCentral Management Pack website:

x86 Download for Windows Server 2003 and Windows Server 2008

x64 Download for Windows Server 2003 and Windows Server 2008

F5 Networks is currently provided product support through the DevCentral Forums:

Support Issues and Filing Bug Reports

The documentation can be found at: http://devcentral.f5.com/wiki/Default.aspx/MgmtPack.HomePage

We appreciate your patience over the last few months as we’ve integrated the breadth of features. We look forward to hearing your feedback, including critical feedback and concerns. We have lots of exciting new work to bring you to ease the management of your network applications, and ensure you get the best value for your current and future network investments.

Thanks,
Stephen
F5 Management Pack - Product Development Manager

F5 Management Pack v1.3.0.715 Featuring Integrated PowerShell Support is Now Available for Download

Blog Author — Wed, 19 Aug 2009 04:36:20 GMT

The Management Pack team is very excited to present this release which features an expanded set of PowerShell APIs and server tasks. The PowerShell API has been integrated with the Operations Manager PowerShell API as well as with the BIG-IP security model. These new features enable some very compelling scenarios, especially in the arena of Data Center Orchestration. You’ll hear more from us on that topic soon, but for now here’s an overview of the new features.

For more in-depth information on these topics including tutorial videos, visit the ManagementPack Home page and the Management Pack Video site.

BIG-IP Security Model Integration

We’ve added the ability to associate Windows NT accounts with BIG-IP accounts. Once you set up this mapping, any server task or PowerShell commandlet that needs to talk to a BIG-IP will log into the BIG-IP under the account that maps to whichever NT account is performing the operation.

As an example, let’s say I’m currently logged in as DOMAIN\joelh. I can map my account to the “admin” account using a new server task or using the following PowerShell command. Note that a secure dialog box will prompt me for the BIG-IP password:

Set-F5.DeviceConfigurationToken –DeviceIp 172.27.25.235 –UserName pooladmin

Now, when I run the “Disable Pool Member” server task, the task will only succeed if the “pooladmin” user has permissions on the BIG-IP to disable this particular pool member:

The security model also applies when using any of the F5 MP commandlets:

$token = Get-F5.DeviceConfigurationToken 172.27.25.235 // retrieves token that is mapped to current NT account
Disable-PoolMember –Token $token –Member 10.10.10.10:80

Operations Manager PowerShell Integration

This piece of integration essentially means that you can pass Operations Manager PowerShell objects into any of the the F5 Management Pack commandlets. This allows you to leverage the full power of the Operations Manager command shell when administering BIG-IPs. Here is a quick example that disables any pool members that have been put into Maintenance Mode:

$pmsNeedingDisable = @(Get-MonitoringObject -MonitoringClass $f5LTMPoolMemberClass -Criteria "EnabledState =`'Enabled`' and (InMaintenanceMode = 1)")
$pmsNeedingDisable | Disable-F5.LTMPoolMember -Token $token

An even quicker way to take advantage of this capability is by opening the Operations Manager Shell from within the context of a discovered object. An example of the power of this capability is shown below. I right-click an LTM pool in the Operations Manager diagram view and open the Operations Manager Shell from the context menu. I can then type a simple command to disable all pool members in the pool:

.

Other Additions

There are many more additions in this release including

New server tasks such as Create Pool Member and Force Pool Member Offline

10 new commandlets

Lots of bug fixes and performance improvements

Visit the Management Pack Home page for the download.

F5 Management Pack v1.0.0.527 with Support for Viprion is Now Available for Download

Blog Author — Tue, 30 Jun 2009 07:46:17 GMT

F5 Management Pack v1.0.0.527 adds support for discovering and managing Viprion devices. This includes discovery of CPUs on each blade with an added Slot ID property in Operations Manager to differentiate them. Additional monitor states are also available in the Viprion health model.

There are many other fixes and additions in this release including:

Improved support for Big-IPs running version TMOS version 10.x

Improved reliability when running on Windows Server 2008

Support for management of up to 20 devices across multiple management servers

An updated big3d agent (10.0.1.600) that includes scalability improvements

Relationship types defined in the F5 Management Pack are now public, enabling creation dependency monitors that reference F5 monitors

Rules that generate alerts now have Alert Suppression enabled

You can find the free downloads as well as documentation, videos, and the support forums on the F5 Management Pack home page.

Real IT Episode 10, And we’re live

Thu, 07 May 2009 08:44:52 GMT

In the final episode of Real IT the team re-caps what has been done so far, gets a surprise addition to their launch plans, and finally sees the project through the live launch in China.

Technorati Tags: F5,DevCentral,Real IT,ECC10

Episode 9, Here we go...

Wed, 29 Apr 2009 04:31:44 GMT

It's finally time! This week, we push the button and go live with our symmetric Web Accelerator deployment. Our users get to see the benefits first-hand, we get some real world results, and we get to hammer out any last minute problems that arise with just about any deployment.

Technorati Tags: F5,DevCentral,RealIT,Real IT,ECC9

Announcing the Release of the F5 Management Pack for Microsoft System Center Operations Manager 2007

Blog Author — Tue, 28 Apr 2009 01:06:35 GMT

My name is Stephen Fisher, and as Product Development Manager, I’m proud to introduce the F5 Management Pack for Big-IP LTM and GTM.

The F5 Management Pack is a software plug-in module for System Center Operations Manager that gives you a comprehensive range of F5 device health data as well as virtual server, pool, and pool member data. You can use this critical information for trending and analysis of your application delivery architecture in order to optimize resource utilization and effectively manage your infrastructure growth.

You can find the software download ~ a free download! on F5’s DevCentral website:

x86 Download for Windows Server 2003 and Windows Server 2008

x64 Download for Windows Server 2003 and Windows Server 2008

F5 Networks is currently provided product support through the DevCentral Forums:

Support Issues and Filing Bug Reports

The documentation can be found at: http://devcentral.f5.com/wiki/Default.aspx/MgmtPack.HomePage

Video tutorials are also available at: http://devcentral.f5.com/Default.aspx?tabid=2187

Thanks,

Stephen Fisher

Real IT Episode 8, We're almost live

Wed, 22 Apr 2009 05:46:49 GMT

This week we talked with IT a couple of times. First we met to give them the go-ahead to fire up the symmetric Web Accelerator solution behind the scenes so we could start testing in the real world. Next, we met to discuss what steps were left before launch. We're almost ready to go live!

Technorati Tags: F5,DevCentral,Real IT,DCC E8

Real IT Video Blog - Don thinks it all rocks

Fri, 17 Apr 2009 08:01:32 GMT

Don takes a shot at expressing the wonder that is today's technology. With a hat tip to Louis CK.

Technorati Tags: F5,DevCentral,RealIT,Real IT,Louis CK,Don Mac Vittie,ECC 7

Real IT Interview - Jason Talks Network Management

Fri, 17 Apr 2009 02:18:32 GMT

Jason discusses his background and experiences with Network Management projects and associated tools.

Technorati Tags: F5, DevCentral, RealIT, Real IT, DCC E7, Network Management, Jason Rahm

Real IT Episode 7, All systems...go?

Wed, 15 Apr 2009 07:02:59 GMT

In this episode of Real IT, Jason delves into the management side of things with IT, the content is reviewed by the China folks, and the DevCentral team makes the call on whether or not to roll the acceleration solution out into the wild for some real world testing.

Technorati Tags: F5,DevCentral,Real IT,DCC E7

Real IT Video Blog - Joe Goes Over Performance Results.

Tue, 07 Apr 2009 23:41:50 GMT

In this video blog, Joe comments on the tech center results and how small optimizations can yeild big results.

Technorati Tags: F5, DevCentral, RealIT, Real IT, ViewState, ASP.Net, Acceleration, Joe Pruitt

Real IT Video Blog - Jason Tackles Multitasking

Sun, 05 Apr 2009 20:02:45 GMT

In this video blog, Jason discusses how he load balances project tasks. His algorithm isn't nearly as clean or effective as Big-IP's are, however.

Technorati Tags: F5, DevCentral, RealIT, Real IT, Jason Rahm

Real IT Episode 6, Results and Planning

Wed, 01 Apr 2009 06:50:26 GMT

The next episode of the series has the team talking with Dan in the Tech Center about the results of their testing. Armed with these results they then go to form a plan of what's to come next with IT.

Technorati Tags: F5,DevCentral,Real IT,DCC E6

Real IT Interview - Nojan talks CDNs with Jeff

Wed, 01 Apr 2009 01:00:58 GMT

Jeff got a few minutes to sit down with Nojan, one of our engineers here at F5, and talk about Content Delivery Networks. They recapped what they are and how they work, as well as delving into some of the pros and cons and some other alternatives. Nojan's experience dealing with CDNs in the past is interesting and quite valuable to the team.

Technorati Tags: F5,DevCentral,Real IT,DCC E6,Interview,Jeff Browning

Real IT Video Blog - Colin awaits test results

Tue, 31 Mar 2009 22:15:47 GMT

Colin takes a minute to talk some more about testing and his anticipation for the results that Dan's team has gotten from the initial testing in the Tech center. He also discusses his experiences with application tuning and his expectations vs. his hopes for what kind of results they'll see from Web Accelerator in front of DevCentral.

Technorati Tags: F5,DevCentral,Real IT,DCC E6,Colin Walker

Real IT Video Blog - Jason Shifts Focus

Sun, 29 Mar 2009 22:09:31 GMT

In this video blog, Jason shifts focus away from acceleration to the point of the project: creating a Chinese version of DevCentral.

Technorati Tags: F5, DevCentral, RealIT, Real IT, Jason Rahm

Real IT Video Blog - Don ponders Betas

Thu, 26 Mar 2009 22:40:20 GMT

Pondering the team's decision not to use a beta product, Don talks about the criteria and implications of using any vendors' beta code in your project - and why we chose not to.

Technorati Tags: Real IT,Episode 5,Beta,Don MacVittie

Real IT Interview - Joe Hicks Talks About Best Practices With Enterprise CDNs

Thu, 26 Mar 2009 03:03:21 GMT

In this interview, F5 Senior Product Manager for Acceleration Joe Hicks talks about best practices when designing Enterprise Content Delivery Networks (ECDNs). He covers problems with web application delivery for foreign countries.

Technorati Tags: F5, DevCentral, RealIT, Real IT, Interview, Acceleration, ECDN, CDN, Joe Hicks

Real IT Video Blog - Joe Discusses A Workaround For Testing ASP Logins

Thu, 26 Mar 2009 02:57:21 GMT

After meeting with Dan Wright and learning how the stress testing application we were using in the Tech Center couldn't handle a special character in the login sequence of the DevCentral application, a unique solution is discovered that didn't require changes to the client application or the server code.

Technorati Tags: F5, DevCentral, RealIT, Real IT, DCC E5, Videoblogs, Joe Pruitt

Real IT Video Blog - Jason Confronts Roadblocks

Wed, 25 Mar 2009 20:09:35 GMT

In this video blog, Jason takes a minute to discuss inevitable roadblocks encountered during a project and the necessity of effective communication in overcoming them.

Technorati Tags: F5, DevCentral, RealIT, Real IT, Jason Rahm

Real IT Video Blog - Colin talks testing

Tue, 24 Mar 2009 23:58:05 GMT

Colin takes a second to comment on where the team is at with testing. He mentions the importance of this step of the process as well as how it feels to be on the side that's pushing for the app to get out sooner, rather than the side telling those people to hold on so they can make the code "perfect".

Technorati Tags: DevCentral RealIT,Testing,Tech Center,Colin Walker

Real IT Episode 5, Bugs, Build-out, and Bungled Shipping

Tue, 24 Mar 2009 23:35:11 GMT

In this episode, the team finds testing errors, starts building the site, and discovers that the hardware for the China side was shipped... Twice! Special guest appearances by Dan Wright and the IT gurus.

Technorati Tags: F5, DevCentral, RealIT, Real IT, DCC E5

Real IT Video Blog - Jason Talks Data Access And Ownership

Thu, 19 Mar 2009 21:59:19 GMT

In this video blog, Jason goes over his concerns about access and ownership to data when dealing with multiple departments within an organization.

Technorati Tags: F5, DevCentral, RealIT, Real IT, Jason Rahm

Real IT Video Blog - Jason Ponders Technology Reuse

Wed, 18 Mar 2009 23:49:11 GMT

The DevCentral China project is driving the investment in acceleration technologies but sharing technology across business units benefits the organization as a whole.

Technorati Tags: F5, DevCentral, RealIT, Real IT, DCC E4, Jason Rahm

Real IT Episode 4, Colos And Testing

Tue, 17 Mar 2009 22:58:21 GMT

In this episode, the team explores co-location sites in China and reviews results, while talking with IT about other possible solutions to the problem of hardware placement in China. Guest appearances by Marty Fagan, Paul Petersen, Dan Wright, and the DC China Team.

Technorati Tags: F5, DevCentral, RealIT, Real IT, DCC E4, Co-location, Testing

Real IT Video Blog - Joe Talks Code Deployment

Mon, 16 Mar 2009 23:15:47 GMT

In this video blog, Joe talks about the reality of running and deploying code into multiple locations.

Technorati Tags: F5, DevCentral, RealIT, Real IT, DCC E4, Joe Pruitt

Real IT Video Blog - Jeff Discusses Localized Sites

Sun, 15 Mar 2009 22:49:47 GMT

In this video blog, Jeff reflects on the role of localized sites with a focus on our rollout of DevCentral Japan.

Technorati Tags: F5, DevCentral, RealIT, Real IT, DCC E4, Jeff Browning

Real IT Video Blog - Jason Reflects On GTM Topology

Thu, 12 Mar 2009 23:47:24 GMT

Jason reflects on making use of GTM and geolocation and it's influence on topology and the tuning needed to make it all work.

Technorati Tags: F5, DevCentral, RealIT, Real IT, Jason Rahm

Real IT Video Blog - Don Considers Big Projects

Thu, 12 Mar 2009 23:28:24 GMT

Don comments on the ups and downs of working on big projects with geographically dispersed teams.

Technorati Tags: F5, DevCentral, RealIT, Real IT, DCC E3, Don Mac Vittie

Real IT Video Blog - Colin Talks About Using The Tech Center

Thu, 12 Mar 2009 23:22:54 GMT

After getting an OK from Dan, Colin considers the steps needed to get all the pieces in place to start testing in the Tech Center.

Technorati Tags: F5, DevCentral, RealIT, Real IT, DCC E3, Colin Walker

Real IT Video Blog - Jason Covers The Deployment Framework

Thu, 12 Mar 2009 00:23:58 GMT

Jason talks about the decisions we've made regarding the framework of our acceleration solution.

Technorati Tags: F5, DevCentral, RealIT, Real IT, DCC E3, Jason Rahm

Real IT Interview - Dan Wright Shows Off F5's Tech Center

Thu, 12 Mar 2009 00:15:01 GMT

Dan Wright gives us a tour of F5's Tech Center.

Technorati Tags: F5, DevCentral, RealIT, Real IT, DCC E3, Dan Wright

Real IT Episode 3, The Extended Team - Part 2

Wed, 11 Mar 2009 23:58:53 GMT
Episode 3, The Extended Team: This week the team checks in with all of the people they depend upon to complete the project in a timely manner, discussing the issues created by "on the fly" projects and fitting them into the schedule with a priority. Guest appearances by Dan Wright, Paul Petersen, Steve Wong, Bob Tacon, Erik Holcomb, and the APAC team..

Technorati Tags: F5, DevCentral, RealIT, Real IT, DCC E3

Real IT Inteview - Joe Talks VMs

Tue, 10 Mar 2009 23:33:28 GMT

Joe discusses the ins and outs of running simulated data centers in a virtual machine environment.

Technorati Tags: F5,DevCentral,RealIT,Real IT,DCC E3,Virtualization,Joe Pruitt

Real IT Video Blog - Jason Talks Acceleration

Tue, 10 Mar 2009 02:28:23 GMT

After Colin's talk with Dawn, Jason ponders the acceleration design and talks about some concerns he sees that the team should be aware of.

Technorati Tags: F5, DevCentral, Real IT, DCC E2, Jason Rahm

Real IT Episode 3, The Extended Team - Part 1

Tue, 10 Mar 2009 00:23:12 GMT
Episode 3, The Extended Team: This week the team checks in with all of the people they depend upon to complete the project in a timely manner, discussing the issues created by "on the fly" projects and fitting them into the schedule with a priority. Guest appearances by Dan Wright, Paul Petersen, Steve Wong, Bob Tacon, Erik Holcomb, and the APAC team..

Technorati Tags: F5, DevCentral, Real IT, DCC E3

Real IT Video Blog - Joe Reflects On Page Size and Caching In Dev

Mon, 09 Mar 2009 00:25:39 GMT

Joe reflects on his thoughts about the page load size for DevCentral and then gets into the effects of caching with regards to development testing.

Technorati Tags: F5, DevCentral, RealIT, DCC E3, VideoBlog, VideoBlogs, Caching, Joe Pruitt

Real IT Interview - Paul's Introduction

Fri, 06 Mar 2009 11:43:39 GMT

Paul gives us a rundown on his role as manager of the Network Engineering team and touches a bit on things that come up on a day to day basis for his team.

Technorati Tags: F5, DevCentral, DCC E2, Interview

Real IT Video Blog - Don talks new projects

Thu, 05 Mar 2009 11:54:59 GMT

Don discusses the choices you have at the start of your projects. Encouraging everyone to look into the things that may have been shelved, he talks about how and why this part of projects is the most fun and the right time to get creative.

Technorati Tags: F5,DevCentral,DCC E2,VideoBlogs,Don MacVittie

Application Designer Library:	http://devcentral.f5.com/wiki/default.aspx/MgmtPack/ApplicationDesigner.html
Holistic Monitoring and Auto Configuration using the F5 Management Pack and IIS 7:	http://devcentral.f5.com/weblogs/jhendrickson/archive/2010/04/23/holistic-monitoring-and-auto-configuration-using-the-f5-management-pack.aspx

Video tutorials:	http://www.vimeo.com/13548421 (Health Based Load Balancing Scenario - Overview) http://www.vimeo.com/13523437 (Health Based Load Balancing Scenario)
PowerPoint presentation:	health-based-load-balancing-scenario.pptx

Application Designer Library	http://devcentral.f5.com/wiki/default.aspx/MgmtPack/ApplicationDesigner.html
Holistic Monitoring and Auto Configuration using the F5 Management Pack and IIS 7	http://devcentral.f5.com/weblogs/jhendrickson/archive/2010/04/23/holistic-monitoring-and-auto-configuration-using-the-f5-management-pack.aspx

Approach	Average CPU Cycles	matches_glob	string match	domain	getfield	nested conditional
matches_glob	31116		4.25%	9.55%	10.89%	16.84%
string match	32496	-4.44%		5.53%	6.94%	13.15%
domain	34400	-10.55%	-5.86%		1.49%	8.06%
getfield	34919	-12.22%	-7.46%	-1.51%		6.67%
nested conditional	37416	-20.25%	-15.14%	-8.77%	-7.15%

message source	attribute name	regex	capture group	sample message
dashboard script	Compression Deflate uses	deflate\.out\.uses='(\d+)'	1	in dc post
dashboard script	Compression LZO uses	lzo\.out\.uses='(\d+)'	1	in dc post
dashboard script	Compression Null uses	null\.out\.uses='(\d+)'	1	in dc post
dashboard script	Dashboard-messageType	message_type='(.+?)'	1	in dc post
dashboard script	Dashboard-reportingSystem	HostName='(.+?)'	1	in dc post
dashboard script	Dashboard-routingEnabled	routing='(.+?)'	1	in dc post
NEDS iRule	NEDSv1-Flow-clientside-http	"(neds\.f5\.conn\.start\.v1)",(\"[\w\.resp\.v1]+\"\,)+\"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}\-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}@\d+\.\d+)\"	1	in NEDS Spec
NEDS iRule	NEDSv1-clientIPaddress	"(neds\.f5\.conn\.start\.v1","[\w\.]+)","(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})	2	in NEDS Spec
NEDS iRule	NEDSv1-clientPort	"(neds\.f5\.conn\.start\.v1","[\w\.]+)","(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:)(\d{1,5})	3	in NEDS Spec
NEDS iRule	NEDSv1-clientCloseBytesIn	(neds\.f5\.conn\.end\.v1)","([\w\.]+)","(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}@\d+\.\d+)",(\d+.\d+),(\d+),(\d+),(\d+)	7	in NEDS Spec
NEDS iRule	NEDSv1-clientCloseBytesOut	(neds\.f5\.conn\.end\.v1)","([\w\.]+)","(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}@\d+\.\d+)",(\d+.\d+),(\d+),(\d+),(\d+),(\d+)	8	in NEDS Spec
NEDS iRule	NEDSv1-clientClosePktsIn	(neds\.f5\.conn\.end\.v1)","([\w\.]+)","(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}@\d+\.\d+)",(\d+.\d+),(\d+),	5	in NEDS Spec
NEDS iRule	NEDSv1-clientClosePktsOut	(neds\.f5\.conn\.end\.v1)","([\w\.]+)","(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}@\d+\.\d+)",(\d+.\d+),(\d+),(\d+)	6	in NEDS Spec
NEDS iRule	NEDSv1-clientCloseTimestamp	(neds\.f5\.conn\.end\.v1)","([\w\.]+)","(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5}@\d+\.\d+)",(\d+.\d+),	4	in NEDS Spec
NEDS iRule	NEDSv1-clientConnectionIngressVlan	(neds[\w\.]+start\.v1\",\"[\w\.]+",")(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}@\d+\.\d+)\",(\d+\.\d+)\,"(\w+)"	4	in NEDS Spec
NEDS iRule	NEDSv1-clientConnectionPolicyName	(neds[\w\.]+start\.v1\",\"[\w\.]+",")(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}@\d+\.\d+)\",(\d+\.\d+)\,"(\w+)"\,(\d+),(\d+),(\d+),\"([\w\.]+)\"	8	in NEDS Spec
NEDS iRule	NEDSv1-clientConnectionStartTimestamp	(neds[\w\.]+start\.v1\",\"[\w\.]+",")(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}@\d+\.\d+)\",(\d+\.\d+)	3	in NEDS Spec
NEDS iRule	NEDSv1-clientIPProtocol	(neds[\w\.]+start\.v1\",\"[\w\.]+",")(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}@\d+\.\d+)\",(\d+\.\d+)\,"(\w+)",(\d+),	5	in NEDS Spec
NEDS iRule	NEDSv1-httpRequestHost	(neds\.f5\.http\.req\.v1)",("[\w\.\"\:\-\@]+)","([\w\.\:\-\@]+)",(\d+\.\d+,\d+),"([\w\.\_\-]+)	5	in NEDS Spec
NEDS iRule	NEDSv1-httpRequestServerPort	(neds\.f5\.http\.resp\.v1)","([\w\.]+)","([\d\.:]+)-([\d\.]+):(\d{1,5})	5	in NEDS Spec
NEDS iRule	NEDSv1-httpRequestTCPReplyNumber	(neds\.f5\.http\.req\.v1)",("[\w\.\"\:\-\@]+)","([\w\.\:\-\@]+)",(\d+\.\d+),(\d+)	5	in NEDS Spec
NEDS iRule	NEDSv1-httpRequestUserAgent	(neds\.f5\.http\.req\.v1)",("[\w\.\"\,\:\-\@]+)","([\w/\._\%\@]+)",("[\w\@\.]*?"),"([\w/\.\s(;\-\:\)]+)	5	in NEDS Spec
NEDS iRule	NEDSv1-httpResponseContentLength	(neds\.f5\.http\.resp\.v1)","([\w\."\,\:\-\@]+)","([\w\/\;\s\=\-]+)","(\d+)	4	in NEDS Spec
NEDS iRule	NEDSv1-httpResponseContentType	(neds\.f5\.http\.resp\.v1)","([\w\."\,\:\-\@]+)","([\w\/\;\s\=\-]+)	3	in NEDS Spec
NEDS iRule	NEDSv1-httpResponseLBTarget	(neds\.f5\.http\.resp\.v1)","([\w\.\,:\-@/;\s\="]+),"(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:\d{1,5})"	3	in NEDS Spec
NEDS iRule	NEDSv1-reportingSystem	(\"neds.+[\w]\.v1\"),\"([\w.]+)\"	2	in NEDS Spec
NEDS iRule	NEDSv1-responseHTTPContentLength	(neds[\w\.]+resp\.v1\",\"[\w\.]+",")(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}@\d+\.\d+)\",(\d+\.\d+),(\d+),"(\d{3})","([\w\/]+)","(\d+)"	7	in NEDS Spec
NEDS iRule	NEDSv1-responseHTTPServerResponseCode	(neds[\w\.]+resp\.v1\",\"[\w\.]+",")(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}-\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\:\d{1,5}@\d+\.\d+)\",(\d+\.\d+),(\d+),"(\d{3})"	5	in NEDS Spec
w3c iRule	W3C Client IP address	client_ip=(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})	1	in dc post
w3c iRule	W3C Client Port	client_port=(\d{1,5})	1	in dc post
w3c iRule	W3C Client username	username=([\w]+)	1	in dc post
w3c iRule	W3C Content Length	content_length=(\d+)	1	in dc post
w3c iRule	W3C HTTP Request	request="(.*)"\ss	1	in dc post
w3c iRule	W3C HTTP version	HTTP/(\d\.\d)"	1	in dc post
w3c iRule	W3C Host header	host=(.+?)	1	in dc post
w3c iRule	W3C Member server	lb_server=(\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}:\d{1,5})	1	in dc post
w3c iRule	W3C Server Response Code	server_status=(\d{3})	1	in dc post
w3c iRule	W3C User Agent	user_agent="(.*)"	1	in dc post
w3c iRule	W3C VIrtual Server name	virtual=(.*?)\s	1	in dc post
w3c iRule	w3c Server Port	lb_server=\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}:(\d{1,5})	1	in dc post
w3c iRule	w3c referer	referer=([\w\./\:\-]+)	1	in dc post
w3c iRule	w3c resp_time	resp_time=(\d+)	1	in dc post

Type of Statistic	# of Stats
Global (Device)	49
CPU	3
Chassis	1
Partition	5
Trunk	9
Interface	9
LtmNode	7
LtmVirtualServer	7
LtmPool	7
LtmPoolMember	7
GtmDataCenter	7
GtmServer	7
GtmVirtualServer	7
GtmWideIP	4
GtmPool	3
GtmPoolVirtualServer	3

DevCentral Main Feed

To all my SysAdmin friends I say - Run your databases through BIG-IP and sleep better at night

F5 Friday: Eavesdropping on Availability

INBAND and PASSIVE MONITORING

Related Posts

DevCentral Weekly Podcast 142 – Enterprise Manager And Some Pizza

A Storage (Capacity) Optimization Buying Spree!

WILS: Load Balancing and Ephemeral Port Exhaustion

Related Posts

Here Comes Payback Time. Prepare for Storage Shortages.

CloudFucius Investigates: The Comeback Cloud

Extreme Automation. Dynamic Control with or without the Cloud.

The World Doesn’t Care About APIs

I heard it said the other day, regarding the OpenStack announcement, that “the world does not care about APIs.”

WE DO. WE JUST CARE MORE ABOUT the MODEL

Maybe the point is that we shouldn’t because the API today is just a URI and URIs are nearly interchangeable.

COMMODITIZATION REALLY MEANS NORMALIZATION

Here’s the rub: having an API is important, but the actual API itself is not nearly as important as what it’s used to exchange.

Related Posts

Dell Buys Ocarina Networks. Dedupe For All?

The Battle of Economy of Scale versus Control and Flexibility

BUT CLOUD AUTO-SCALES and STUFF!

ECONOMY of SCALE MAY be INHIBITING SCALE

The irony is that the economy of scale offered by cloud may well be biting cloud in the proverbial derrière as it becomes the inhibitor to effective scale by limiting or making extremely difficult the architectural changes necessary for an application to scale in a cloud environment.

Related Posts

IT and Data: If Not Me, Then Who? If Not Now, Then When?

F5 Friday: Beyond the VPN to VAN

When SSL VPNs were first introduced they were a welcome alternative to the traditional IPSEC VPN because they reduced the complexity involved with providing robust, secure remote access to corporate resources for externally located employees.

THE NEXT GENERATION of ACCESS CONTROL

That change in focus has led to what should be the next step in the evolution of remote access – from SSL VPN to secure access management, to managing application access by policy across all users regardless of where they might be located.

Related Posts

DevCentral Weekly Podcast 141 - #CameraFail

Colin And Jeff Discuss Their Trip To Japan

F5 Application Designer: Health Based Load Balancing

1. The F5 Application Designer Extensions: a Quick Overview

2. Health Based Load Balancing - Overview

3. Prerequisites

4. Preparation

5. Scenario

6. Resources

F5 Application Designer: Load Balancing Scenarios

1. The F5 Application Designer Extensions: a Quick Overview

2. Load Balancing Scenarios

2.1. Health Based Load Balancing (Scenario #1)

2.2. Performance Threshold Rule Based Load Balancing (Scenario #2)

2.3. Performance / System Metrics (Provisioning) Based Load Balancing (Scenario #3)

Get Out Your Crayons. We Need to Color Us Some Bits

THE NEED STILL EXISTS

Interestingly enough, the need for prioritization and bandwidth management still exists and, in many cases, it could become of paramount importance to the successful implementation of a mature cloud.

BACK to BEGINNING

Sometimes the solution really is to go back to the beginning, to our roots, to the network.

EIGHT BITS

What seems clear, to me at least, is that eventually we’re going to run into scenarios in which we need something akin to Operations Performance Management (OPM) to ensure that management and control messages are delivered in a timely fashion.

Related Posts

Zombies! This Year’s Vampires Are Zombies!

Searching DevCentral Just Got Easier

OpenSearch

Creating the Search Engine

Exposing it to Browsers

Installing in FireFox

Installing in Google Chrome

Let’s Face It: PaaS is Just SOA for Platforms Without the Baggage

At some point in the past few years SOA apparently became a four-letter word (as opposed to just a TLA that leaves a bad taste in your mouth) or folks are simply unwilling – or unable – to recognize the parallels between SOA and cloud computing .

CLOUDS AREN’T CALLED “as a SERVICE” for NOTHING

Whether we’re talking about compute, storage, platform, or infrastructure as a service the operative word is service.

SOA IS ALIVE and LIVING UNDER an ASSUMED NAME for ITS OWN PROTECTION

If we return to the early days of SOA you’ll find this is exactly the same prophetic message offered by proponents riding high on the “game changing” technology of that time.

Related Posts

CloudFucius Asks: Will Open Source Open Doors for Cloud Computing?

Many Stops is Good for Vacation, Not for WAN Opt

HTTP Now Serving … Everything

I read an interesting article during my daily perusal of most of the Internet (I’ve had to cut back because the Internet is growing faster than my ability to consume) on “Virtual Micro Networks.”

What’s driving creation of Virtual Micro Networks

Context is Everything

Dynamic Infrastructure Needed

What makes this all even more complicated (yes, it does get worse as a matter of fact) is that not only is the same protocol used to deliver different types of data but in many cases it may be delivered to the same user in the same session.

Related Posts

Securing the Corporate Intranet with Access Policy Manager

A funny Thing Happened on the Way to the Branch Office…

Defeating Attacks Easier Than Detecting Them