With all the talk going around today about the stolen Twitter documents, security of personal accounts, and password strength, there have been quite a few posts from well-wishing security folks on how to keep yourself and your online data secure. One of the Good Idea(TM) posts was by Seth Simonds titled “5 Keys To Keeping Personal Information Safe Online.” The majority of Seth’s tips are about password security and how to use unique and unidentifiable passwords for your various online destinations.

Although the gist of his post is excellent, I’m not sure I 100% agree with his suggestions on using random, nonsensical text strings for either passwords or security questions. There are few rules that people follow when passwords are concerned:

• People tend to choose easy to remember passwords because it’s not something we’re used to (in the grand DNA scheme of things). Sure, we can remember our PIN for our debit card without a problem, but that’s only a 1:1 relationship, typically doesn’t change over time (sometimes over our entire lives), and is only 4 numbers. For a fun experiment: poll your friends by asking them when they last changed their ATM PIN. When forced to remember N:N:N website:username:password combinations, we default to easy and either re-use passwords or we go with something extremely easy to guess, such as DogsName2007.
• Seth’s comment to counter this is to go to the extreme opposite: create passwords that are unique to every site and are completely random, then write them down. This is almost as bad. Any password that’s so hard we have to write it down to remember will never be remembered without that sheet of paper. Consequently that paper will become commonplace around our workstation. It will be left out on our desk for visitors to see; it will be kept in our laptop bags for when we’re traveling; it will get thrown in the trash by mistake and then we have to start all over with 20 different websites and a new piece of paper. If something is too hard to remember then it will never be remembered; security that’s too hard isn’t good security.

But thankfully there’s middle ground between too easy and too difficult, one that works extremely well once it’s put into practice (and I’ll even thrown in a twist at the end for those feeling adventurous ;). In the spirit of sharing security tips on a day when it’s all over the news, here are my tips for creating and using both secure and easily recalled passwords without making them easy to guess. It’s three easy steps to create almost random passwords (at least random if you don’t know the pattern) that are unique to you and unique to every site you visit:

1. Choose a random two syllable compound word that has nothing to do with you personally. Let’s use ‘hotdog’ for this example. We’ll call this root word.
2. Grab the name of the website that’s asking you to create a password. You can either use the entire site name if you want a long password or you can use an abbreviated version. Our example will be ‘Amazon.’
3. Use numeral substitution for the website letters, changing ‘Amazon’ to ‘4m4z0n’. This is the only tricky part of the 3-step system if you’re not used to it, but after a while it becomes second nature.

Now concatenate steps 1 and 3 by appending the numeric website word to your root word to create a unique password for every site that requires login credentials. Here’s a list of example sites using the above system:

• Amazon: hotdog4m4z0n
• GMail: hotdoggm41l
• Technorati: hotdogt3chn0r4t1
• BestBuy: hotdog835t8uy
• Yahoo: hotdogy4h00

So on and so forth, and that’s it. This nice thing about this system is that it gives you flexibility to use unique passwords for every site, be able to remember them, and keep things secure by changing your root word as frequently as you would like. Next month your Amazon password can change from ‘hotdog4m4z0n’ to ‘dogsled4m4z0n’. The second part of the password never changes and is unique to each site; you only have to remember the root word. This system will also typically result in passwords >8 characters and includes letters and numbers, two key factors in good passwords.

And the twist? Use mixed case for your two syllable word: hotdog becomes HotDog. This is why a two syllable compound base word works so well for this system, the first character of each word can be capitalized.

This password system will allow you to use relatively strong passwords that are unique to every site without requiring pen, paper, or the memory capacity of an elephant.

I’m taking a break from virtualization to spend a few minutes with my roots: security. You know what they say: you can take the boy out of security but can’t take security out of the paranoid, tin-foil hat wearing, walls painted with wifi-blocking paint boy. :)

My co-worker, Peter Silva (the real security guy in the house), passed along some research yesterday about a group of individuals who claim to be able to predict a US citizen’s social security number based on a few small pieces of publicly available information – name, birth date, and birth city – and they say they can gather this data from your social media breadcrumbs. Now you’re probably thinking “Big deal. No one on the Interwebs(TM) knows where I was born or when, so I’m safe.” You and me both, friend, because we don’t use Facebook. Oh wait, you do you use Facebook? And you have your birthday posted on your page? And you’ve made comments on someone’s wall about growing up on a farm in Kentucky? Oh, well it sounds like you’ve just been socially engineered and you didn’t even know it. Watch that SSN#, you never know who’s looking. :)

Social engineering has long been both a staple and a foil of security. Traditional social engineering attacks focus on usernames, passwords, bank PINs, or “forcing” a user to do something malicious, such as connect a virus-laden USB key the found in a parking lot to their laptop. Password engineering is probably the most common, because most passwords are rooted in something users know extremely well and something they can remember. It’s fairly common to find passwords that include nicknames, anniversary dates, birth dates, pet names, spouse names, etc. Bob was born in NYC, loves the Yankees, has a dog named Chase, and was born in 1972. His password may be “YankChase72” If I know enough about Bob, or can coax certain personal information out of him then I have a better chance of brute force attacking his authentication credentials.

On the flip side of the password coin, it’s becoming more common to find businesses that use extremely specific personal information to verify identity: almost social engineering for good. The theory is that no one but Bob will know the address where he lived when he applied for his first credit card, or if his sister recently purchased a large plot of land in Montana. If Bob’s bank detects any type of fraud on his account, they can call Bob and ask him these questions that only he will be able to answer (in theory) and determine with a good bit of accuracy (in theory) if Bob really is Bob. While this model is a good start, it still suffers from flaws. This model relies on the real Bob actually being able to answer these questions correctly, but can anyone expect Bob to remember which apartment he lived in during college when he applied for his first credit card? Or maybe Bob hasn’t spoken to his sister in 2 years and has no idea she’s become Montana’s latest land barren.

This exact problem happened to yours truly recently. My credit card issuer detected what they thought was fraud on my account (turns out it wasn’t, they were just being over-vigilant) and I got a call from them letting me know what was going on. Props to them for being pro-active, but we had to have a few calls to clear things up due to too much social engineering security on their part. Here’s how the first call went (names and details have been changed to protect the innocent, ie me, so you can’t social engineer your way into my wallet):

Bank: Hi Mr. Murphy. I’m going to ask you a few questions to verify your identity.
Me: Ok.
Bank: What’s your SSN?
Me: 123-45-6789
Bank: What state did you grow up in?
Me: Colorado
Bank: Uhm…123 isn’t a social security prefix used in Colorado.
Me: I know, I wasn’t assigned that SSN in Colorado, I was assigned that SSN in the state I was born.
Bank: Oh. What state was that?
Me: Vermont
Bank: Ah, ok, that matches what we have on file. One last question: what color was the car you bought with your first car loan?
Me: Pancake Brown.
Bank: Thanks, Alan. You’ve been verified.

My personal experience coupled with the claim that SSN’s can be predicted based on personal information really got me thinking about the overlap of social engineering and social networking. Before the advent of social networking, I think most people were reticent to share personal information. If a stranger approached me on the street and asked me my birthday and birth city I’d assume they were trying to scam me. Now people willingly post this information for everyone to see, and freely share all types of personal information.

Another example of social engineering and social networking overlap is a recent story on an MI6 agent’s wife who posted their family’s home address on her Facebook page. “Why yes, my spouse does work in national security. And we’re having a garage sale at 123 Elm St. this weekend.” Couple what we’re willing to share in text on Facebook with what we post on YouTube and we’re building a table of contents for our lives that anyone can access, use, and steal. This (rhetorical) question gets asked too often these days, but will the Myspace/Twitter/Facebook generation have any sense of personal and private information as they grow up?

Me, I think some things should be kept personal, and there’s no need for the entire world to know that I was born in Vermont but grew up in Colorado, and thus deduce that I’m a ski bum. But I realize I’m in the minority. I’m only one of 2 people in my social circle who doesn’t have a Facebook page, and the two of us are standing strong in solidarity. When I want to know what he’s doing, I call him.

If you’re going to guess my SSN, then you’re going to have to do it the old fashioned way and trick it out of me in person – which in the end is easier than engineering my birth date, city of birth, and then trying to guess my unique serial number combination.

Note: F5 BIG-IP v10 launched earlier this month with 120+ features focused on the IT Agility and the Dynamic Data Center. This is the conclusion of a multi-part series on matching the needs of IT with the needs of business through IT Agility.

IT agility. Your way.

This “Data Center as a Service Mecca” won’t happen overnight and it won’t happen in isolation; that movement will need to incorporate fundamental data center changes to enable IT agility, to meet the changing needs of the business. There are four discrete functional  changes that need to occur in any existing data center to begin bridging the gap to the dynamic data center: unification, visibility,
context, and decision.

• Unification: Strategic technologies overlap in the data center and are
  brought together and managed as one unit. Open standards and APIs are
  in place for easy integration with new and existing technologies.
• Visibility: All application services in the data center become transparent;
  application data is managed bi-directionally in and out of the data center.
  Communication with off-board services, such as the cloud, originates and
  is controlled from inside the data center as part of a local, logical unit.
  Reporting and management data is gathered and managed in one central
  location.
• Context: Application and user data is managed and controlled as a single
  unit. Application access is dictated by the user environment for each unique
  data request. Reporting and management data is evaluated after every
  application response. Resource availability is paramount to evaluating user,
  application, network, and service contextual requests.
• Action: Resource unification, visibility, and context are brought together
  at every step in the application and data-deliver stack to move application
  data in and out of the data center in the best possible manner for that
  instance.

Each one of the migratory steps is required, in sequence, to propel the static data center of today into the dynamic data center of tomorrow. First, Unification (or Integration) across all disparate parts of the data center enables each of the units to move out of their silos into working together as an ecosystem service. Visibility gives insight into how each of these now unified parts of the data center work
together to deliver a service. Then Context provides the ability to make decision based on current circumstances and the status of each service within the data center, which is based on visibility data that’s been collected from each unified component. And finally there’s Action—acting on decisions spawned from context. Every service-enabled action in the data center will need to be driven by context, visibility, and unification.

As these four ideas are implemented throughout IT, the data center will begin to change from the traditional model of yesterday to tomorrow’s dynamic model. The data center will begin to move from isolated to shared, from network-centric to application-centric, and from a physical system focus to a services-driven architecture.

F5 BIG-IP v10: Enabling the Dynamic Data Center and IT Agility

Organizations, from the CIO down, are able to take advantage of this new service agility model to align IT with the goals of the business. By focusing on the data center—the core component to IT and business technology and application needs—businesses can better align their financial goals with technology goals. In the traditional model it was very difficult to create an agile environment where all of the services in a data center could be mobile and fluid and move with business need. With the new computing models introduced into the data center over the last three years, businesses can finally make this vision a reality by bridging the old with the new, the business goals with the technology needs, truly creating “IT agility. Your way.” New applications can be provisioned in a matter of hours rather than days, weeks, or months. Services can be moved between physical and virtual data centers, in and out of the cloud, as needed. Networks can be dynamically reconfigured as new applications are rolled out to customers. Disaster recovery can move from a one-to-one physical/secondary data center model to a one-to-many redundant “pay as you need” model. And most importantly, data centers are no longer relegated to physical buildings with walls; today, data centers can exist as virtual components built around applications.

The data center of tomorrow is no longer a vision on paper, or a futuristic “pie in the sky” idea. Due to rapid advancements in IT technologies such as virtualization, system management, and cloud computing, the data center of tomorrow has arrived today. This is a movement that addresses the needs of today’s CIO and finally brings the data center out of the annals of IT to the front of the business’
bottom line. The data center of today is built on services and applications rather than servers and networks. We now have the technology to define, design, develop, deploy, and manage the entire data center as a service. The data center of today is poised to deliver unified
application and data delivery services in a way IT has never before witnessed.

For more information on how F5 BIG-IP v10 can power your dynamic data center with IT Agility, visit F5’s site or the litany of excellent DevCentral content available right here.

Note: F5 BIG-IP v10 launched earlier this month with 120+ features focused on the IT Agility and the Dynamic Data Center. This is a multi-part series on matching the needs of IT with the needs of business through IT Agility.

F5 BIG-IP v10: Unifying Application and Data Delivery Services

A New Data Center Computing Model

Issues such as consolidation, shared services, and reducing costs are the CIO’s goals to help aligning business and IT, but they aren’t step-by-step roadmaps for IT. They’re more along the lines of mandates for IT. IT’s main concern is how to consolidate the data center while keeping the business online. The steps used by IT to achieve these business goals lie in the data center itself, in building a dynamic data center that is fl uid and can move with the business. In order for the data center to achieve this mobility and break out from the physical elephantine
traditional data center model, it must be designed with flexibility at every step, including the way computing is handled at the core of the  data center.

The primary driver to adopting a dynamic data center is the advent of a new computing model, a model where discrete resources are allocated for applications and are decoupled from physical boundaries (such as servers and data centers). The new model enables IT to build data centers around the applications rather than the network; to break through the physical requirements of primary and secondary single-purpose computing sites; and, most importantly, to begin treating all parts of the data center—network, IP, storage, and applications—as
fluid services rather than isolated technologies.

In the traditional data center, you have to install a new physical server, run cabling to a physical switch, and manage those components individually when you want to spin up a new application. Today, applications no longer need servers—they need services. Services are groupings of resources used by an application—such as computing resources (CPU, RAM, bus), network resources (bandwidth, VLANs, routes), and storage resources (SAN heads, platters, fi le systems). This new computing model—where you can “check out” discrete resources for any application rather than isolating that application to a physical box—is opening up new doors for where applications live, how they run, and what technologies you use to deliver those applications. Application services, and the infrastructure they rely on, become dynamic and fluid.

The Dynamic Data Center

The move to the dynamic data center is both an evolutionary progression as well as a revolutionary change. The data center has been in a constant state of evolution for decades, moving from a physically siloed vertical environment to a more unified, shared horizontal model. There has been a gradual uptake in the integration between storage and networking services and the network has moved into more of an active fabric role—a role where the network becomes an active participant in application delivery and can dynamically change to address application needs and work with the application as part of the delivery infrastructure—rather than simply a data transport commodity.

In the past three years, we’ve seen another revolutionary change in the data center: virtualization. Virtualization disassociates applications and services from the constraints of one-to-one hardware, enabling the data center to become more agile. The rapid adoption of virtualization in the data center has changed both the actual components that make up the data center, as well as how you approach them, by using these new technologies and services they create.

Virtualization, more than any technology we’ve seen in the past 20 years, has fundamentally changed the data center and has enabled businesses to stop treating the data center as a physical room full of servers and instead treat the computing resources within the data center—and even the data center itself—as a true service designed to deliver applications. Virtualization is the motivator for new computing models such as cloud computing and Infrastructure-as-a-Service (IaaS). This new computing model, at its core, is being driven by the need to leverage new data center technologies to deliver applications and services. Businesses aren’t building data centers to hold servers; they’re building them to deliver applications. If businesses fully embrace this new agile environment and change the model in which applications are delivered, the traditional physical data center model is unnecessary, as the dynamic data center can reside in any format in any location, focused on application delivery rather than hosting.

Next Up: Delivering IT Agility with BIG-IP v10

Note: F5 BIG-IP v10 launched earlier this month with 120+ features focused on the IT Agility and the Dynamic Data Center. This is a multi-part series on matching the needs of IT with the needs of business through IT Agility.

F5 BIG-IP v10: Optimize Your Business

IT is a necessary function of any business today, yet it’s often thought of as an independent entity within the organization. That separation is beginning to change as IT is recognized as the backbone of the enterprise and thus a cost center that impacts the business (along the lines of facility and HR costs). This has led CIOs and IT to engage in macro movements in data center design, focusing on:

• Consolidation: Paring down hardware and fixed costs by focusing on necessary services and resources.
• Shared Services: Doing away with the single-point appliance or service and moving to a more utility-driven model.
• Budget and Costs: Saving CapEx and OpEx costs associated with launching new technologies—getting the largest ROI from every data center decision.

These new data center initiatives all originate from a focus in the 2009 IT market that includes the rapid deployment of revenue-generating technology, making better use of information, and managing change. Through these changes, the CIO is both using technology to solve the business needs and using the business to optimize IT, creating a symbiotic relationship between two disparate organizations in the enterprise. Rather than dropping limitless money into IT to expand the infrastructure, the CIO is using IT to drive down costs throughout the business. The data center of 2009, in a term, must be adaptable to the business.

Like alleviating technology silos in the data center, this IT business movement is also bringing together groups within IT that have typically been segregated. Historically, enterprise IT has had individual departments for technologies such as servers, networking, and storage. Segmenting IT departments based on individual data center components is the antitheses to an agile and adaptable environment. For example, how can the application team begin thinking about moving their application into the cloud when the application is so tightly coupled with physical data center hardware owned by the systems team? As the CIO’s consolidation requirements trickle down to include shared services and focus on efficiency costs, these disparate IT roles and groups are beginning to overlap and work together towards a unified goal and achieving a greater ROI.

Next Up: Unified Application and Data Deliver Services

F5 announced today the release of the new F5 Management Pack for Microsoft System Center Operations Manager 2007 (sorry, wish I had an acronym or shorter name but a product this cool needs a big name like that :). In a nutshell, this new management pack extends the technology integration between Microsoft and F5 and will allow System Center users and F5 customers to:

1. Monitor the status of F5 devices and the entire Application Delivery Networking infrastructure as one unit, including application health, network statistics, and application traffic resource availability.
2. Integrate the Application Delivery Network and the back-end systems into one management piece, so application and systems can be managed together.
3. Automate application and network remediations as systems and Hyper-V events occur in the data center.
4. Provision new application delivery resources when new services, such as Hyper-V instances hosting applications, are spun up by System Center.

The System Center Operations Manager architecture allows partners to write their own plug-ins, Management Packs, and integrate those into the entire System Center family. F5 was able to easily adapt to the System Center environment by relying on iControl, the open management API used across all BIG-IP products for both monitoring and managing the devices. When two cleanly written, highly functional APIs come together…let the provisioning begin. :) Let’s look at an scenario:

• The System Center family of products manages both physical systems, such as application servers and F5 BIG-IP devices, as well as Hyper-V virtual systems with System Center Virtual Machine Manager (VMM). Let’s say System Center is managing an Application Delivery Network of Sharepoint web and search services running on both physical and virtual systems.
• All Sharpoint application traffic is routed bi-directionally through F5 BIG-IP Local Traffic Manager (LTM).
• Over the weekend, demand for Sharepoint is low and services have been spun down via System Center to conserve resources. At 8:00 AM on Monday morning, however, there is a spike in traffic as application requests ramp up for the work day.
• Traditionally, System Center would provision new back-end services to handle the spike: Hyper-V or physical server instances, Sharepoint apps and clusters, etc. This was a great first step for provisioning but required F5 BIG-IP LTM to be pre-configured to handle the max capacity (ie the Sharepoint pools had to be pre-populated), nullifying the benefits of elastic computing on the ADN.
• Now, with the F5 Management Pack, System Center can provision new services on F5 BIG-IP as well. As new services are spun up in the back-end of the data center, new application delivery services can be spun up at the same time in the front-end of the data center.
• When demand decreases, say on Friday at 6:00 PM, those same services can be de-provisioned on both the back-end servers and the front-end application network with F5 BIG-IP.

And that’s provisioning. In addition, the F5 Management Pack can relay critical application health and network data back to System Center to help make those critical provisioning decisions. If an application is slow to respond, System Center can now determine if that latency is on the application or the network side and make remediations as needed.

At the end of the day the F5 and System Center management integration makes sense for the applications. All application traffic to the physical and virtualized application serves is running through F5 BIG-IP; management decisions that are made on the back-end have a direct impact on that traffic flow and availability. Now, with the F5 Management Pack for System Center Operations Manager, that application traffic management can happen with the back-end physical and virtual services, treating your Application Delivery Network as a complete delivery system.

And the best part? You can download the F5 Management Pack for System Center 2007 right here on DevCentral. There’s no reason not to drop this in your System Center management platform today and start using it right away.

For more information on how this integration works, watch an example in action of empowering your Application Delivery Network with F5 and Microsoft. Of if you’re at the Microsoft Management Summit (MMS) in Las Vegas stop by booth 631 and talk to F5 directly.

Note: F5 BIG-IP v10 launched earlier this month with 120+ features focused on the IT Agility and the Dynamic Data Center. While those features are critical for administrators they’re also necessary for management on up through the CxO, albeit in a different format (technical details vs. applied benefit). For the next few days I’m going to be extrapolating the typical geeky feature list and talk about how BIG-IP v10 enables IT Agility on a data center level, helping to move your data center from the monolithic and static building it’s in today to a dynamic and mobile data center located on-premise, in the cloud, or a hybrid.

F5 BIG-IP v10: IT Agility. Your Way.

The term dynamic data center has been a prevalent, forward-thinking component of IT for the past few years. Like most technology buzzwords that include “dynamic,” the dynamic data center means different things to different people. Because the word dynamic has so many meanings when used in a technical context and rarely means the same thing to different audiences, the dynamic data center phrase often loses substance. On one end of the spectrum, the dynamic data center is nothing more than a marketing phrase used to make people think about how their data center is changing and evolving. It’s usually not backed up with much substance beyond “Tomorrow your data center will be different!” and there’s no talk about how the data center will change. Using the dynamic data center as a more generic term that applies to various business objectives is great for marketing, but it doesn’t address the business and technology needs of how the data center is changing. On the other end of the spectrum, the dynamic data center is a completely new computing paradigm with shared, yet secure resources providing a dynamic “Just-in-time (JIT)” computing model. While the long-term goal is to achieve a true JIT dynamic data center, the reality of the dynamic data center today lies somewhere in the middle of the spectrum, creating a true data center movement: using technologies readily available to better align IT needs with the needs of the business.

The Traditional Data Center

In order for there to be a new data center model there must be an “old” data center model to move away from. In reality, this is where most data centers begin—at the static, physical data center, resulting from decades of enterprise computing evolution and based on assumptions of how the data center delivers applications to users. Client server applications, for example, assumed a farm of physical servers were sitting behind a firewall and communicating with users through a traditional load balancer. Physical data centers were built to focus on speeds and feeds, with plumbing built to handle network congestion and traffic direction. This traditional model is still a tried and true architecture.

Historically, enterprises have built out these static, physical data centers from a central location, either by building a structure themselves or by leasing rack space from a hosting provider. These physical data centers require special considerations, such as protection against natural disasters (earthquake, hurricane, and tornado proofing), safeguarding from massive power runs, and the addition of large HVAC systems to maintain a safe running environment for the systems. And these data centers need large, single-purpose systems and servers to run applications; are network-centric with servers and applications built around network cables, racks, switches, and routers; and are isolated and self-contained, single-function structures.

But those considerations apply only to the first, or primary, data center. A secondary, redundant data center also needs to be built or leased as well, providing a rollover location for all application services in the data center should the primary become unavailable due to a human, system, or natural failure. In essence, every physical data center requires a geographically removed twin, just in case. Every part of the primary data center needs to be replicated at the secondary location and kept in constant synchronization with the primary.

While this bifurcated data center model works well for high availability, it’s an extremely expensive architecture to build and maintain. Capital expenses to create and monitor parity between systems and facilities within each data center, along with operating expenses to manage each location and keep each in sync, can continue to drain operating funds. Despite the challenges with this model, it has
been the de facto architecture for redundant system and application distribution. When an application is mission critical, virtually no expense is too high to change this architecture, even if maintaining multiple geographically disparate data centers is required. However, this large capital expense outlay to build multiple, monolithic application delivery data centers can be an inefficient use of IT budget, consuming dollars for services that sit dormant for days, months, and even years. That high cost barrier to entry for mission-critical applications and services is changing, and new computing models are attempting to do away with that barrier all together.

Next Up: Optimizing Your Business with and for IT Agility.

Enabling the Dynamic Data Center with BIG-IP v10

BIG-IP v10 is a breakthrough in the Application Delivery Networking (ADN) for many reasons, all of which revolve around understanding how applications are delivered outside the data center to users and services (ie context). Contextual awareness is a cornerstone to Unified Application Delivery and Data Services (UADDS) – delivering applications in out of your data center, your hosted environment, your cloud, in a manageable and predictable manner. One of the revolutionary UADDS features in BIG-IP v10 is Inband Passive Monitors: non-obtrusively monitoring application delivery and making adjustments before there is a problem.

There are (or were at least, until BIG-IP v10) two primary types of application monitors: active and passive. [NOTE: For more details on these types of monitors, check out the whitepaper links at the bottom of this post]. Active monitors are like quality control, taking random samplings of the applications to make sure they’re up and running. Not all applications can tolerate active monitors, however, which can actually jam up the queue on the app server, eventually building up enough to crash the server. In this case, the active monitoring itself can have a detrimental effect on the application server it’s trying to monitor.

Traditional passive monitors are like the people that watch for broken Oreos as they come off the Nabisco assembly line before packaging; if there are too many broken cookies then the line is shut down for repair. F5 pioneered passive monitoring years ago, and for good reason; they serve their purpose and place but they can’t fix the problem while it’s occurring. Passive monitors only see the Oreos after they’re all smashed up and lost, not before when they’re still beautifully round cream-filled mini-sandwiches or during production as they’re being crushed.

So what’s the solution? Inband Passive Monitoring. Combine the best of both types of monitors and create a monitor that:

1. Transparently monitors all application traffic to a service, and
2. Can insert a remediation immediately in the application delivery process when a problem is detected

Let’s look at an example:

• You’re running a Sharepoint service behind a VIP with 100 Sharepoint nodes, with an Inband Passive Monitor specific to Sharepoint attached to each node. The failure threshold is set to 10 failures/minute (more on that below).
• BIG-IP detects that Node #39 has started returning “503 Service Unavailable” HTTP responses, yet the rest of the nodes are fine. But node #39 is only doing this about once every few minutes. BIG-IP is configured to resend the request when this happens rather than passing the 503 onto the user. So far the re-requests are working fine, returning happy “200” responses on the 2nd pass.
• As traffic increases so does the rate at which Node #39 is returning 503s, unfortunately; something isn’t right. As soon as #39 exceeds 10 503 failures/minute, the Inband Passive Monitor takes over and does a few things:
  1. Takes #39 out of rotation (the node pool) so no new requests will be sent to the dying #39.
  2. Invokes an active monitor on #39 to determine if the problem is consistent, even if no active monitor has previously been attached to this service – an autonomous monitor as it were. The active monitor will continue to test #39 out-of-band based on a threshold configured by the administrator. If #39 shapes up during that threshold the Inband Passive Monitor will return #39 to the assembly line with a pat on the head, all trouble forgotten. If, however, #39 fails to shape up, it’s time to ship it out. The node is marked down for good, the administrator is notified, and the Inband Passive Monitor assumes that #39 is headed off to the application server farm in the sky.
  3. The Inband Passive Monitor will continue to monitor the “location” where node #39 used to live, and once it sees that there’s a new #39 it will make sure that it’s working correctly before adding it back into the active pool of nodes. Slick.

As you can see form this example, the Sharepoint application service associated with the BIG-IP v10 VIP never experiences an interruption. The few requests that would have traditionally received a “503 Service Unavailable” error were never the wiser since the Inband Passive Monitor proxied that failed response and re-requested the service on behalf of the requestor. And the one faulty node was detected without actually engaging the node until the faults exceeded the administrative threshold.

This is just the tip of the iceberg. You can read (and listen!) more about Inband Passive Monitors in the whitepaper or via the F5 OnDemand Audio Whitepaper version, your choice. Either way, you’ll find more details about why BIG-IP v10 is helping you to revolutionize contextual application delivery in your dynamic data center.

Enabling the Dynamic Data Center with BIG-IP v10

As you probably saw this morning, F5 BIG-IP v10 has landed, marking a change in the data center, creating a dynamic, agile IT data center based on Unified Application Data and Delivery Services (UADDS). Yes, it’s completely fine to stand up at your desk and start cheering on the revolution!

Under the hood there are many revolutionary technologies that enable UADDS, and one of my favorite features of BIG-IP v10 is Resource Provisioning: the ability to slice up hardware compute resources (CPU and RAM) to different modules on the system. For example, if you’re deploying BIG-IP LTM to serve mostly as your application firewall with BIG-IP ASM, you may to allocate more resources to ASM than to LTM features such as iRules. This gives the application administrator complete  control over granular BIG-IP resources, allocating them to modules as needed. Resources can also be added to evaluation modules allowing a test environment to scale the available resources to an evaluation module as needed for stress testing. 

Hardware virtualization slicing has suffered from some challenges in the past. Although modern virtualization does carve up hardware resources for sharing amongst multiple virtual systems, BIG-IP v10 Resource Provisioning is different from the traditional 1:Many hardware virtualization approach. In the traditional model, there’s no isolation between virtual systems using shared hardware resources. All processes from all virtual processes are handled in a “flat” manner by the physical CPUs. With BIG-IP’s isolation model specific portions of the hardware are allocated for a specific virtual instance and contained within that fenced area.

And even better than just slicing, Resource Provisioning also allows dynamic growth, or bursting, of resources as needed and as defined by the administrator. One of the problems with traditional hardware virtualization slicing is that it creates hard barriers that can starve resources. BIG-IP v10 Resource Provisioning removes this limitation by allowing the modules to burst and share resources when need dictates. The resource bursting policy can be defined by the administrator with hard percentages or with soft limits, allowing BIG-IP to allocate resources as needed during resource limitations.

In a nutshell, Resource Provisioning is a perfect solution to help manage computing resources as part of your Application Delivery Network (ADN). For more information on Resource Provisioning, check out the whitepaper or grab the F5 OnDemand Audio Whitepaper and listen to it on the go.

I typically try to stay away from prediction lists – that’s not to say that I don’t think about and plan for the future, I just try to avoid saying “In the Year 2000…” (in my best falsetto) at the beginning of each new year. There are so many unknowns, things we can’t predict or even begin to imagine will happen, that interject themselves throughout the year and send things all askew. It’s the old The Butterfly Effect cliché in action; prediction lists are usually either too specific to be accurate or to broad to be applicable.

That said, however, I do think that 2009 will be the year we see the enterprise virtualization focus mostly shift away from virtual platforms and application-focused VMs towards virtualizing the desktop. Those platforms and app VMs will still be a requirement, and with the pending release of VMware’s vSphere we’ll continue to see this area grow, but not with the focus and attention that we’ve seen over the past two years.  This year I think we’ll see VDI step into the spotlight and get all the attention.  IT need is driving this to some degree; VDI (in theory) will help manage costs – hardware, management, headcount, etc – and will facilitate the continual growth of telecommuting (something I imagine more enterprise will embrace as they save on facility costs this year). But more than need, VDI is The Next Big Thing (TNBT) in virtualization, the next cool technology that all us engineers love playing with and seeing what cool things we can do with it. Virtual platforms are so 2008. :)

But there are some major challenges with VDI; some similar to the challenges we saw (and are still seeing) when virtual platforms sprang from desktop playing to data center production overnight, and some very specific to virtual desktops. In particular, there are two challenge areas that I’m concerned about with mass adoption of VDI:

1. Delivery Performance: The #1 challenge I see to VDI is figuring out how to optimize and deliver images stored in a massive centralized data center out to users all over the world over varying network conditions. The same challenges we see for delivering any network application over latency-heavy links will become painfully obvious when screens are continually re-drawn. The big players, VMware and Citrix, are both creating systems to optimize and segment delivery both on the wire and on the back-end (check out VMware View’s integration with ThinApp and their support for thin clients as examples). Even the LAN, for VDI solutions inside the office, will have to adapt to massive data increases on the wire with VDI.
2. Video Performance: One of the ongoing challenges to virtual machine clients on the desktop is video graphics performance. Rendering advanced graphics through a hypervisor is extremely expensive process-wise and not trivial to implement. On a standard local client, such as running Vista on your laptop, things like 3D image rendering happens with direct access to the video card; in other words, it’s done in hardware, not software. With virtual guests that rendering has to be done through a software layer, and we’re seeing companies every day release minor tweaks to allow incremental support for advanced graphics. Virtual Computer has built a specialized implementation of Xen to handle 3D graphics. Parallels’ latest Desktop 4.0 build release fixes a bug in a game manufacturer’s facial rendering engine so players can now smile.  Yep, A patch had to be added to show virtual happiness. :) This graphics limitation is going to severely limit who can deploy VDI: enterprise users updating XLS and DOC files, no problem; medical, design, gaming, video – any graphics-dependent industry will have to wait.

And of course there’s always management and security, but those are more fun to talk about after the solution has been deployed, right?  ;)

Solutions exist today for #1, network performance. Many, many companies are looking at ways to optimize delivery on the software side, improving the terminal services/RDP experience, but this will take time.  Tools like BIG-IP LTM can optimize the VDI experience today by managing access to and delivery of VDI images, and in fact we have a great deployment guide on using LTM with VMware’s VDI products. #2, the graphics problem, well, that’s going to take some time. The VDI companies will continue to work on solutions for rendering high-end graphics, but they’re bound by physical hardware limitations in architecture and deployment. That will be something that needs to be overcome, and we’ll need to move away from this “VDI build 0.42.1 addresses the ‘mouth won’t open’ glitch when a character eats bacon” model and support 3D rendering out of the box.    

There are still a few barriers to mass adoption, for VDI to become an “everything, everywhere, for everyone” technology. 2009 will definitely see companies pushing out VDI to the masses, either because they’re immune to or solved the two challenges above, or because they want to push the envelope and test out any technology that can possibly save them money and make life easier. Until then, I’m fine running good ol’ XP in a VM without any fancy bacon-eating character graphics. :)

Sticking with my VM/Roadwork analogy; seems to hold up rather well.  :)

I've been working a good bit lately on VDI architecture in the data center.  Not so much the implementation of things like VMware View and building brokers and such, but one level up on topics like application traffic management and universal access and user policies.  Fun stuff  but I do wish I could add a few more hours in the work day so I could spend more time on these ideas - I still haven't given ThinApp the time it deserves (don't tell the dog: he expects any extra hours in the day to be dutifully turned into trips to the dog/snow park).

One of the ideas that's been picking at me lately is the migration of client data from the user network into the data center with VDI.  I telecommute and have a home computing environment that keeps my F5 VPN connection (to BIG-IP SAM, Secure Access Manager, no less ;) back to corporate sandboxed: my internal work data (email, intranet) flows over my SSL VPN, and the rest of my work-related traffic (reading the internet, blogging, tweeting) goes over my direct connection.  There is no sharing of data - I only route necessary work data through the VPN.  I can do this because I have control over my local work computing environment (along F5 IT, of course, who enables this via my SSL VPN access policy) - my application traffic is sourced from my home office so I can control how it leaves my home office.  My networking and routing environment is local. Contrast this model with the traditional VDI model (granted this is a bit extreme for example and assumes that all of my work-tethered network traffic is pushed through my remote desktop).

This type of distributed routing architecture changes, however, when users move to a fully remote and distributed VDI implementation where we don't own our desktop networking.  My "local" desktop is now hosted in a data center along side 100s of other "local" desktops as well as non-desktop applications.  The only work traffic that comes and goes in my home office is desktop/GUI transport and encapsulation (insert your choice of RDP, ICA, or some future transport protocol here) funneling everything back to the corp data center.  All my user-spawned work-related traffic - the stuff I normally spawn from my local desktop, such as browsing my Google Reader feeds - is now created in a remote data center and sourced out of that data center. The application network request doesn’t come from my home office. This is good (in theory) for my local broadband connection because I'm not using it for anything beyond my VDI client.  This is bad, though, for the network connection in and out of the data center that's hosting my desktop. Now all my user traffic is sharing ports, switches, and pipes with everyone else’s desktop traffic. And bad for my user experience because my source traffic is now competing with everyone else (just like I was in the office, which is one of the reasons I benefit from working remotely today) and then re-packaged and tunneled down to me, adding processing and delivery time to my experience. I get hit on waiting for my requests to leave and return to the data center and to have the response remotely displayed and packed for delivery over VDI. 

The best example of this is Web 2.0 traffic, sites like Google Maps.  A quick map request for Alexandria, Egypt consumes ~1Meg of HTTP traffic from desktop request to desktop response. Today for me, that's 1Meg of traffic on the segmented non-VPN’d portion of my broadband connection and it's easy- takes ~5 seconds to pull up Maps, make the request, and get back a sat picture of Alexandria. . My employer doesn't have to manage or deal with that traffic. But when I'm using a remote desktop, that's 1Meg of traffic round-trip from my VDI image in a data center, going out the link shared with the rest of my co-worker's VDI images, and most likely some other non-VDI traffic, and half of that (the response) is packaged for delivery over RDP/ICA in a GUI. I'm not generating that request from my home connection - I'm generating it on and over work's network resources from my VDI image.

So at what point does the benefit from having managed desktops in the data center (and believe me, I think there are huge benefits to this model from the management and security sides) outweigh the cost and limitations of moving all of this user-based traffic into the data center?  What impact does all this VDI client traffic impose on the data center network? There are network-level solutions for traffic management and optimization, such as BIG-IP LTM, optimizing RDP/ICA to the user on one side and optimizing HTTP on the other. But what about outbound from the data center, and more importantly for the best user experience and security issues, correlating the client-side traffic with the data center-side user traffic?

I’m really looking forward to what VDI looks like a few years in the future, when managing remote desktops is not hampered by discreetly managing user traffic in the data center.  We’ll get there, as long as we keep thinking about how all these new virtual technologies impact the network. Think about the infrastructure first and we’ll reap the benefits. 

Lori and are on the sharing wavelengths today (well, except for ideas on remote vs. local software testing, but we can’t always agree on everything, right?  ;) ).  She hit the app side, so here’s a take on the virtual platform side.

I just read Andreas’ post about “virtual routing” and moving VMs to match and optimize traffic flows (yes, it’s a few days old, but catching up on email and blogs takes time after vacation) and I’ll have to say my head started spinning – and not the good kind of spinning. The basic suggestion is that because we can move VMs around as needed we should move them constantly to match traffic patterns and destinations on the network. As traffic patterns and routes change, we move VMs to follow the routes Really? You want to, in essence, move data centers to follow traffic patterns over the same network that’s supporting said traffic, constantly? 

Let's look at this from another perspective: imagine you own a local chain of gas stations. These gas stations provide services for users (gas, food, coffee, etc) as they travel along various highways. They come to you when they need something, said service, otherwise they drive right past you. Your gas stations are located along major interstates at various exits as well as smaller highways and byways throughout the state. Your business accelerates during high-peak travel times (rush hour, weekends, vacation periods) and drops off during slow times (2:00 AM, Christmas, etc). In this slower economy, you're looking for ways to maximize your revenue and you've noticed an interesting trend: drivers are staying closer to home on the weekends and shopping local, avoiding long travel trips in order to save money. You typically don't have gas stations on local 2-lane roads and you can't afford to open more stations, so you're faced with two options:

1. Move your gas stations from the highways to local 2-lane roads: Sure, it's an option, but you would be sacrificing valuable peek highway revenue.
2. Drive traffic from the towns to the highway and to your gas stations: Using signs and promotions such as "Free Coffee with Fill-Up!" - "2 Hot Dog Burritos for $1 with Large Coke!" - "5¢ Gas Discount for Local Residents!" - you can somewhat control how people find your gas station. You can control traffic flow to your service station.

What Andreas (along with Doug) is suggesting is a morphing of #1: Moving the gas station 2 times/day to accommodate traffic patterns and ignoring how and why traffic is driven to your station. Gas station is on exit #421 during morning and evening rush hour and then is moved to the intersection of Maple & Elm during off-peak hours. There’s no care or concern about the traffic patterns, the station is just following the flow. 

This doesn’t resonate with me on many levels. First off, the basic principle of moving a server to match the traffic rather than managing traffic to the server. Traffic is smaller and easier to manage than servers, even with virtual servers. Managing constant VM movement with something like vCenter, clusters, resource pools, etc, is a surmountable task. Second, the process of moving the VMs themselves will have a cost wrt management, downtime, bandwidth (in the event Storage vMotion is used to follow the traffic). As an example, a quote from the post:

“VMotioning nodes (servers) to optimize the flow of traffic on the network”

Hmmm…How will moving a server optimize the traffic flow to that server? Again back to our gas station example: moving the gas station won’t change the traffic patterns at all. And are we talking LAN or WAN? If we’re talking LAN, then there should be traffic management devices, like BIG-IP LTM, in place already to manage optimized access to the apps running on the VMs. If you’re talking WAN, moving VMs cross-WAN on the fly has its own set of issues with traffic and storage.

Your goal when pulling into a gas station is to request a service: gas, coffee, food.  Moving the gas station just addresses cars driving into the parking lot. As always, we should be focused on the service being delivered in addition to the service station.

Don’t get me wrong: the mobility is one of the coolest benefits of VMs, and tools like vMotion and Storage vMotion are definitely changing the way we think about how and where we locate servers. But moving the servers instead of managing the traffic just because we can ain’t the way to go, IMO. Build the platforms for the services, build the infrastructure for the services, and then manage the traffic and access to the services. We have ADCs, let’s use them and manage the traffic and not micro-manage the service stations. And this quote:

“…solve for least switch hops per flow”

I think there’s a whole ‘nother blog post on how this idea becomes moot with something like DVS across multiple physical platforms. But I’ll take a break for now. :)

About two years ago, I was part of the team at F5 that helped design and build our Technology Center – a fully functioning showcase lab housing  all of our products as well as other infrastructure and application partner technologies such as VMware, Microsoft, Dell, etc.  My role (besides cable monkey during install week ;) was to define and isolate the security risks associated with guest virtual machines and segmentation (both physical and logical) of the virtual network and working spaces.  Probably one of the coolest and most fun projects I’ve ever been involved in; the Tech Center really is a work of Data Center art. 

One of the areas I was most concerned about from the get-go was the portability and availability of guest VMs; customers and partners can bring in their own virtual machines to provide back-end services during their testing time at the TC and we’ll run those on our virtual platforms. Now while all of our partners are trustworthy people, we paranoid types know that trust in the security realm is a tough thing to come by, so we needed to create a fully-functional yet safe environment for these VMs to run -- safe for them so there’s minimal sharing of resources with other customers and partners that may be visiting, and safe for us since it’s our infrastructure. From day one I’ve been concerned about the check-in/check-out possibilities with virtual machines; it equates to building a secure box that’s ready to be racked, and just before you rack it you let a stranger take it home for a week, then that stranger brings it back, you rack, cable, boot, and let it run loose.  Scary, huh. 

We solved that problem in the Tech Center using extremely tight networking configurations (walled gardens are your friends) via BIG-IP LTM and hardware isolation for our partners and their specific virtual machine environments, with good management on top to make sure everything functions as expected.  To date it’s been an excellent solution.

Fast forward two years later and meet VMware View: VMware’s VDI management solution that, in theory, should lead us down a path of near complete mobile desktop computing which, as a side benefit, should drastically improve desktop security.  With one huge exception: Offline Desktops. Still an experimental feature today, Offline Desktops allow an end-user to check out their VDI image (which is a complete VM: OS, apps, everything) and take it with them. Let’s say Alice (or maybe Mallory is a better example? ;) is in the office on Tuesday morning running her desktop remotely over the LAN, she checks out her desktop and hops on a plane, works on the flight, and then jumps on the VPN when she’s in her hotel and check her desktop back in, changes and all. Rinse. Repeat. Cool idea with huge benefits, but…

The paranoid me writing this blog immediately started locking doors, lowering blinds, checking the lamps for bugs, etc. I see this as a huge security risk: once the desktop is out of the DC and out of IT’s control, it’s immediately suspect. Just like one of the primary use cases for NAC where external, unknown laptops aren’t allowed to jack directly into the corporate network without auth and some level of validation and sanity check, any VDI image checked out via View and then checked back in should be treated as a new, external device, requiring quarantine, inspection, sanitization, re-authentication, the whole ball of wax. The risks are even more severe for VMs that are allowed to leave and re-enter the data center due to shared resources that run those VMs. The risk is no longer limited to a segmented network; it now extends to the entire VDC platform.

Could this simple, highly usable and beneficial feature open the door for VDI Bombs? A Frankensteinian marriage of trojan bombs (plant today, blow up on delivery or at a later time) delivered via VMs and targeted towards the host hypervisor, network, or CPU? Are IT departments going to build sophisticated quarantine environments for VDI VMs that are checked back in, and if so, will those tools be available soon enough to catch these bombs before time runs out? Or maybe this is a perfect opportunity to see VMSafe used as intended. Seems like the check-in/sync operation would be the perfect time to scan the guest kernel, filesystem, VM tools, etc, via VMsafe. 

I think a completely mobile virtual desktop is a great idea, even if it just pacifies us until true application virtualization becomes a reality, especially for IT client management.  But how much more management and security will be needed in the data center to make this a functional reality? Too much to justify the benefit? We’ll see. And while this concern has been around for a while, even when we were planning for somewhat mobile server VMs, VMware is the first company to make this easily accessible to end-users.  This scenario is applicable to any virtual machine architecture where VMs can come and go in and out of the data center. This issue is not restricted to just VMware.

A lot to think about as VMs come full circle from end-user desktops with Workstation through the Data Center with ESX and now back to end-users with View, and hopefully a good chunk of that thinking involves security planning for portable VDI images. I already have ideas for bumper stickers: “Stop VDI Bombs!”

Yesterday I blogged about how the lack of planning and scalable provisioning of any kind can cause major problems when traffic spikes hit. Today let’s look at how companies who are planning for spikes are preparing, even if it’s not completely elastic.

Rich Miller had an excellent post-election blog post a few weeks ago on sites scaling up for election traffic on Data Center Knowledge. As he points out in a later post, traffic hit record levels through Akamai's CDN on election night. Some companies adequately planned for the burst, other didn't. Spike management isn't something new, however we do deal with massively larger amounts of traffic than we have in the past, and our traffic usage is different. An election that everyone is watching is an excellent case study for these new traffic patterns. Me, I was sitting in front of the TV on election night with my laptop open to MSNBC and twitter, CNN mobile on the iPhone (primarily b/c I enjoyed seeing all those 404 and 500 errors that were showing up on CNN mobile; I know, I'm evil :) ). And I'm guessing this was the norm for people who use the Internet as their primary news source, like me. And the company responses that Rich covers, that did plan for the election spike and anticipated this flood of traffic, are interesting to me on two fronts:

1. The lack of the V-word: Surprisingly in this day and age, none of the companies interviewed said they were relying on any virtualization solutions to scale for their traffic. All the remedies involved physical servers and physical space in a data center or with a hosting company. But with all the hype (and b/c it's all I think about all day), I expected to see something about VMs or virtual storage as part of their spike management plans. On one hand this is encouraging that yes, the world can still spin without VMware or Microsoft virtual platforms. On other other, though, the election should serve as a perfect use case for provisioning and scaling using tools like virtual machines. This election is the best example I can imagine for "elastic computing," and I'm surprised that it wasn't first in responses from these companies. The ability to provision up and de-provision down as need based on real-time, immediate traffic needs is the long term bread-and-butter for virtual platforms; companies like BlueLock and Joyent know this today and have built virtual hosting solutions around provisioning scale for both infrastructure and the applications. So why not use the virtual tools available today as part of your scaling and provisioning needs, rather than having to plan for a spike by pre-ordering batches of servers and waiting weeks for them to go online?
2. Focus on the Apps: I have to say it warms my heart anytime someone mentions applications in the data center -- I'm a softie for those darned apps! :) All of the examples in his post were customers who were expecting an increased need for their application: a political blog, a CDN that hosts political websites, Twitter, etc. Their concern isn't with scaling core infrastructure (switches, routers, cables, trunks, etc), it's with scaling the application platforms (servers, OS', webservers, etc); again, props to those hosting providers who have already built out virtual infrastructure platforms to allow VM and application scale and provisioning as needed.

The phrase "old school" kept popping up in my head as I was reading the post. Are these companies sticking with what works, what's tried and true, by provisioning physical servers well in advance of the expected spike? Or does this show that virtual platforms are still in their infancy and companies that know how to plan for and manage massive amounts of application data traffic don't yet trust virtual solutions? I would probably lean somewhere in the middle, and until virtual platforms and dynamic provisioning proves itself, we'll continue to see dynamic provisioning in the VDC as more of a test case rather than a real-world use case.

For those who follow the music business, or for anyone that just reads and/or watches the news, you’ll know that 80’s metal icons Guns n’ Roses finally released the much anticipated “Chinese Democracy” album on Sunday, Nov. 23rd, 2008.  I say finally because we’ve been waiting on it for years.  Every year just before Christmas rumors began to fly about a possible release date that year.  And this year was no different than other years; those of us that were actually looking forward to this release hedged our bets once again and said “Sure, I’ll believe it when I see it,” including Dr. Pepper.  Who even knew the Dr. was a Guns fan? 

See way back in March of ‘08, Dr. Pepper ran a campaign – if Chinese Democracy came out this year, anyone who wanted one would get a free 20oz bottle of Dr. Pepper.  I’m assuming Axl took that as a throwdown challenge becuase he delivered, leaving the Dr. to start giving out free bottles.  The Dr. came through on their end of the bargain and opened up their website for the free 20oz requests for 24 hours, only valid on the day Chinese Democracy hit shelves, Nov. 23rd.  The official word came down about 2 months ago that indeed, Democracy would actually drop as expected (with distribution and promo details to back it up), leaving the great Dr. ~2 months to plan for the onslaught of freebie requests.  If we know anything about humanity it’s that no one can say no to “Free.” 

I’m not sure what Dr. Pepper was expecting with their offer, but it certainly didn’t plan for the flood of internet traffic that would be directed their way beginning at 12:01 AM EST on the 23rd.  Within hours their site became unresponsive, going through phases of availability: first, the intro Flash got stuck with no way to skip past it; then they finally caught on and started redirecting all users to a text-only portion of the site where the “Free Dr. Pepper” link was buried way down at the bottom which directed to a page that wouldn’t load; and finally, the coup de grace, the entire site went down and was returning nothing more than a 503 Service Unavailable error.  As of this writing, we’re still at 503, and I’m still sans a Dr.

This very temporary unplanned (well, they had two months to plan, but let’s stick with un- for a second) outage is a textbook case for elastic provisioning: spinning up services infinitely to address temporary need and then spinning those services back down when traffic returns to normal levels.  It’s like cable or satellite Video on Demand requests spiking during a snow storm on a Friday night.  Services that normally handle X amount of traffic are suddenly forced to handle exponentially larger volumes of traffic, typically unexpectedly, but they still need to deliver that service.

To give them the benefit of the doubt, maybe Dr. Pepper didn’t anticipate the massive traffic volumes brought on by every major news source in the country picking up on the Gn’R bet, but that’s the point of elastic provisioning: accommodating service need without anticipating fixed levels. All they needed was the ability to scale for 24 hours, then things could have been returned to normal. After the 24-hours, a simple splash page could have been displayed stating that the offer was over and directing their real website traffic to the original site via a click-through. Caching that notice page with a BIG-IP LTM could have kept probably 99% of the traffic off the real site until the hype died down by delivering that notice directly from LTM, and throttling the non-legitimate site requests, rather than putting the load back on the webservers themselves.

So what went wrong? How much is this free offer (which has now been extended 18 hours, although it doesn’t matter if the site is still down) going to ultimately cost them in marketing, name, and not so good press? I’m guessing less than it would have cost to spin up a temporary site with the free offer that could scale for 24 hours.

And this one hits home: I rarely drink soda today but do have fond memories of the Pepper; I was looking forward to throwing on some 80’s hair metal and kicking back like I was in 7th grade again. But the Dr. teased me; they offered and didn’t deliver.  At least Axl finally did, but I’m not waiting 17 years for a free Dr. Pepper; I’ll stick to coffee.  :)