Sticking with my VM/Roadwork analogy; seems to hold up rather well.  :)

I've been working a good bit lately on VDI architecture in the data center.  Not so much the implementation of things like VMware View and building brokers and such, but one level up on topics like application traffic management and universal access and user policies.  Fun stuff  but I do wish I could add a few more hours in the work day so I could spend more time on these ideas - I still haven't given ThinApp the time it deserves (don't tell the dog: he expects any extra hours in the day to be dutifully turned into trips to the dog/snow park).

One of the ideas that's been picking at me lately is the migration of client data from the user network into the data center with VDI.  I telecommute and have a home computing environment that keeps my F5 VPN connection (to BIG-IP SAM, Secure Access Manager, no less ;) back to corporate sandboxed: my internal work data (email, intranet) flows over my SSL VPN, and the rest of my work-related traffic (reading the internet, blogging, tweeting) goes over my direct connection.  There is no sharing of data - I only route necessary work data through the VPN.  I can do this because I have control over my local work computing environment (along F5 IT, of course, who enables this via my SSL VPN access policy) - my application traffic is sourced from my home office so I can control how it leaves my home office.  My networking and routing environment is local. Contrast this model with the traditional VDI model (granted this is a bit extreme for example and assumes that all of my work-tethered network traffic is pushed through my remote desktop).

This type of distributed routing architecture changes, however, when users move to a fully remote and distributed VDI implementation where we don't own our desktop networking.  My "local" desktop is now hosted in a data center along side 100s of other "local" desktops as well as non-desktop applications.  The only work traffic that comes and goes in my home office is desktop/GUI transport and encapsulation (insert your choice of RDP, ICA, or some future transport protocol here) funneling everything back to the corp data center.  All my user-spawned work-related traffic - the stuff I normally spawn from my local desktop, such as browsing my Google Reader feeds - is now created in a remote data center and sourced out of that data center. The application network request doesn’t come from my home office. This is good (in theory) for my local broadband connection because I'm not using it for anything beyond my VDI client.  This is bad, though, for the network connection in and out of the data center that's hosting my desktop. Now all my user traffic is sharing ports, switches, and pipes with everyone else’s desktop traffic. And bad for my user experience because my source traffic is now competing with everyone else (just like I was in the office, which is one of the reasons I benefit from working remotely today) and then re-packaged and tunneled down to me, adding processing and delivery time to my experience. I get hit on waiting for my requests to leave and return to the data center and to have the response remotely displayed and packed for delivery over VDI. 

The best example of this is Web 2.0 traffic, sites like Google Maps.  A quick map request for Alexandria, Egypt consumes ~1Meg of HTTP traffic from desktop request to desktop response. Today for me, that's 1Meg of traffic on the segmented non-VPN’d portion of my broadband connection and it's easy- takes ~5 seconds to pull up Maps, make the request, and get back a sat picture of Alexandria. . My employer doesn't have to manage or deal with that traffic. But when I'm using a remote desktop, that's 1Meg of traffic round-trip from my VDI image in a data center, going out the link shared with the rest of my co-worker's VDI images, and most likely some other non-VDI traffic, and half of that (the response) is packaged for delivery over RDP/ICA in a GUI. I'm not generating that request from my home connection - I'm generating it on and over work's network resources from my VDI image.

So at what point does the benefit from having managed desktops in the data center (and believe me, I think there are huge benefits to this model from the management and security sides) outweigh the cost and limitations of moving all of this user-based traffic into the data center?  What impact does all this VDI client traffic impose on the data center network? There are network-level solutions for traffic management and optimization, such as BIG-IP LTM, optimizing RDP/ICA to the user on one side and optimizing HTTP on the other. But what about outbound from the data center, and more importantly for the best user experience and security issues, correlating the client-side traffic with the data center-side user traffic?

I’m really looking forward to what VDI looks like a few years in the future, when managing remote desktops is not hampered by discreetly managing user traffic in the data center.  We’ll get there, as long as we keep thinking about how all these new virtual technologies impact the network. Think about the infrastructure first and we’ll reap the benefits. 

Lori and are on the sharing wavelengths today (well, except for ideas on remote vs. local software testing, but we can’t always agree on everything, right?  ;) ).  She hit the app side, so here’s a take on the virtual platform side.

I just read Andreas’ post about “virtual routing” and moving VMs to match and optimize traffic flows (yes, it’s a few days old, but catching up on email and blogs takes time after vacation) and I’ll have to say my head started spinning – and not the good kind of spinning. The basic suggestion is that because we can move VMs around as needed we should move them constantly to match traffic patterns and destinations on the network. As traffic patterns and routes change, we move VMs to follow the routes Really? You want to, in essence, move data centers to follow traffic patterns over the same network that’s supporting said traffic, constantly? 

Let's look at this from another perspective: imagine you own a local chain of gas stations. These gas stations provide services for users (gas, food, coffee, etc) as they travel along various highways. They come to you when they need something, said service, otherwise they drive right past you. Your gas stations are located along major interstates at various exits as well as smaller highways and byways throughout the state. Your business accelerates during high-peak travel times (rush hour, weekends, vacation periods) and drops off during slow times (2:00 AM, Christmas, etc). In this slower economy, you're looking for ways to maximize your revenue and you've noticed an interesting trend: drivers are staying closer to home on the weekends and shopping local, avoiding long travel trips in order to save money. You typically don't have gas stations on local 2-lane roads and you can't afford to open more stations, so you're faced with two options:

1. Move your gas stations from the highways to local 2-lane roads: Sure, it's an option, but you would be sacrificing valuable peek highway revenue.
2. Drive traffic from the towns to the highway and to your gas stations: Using signs and promotions such as "Free Coffee with Fill-Up!" - "2 Hot Dog Burritos for $1 with Large Coke!" - "5¢ Gas Discount for Local Residents!" - you can somewhat control how people find your gas station. You can control traffic flow to your service station.

What Andreas (along with Doug) is suggesting is a morphing of #1: Moving the gas station 2 times/day to accommodate traffic patterns and ignoring how and why traffic is driven to your station. Gas station is on exit #421 during morning and evening rush hour and then is moved to the intersection of Maple & Elm during off-peak hours. There’s no care or concern about the traffic patterns, the station is just following the flow. 

This doesn’t resonate with me on many levels. First off, the basic principle of moving a server to match the traffic rather than managing traffic to the server. Traffic is smaller and easier to manage than servers, even with virtual servers. Managing constant VM movement with something like vCenter, clusters, resource pools, etc, is a surmountable task. Second, the process of moving the VMs themselves will have a cost wrt management, downtime, bandwidth (in the event Storage vMotion is used to follow the traffic). As an example, a quote from the post:

“VMotioning nodes (servers) to optimize the flow of traffic on the network”

Hmmm…How will moving a server optimize the traffic flow to that server? Again back to our gas station example: moving the gas station won’t change the traffic patterns at all. And are we talking LAN or WAN? If we’re talking LAN, then there should be traffic management devices, like BIG-IP LTM, in place already to manage optimized access to the apps running on the VMs. If you’re talking WAN, moving VMs cross-WAN on the fly has its own set of issues with traffic and storage.

Your goal when pulling into a gas station is to request a service: gas, coffee, food.  Moving the gas station just addresses cars driving into the parking lot. As always, we should be focused on the service being delivered in addition to the service station.

Don’t get me wrong: the mobility is one of the coolest benefits of VMs, and tools like vMotion and Storage vMotion are definitely changing the way we think about how and where we locate servers. But moving the servers instead of managing the traffic just because we can ain’t the way to go, IMO. Build the platforms for the services, build the infrastructure for the services, and then manage the traffic and access to the services. We have ADCs, let’s use them and manage the traffic and not micro-manage the service stations. And this quote:

“…solve for least switch hops per flow”

I think there’s a whole ‘nother blog post on how this idea becomes moot with something like DVS across multiple physical platforms. But I’ll take a break for now. :)

About two years ago, I was part of the team at F5 that helped design and build our Technology Center – a fully functioning showcase lab housing  all of our products as well as other infrastructure and application partner technologies such as VMware, Microsoft, Dell, etc.  My role (besides cable monkey during install week ;) was to define and isolate the security risks associated with guest virtual machines and segmentation (both physical and logical) of the virtual network and working spaces.  Probably one of the coolest and most fun projects I’ve ever been involved in; the Tech Center really is a work of Data Center art. 

One of the areas I was most concerned about from the get-go was the portability and availability of guest VMs; customers and partners can bring in their own virtual machines to provide back-end services during their testing time at the TC and we’ll run those on our virtual platforms. Now while all of our partners are trustworthy people, we paranoid types know that trust in the security realm is a tough thing to come by, so we needed to create a fully-functional yet safe environment for these VMs to run -- safe for them so there’s minimal sharing of resources with other customers and partners that may be visiting, and safe for us since it’s our infrastructure. From day one I’ve been concerned about the check-in/check-out possibilities with virtual machines; it equates to building a secure box that’s ready to be racked, and just before you rack it you let a stranger take it home for a week, then that stranger brings it back, you rack, cable, boot, and let it run loose.  Scary, huh. 

We solved that problem in the Tech Center using extremely tight networking configurations (walled gardens are your friends) via BIG-IP LTM and hardware isolation for our partners and their specific virtual machine environments, with good management on top to make sure everything functions as expected.  To date it’s been an excellent solution.

Fast forward two years later and meet VMware View: VMware’s VDI management solution that, in theory, should lead us down a path of near complete mobile desktop computing which, as a side benefit, should drastically improve desktop security.  With one huge exception: Offline Desktops. Still an experimental feature today, Offline Desktops allow an end-user to check out their VDI image (which is a complete VM: OS, apps, everything) and take it with them. Let’s say Alice (or maybe Mallory is a better example? ;) is in the office on Tuesday morning running her desktop remotely over the LAN, she checks out her desktop and hops on a plane, works on the flight, and then jumps on the VPN when she’s in her hotel and check her desktop back in, changes and all. Rinse. Repeat. Cool idea with huge benefits, but…

The paranoid me writing this blog immediately started locking doors, lowering blinds, checking the lamps for bugs, etc. I see this as a huge security risk: once the desktop is out of the DC and out of IT’s control, it’s immediately suspect. Just like one of the primary use cases for NAC where external, unknown laptops aren’t allowed to jack directly into the corporate network without auth and some level of validation and sanity check, any VDI image checked out via View and then checked back in should be treated as a new, external device, requiring quarantine, inspection, sanitization, re-authentication, the whole ball of wax. The risks are even more severe for VMs that are allowed to leave and re-enter the data center due to shared resources that run those VMs. The risk is no longer limited to a segmented network; it now extends to the entire VDC platform.

Could this simple, highly usable and beneficial feature open the door for VDI Bombs? A Frankensteinian marriage of trojan bombs (plant today, blow up on delivery or at a later time) delivered via VMs and targeted towards the host hypervisor, network, or CPU? Are IT departments going to build sophisticated quarantine environments for VDI VMs that are checked back in, and if so, will those tools be available soon enough to catch these bombs before time runs out? Or maybe this is a perfect opportunity to see VMSafe used as intended. Seems like the check-in/sync operation would be the perfect time to scan the guest kernel, filesystem, VM tools, etc, via VMsafe. 

I think a completely mobile virtual desktop is a great idea, even if it just pacifies us until true application virtualization becomes a reality, especially for IT client management.  But how much more management and security will be needed in the data center to make this a functional reality? Too much to justify the benefit? We’ll see. And while this concern has been around for a while, even when we were planning for somewhat mobile server VMs, VMware is the first company to make this easily accessible to end-users.  This scenario is applicable to any virtual machine architecture where VMs can come and go in and out of the data center. This issue is not restricted to just VMware.

A lot to think about as VMs come full circle from end-user desktops with Workstation through the Data Center with ESX and now back to end-users with View, and hopefully a good chunk of that thinking involves security planning for portable VDI images. I already have ideas for bumper stickers: “Stop VDI Bombs!”

Yesterday I blogged about how the lack of planning and scalable provisioning of any kind can cause major problems when traffic spikes hit. Today let’s look at how companies who are planning for spikes are preparing, even if it’s not completely elastic.

Rich Miller had an excellent post-election blog post a few weeks ago on sites scaling up for election traffic on Data Center Knowledge. As he points out in a later post, traffic hit record levels through Akamai's CDN on election night. Some companies adequately planned for the burst, other didn't. Spike management isn't something new, however we do deal with massively larger amounts of traffic than we have in the past, and our traffic usage is different. An election that everyone is watching is an excellent case study for these new traffic patterns. Me, I was sitting in front of the TV on election night with my laptop open to MSNBC and twitter, CNN mobile on the iPhone (primarily b/c I enjoyed seeing all those 404 and 500 errors that were showing up on CNN mobile; I know, I'm evil :) ). And I'm guessing this was the norm for people who use the Internet as their primary news source, like me. And the company responses that Rich covers, that did plan for the election spike and anticipated this flood of traffic, are interesting to me on two fronts:

1. The lack of the V-word: Surprisingly in this day and age, none of the companies interviewed said they were relying on any virtualization solutions to scale for their traffic. All the remedies involved physical servers and physical space in a data center or with a hosting company. But with all the hype (and b/c it's all I think about all day), I expected to see something about VMs or virtual storage as part of their spike management plans. On one hand this is encouraging that yes, the world can still spin without VMware or Microsoft virtual platforms. On other other, though, the election should serve as a perfect use case for provisioning and scaling using tools like virtual machines. This election is the best example I can imagine for "elastic computing," and I'm surprised that it wasn't first in responses from these companies. The ability to provision up and de-provision down as need based on real-time, immediate traffic needs is the long term bread-and-butter for virtual platforms; companies like BlueLock and Joyent know this today and have built virtual hosting solutions around provisioning scale for both infrastructure and the applications. So why not use the virtual tools available today as part of your scaling and provisioning needs, rather than having to plan for a spike by pre-ordering batches of servers and waiting weeks for them to go online?
2. Focus on the Apps: I have to say it warms my heart anytime someone mentions applications in the data center -- I'm a softie for those darned apps! :) All of the examples in his post were customers who were expecting an increased need for their application: a political blog, a CDN that hosts political websites, Twitter, etc. Their concern isn't with scaling core infrastructure (switches, routers, cables, trunks, etc), it's with scaling the application platforms (servers, OS', webservers, etc); again, props to those hosting providers who have already built out virtual infrastructure platforms to allow VM and application scale and provisioning as needed.

The phrase "old school" kept popping up in my head as I was reading the post. Are these companies sticking with what works, what's tried and true, by provisioning physical servers well in advance of the expected spike? Or does this show that virtual platforms are still in their infancy and companies that know how to plan for and manage massive amounts of application data traffic don't yet trust virtual solutions? I would probably lean somewhere in the middle, and until virtual platforms and dynamic provisioning proves itself, we'll continue to see dynamic provisioning in the VDC as more of a test case rather than a real-world use case.

For those who follow the music business, or for anyone that just reads and/or watches the news, you’ll know that 80’s metal icons Guns n’ Roses finally released the much anticipated “Chinese Democracy” album on Sunday, Nov. 23rd, 2008.  I say finally because we’ve been waiting on it for years.  Every year just before Christmas rumors began to fly about a possible release date that year.  And this year was no different than other years; those of us that were actually looking forward to this release hedged our bets once again and said “Sure, I’ll believe it when I see it,” including Dr. Pepper.  Who even knew the Dr. was a Guns fan? 

See way back in March of ‘08, Dr. Pepper ran a campaign – if Chinese Democracy came out this year, anyone who wanted one would get a free 20oz bottle of Dr. Pepper.  I’m assuming Axl took that as a throwdown challenge becuase he delivered, leaving the Dr. to start giving out free bottles.  The Dr. came through on their end of the bargain and opened up their website for the free 20oz requests for 24 hours, only valid on the day Chinese Democracy hit shelves, Nov. 23rd.  The official word came down about 2 months ago that indeed, Democracy would actually drop as expected (with distribution and promo details to back it up), leaving the great Dr. ~2 months to plan for the onslaught of freebie requests.  If we know anything about humanity it’s that no one can say no to “Free.” 

I’m not sure what Dr. Pepper was expecting with their offer, but it certainly didn’t plan for the flood of internet traffic that would be directed their way beginning at 12:01 AM EST on the 23rd.  Within hours their site became unresponsive, going through phases of availability: first, the intro Flash got stuck with no way to skip past it; then they finally caught on and started redirecting all users to a text-only portion of the site where the “Free Dr. Pepper” link was buried way down at the bottom which directed to a page that wouldn’t load; and finally, the coup de grace, the entire site went down and was returning nothing more than a 503 Service Unavailable error.  As of this writing, we’re still at 503, and I’m still sans a Dr.

This very temporary unplanned (well, they had two months to plan, but let’s stick with un- for a second) outage is a textbook case for elastic provisioning: spinning up services infinitely to address temporary need and then spinning those services back down when traffic returns to normal levels.  It’s like cable or satellite Video on Demand requests spiking during a snow storm on a Friday night.  Services that normally handle X amount of traffic are suddenly forced to handle exponentially larger volumes of traffic, typically unexpectedly, but they still need to deliver that service.

To give them the benefit of the doubt, maybe Dr. Pepper didn’t anticipate the massive traffic volumes brought on by every major news source in the country picking up on the Gn’R bet, but that’s the point of elastic provisioning: accommodating service need without anticipating fixed levels. All they needed was the ability to scale for 24 hours, then things could have been returned to normal. After the 24-hours, a simple splash page could have been displayed stating that the offer was over and directing their real website traffic to the original site via a click-through. Caching that notice page with a BIG-IP LTM could have kept probably 99% of the traffic off the real site until the hype died down by delivering that notice directly from LTM, and throttling the non-legitimate site requests, rather than putting the load back on the webservers themselves.

So what went wrong? How much is this free offer (which has now been extended 18 hours, although it doesn’t matter if the site is still down) going to ultimately cost them in marketing, name, and not so good press? I’m guessing less than it would have cost to spin up a temporary site with the free offer that could scale for 24 hours.

And this one hits home: I rarely drink soda today but do have fond memories of the Pepper; I was looking forward to throwing on some 80’s hair metal and kicking back like I was in 7th grade again. But the Dr. teased me; they offered and didn’t deliver.  At least Axl finally did, but I’m not waiting 17 years for a free Dr. Pepper; I’ll stick to coffee.  :)

I like this piece because I introduce three concrete steps that IT departments can take today to help guard against security attacks tomorrow.  These aren’t necessarily revolutionary ideas but they are tangible, tactical steps that can be performed today during the planning, architecture, and roll-out phases of virtual platform installations and migrations.  From the article:

“In general, IT departments should focus on three virtualization areas as part of their entire virtualization security architecture:

   1. Segmentation of VMs by location
   2. Segmentation of VMs by service type
   3. Proactive security management throughout the VM lifecycle

These three areas will help IT departments protect their virtual infrastructure against current threats as well as help mitigate the threat of future attacks.”

I’m all about baby steps and thinking about management first - I don’t talk about specific threat vectors in this piece intentionally.  Right now I’m very much in the design and planning stages of virtsec for IT departments.  Remember the 4Ds: Define, Design, Develop, Deploy? This piece is all about the first two: know your risks and design an architecture that be used to manage those risks.  Sure, if you’re keeping score, the 3 solutions above are actually in the Deploy category but I want to emphasize the planning portions of those solutions. 

Start by planning today for whatever the virtsec world will throw at you tomorrow.

-Alan

Technorati Tags: data center,management,security,virtual data center,virtual platform,virtual security,virtualization,virtualization security