F5 Friday: Eavesdropping on Availability

One of the core benefits of cloud computing and application delivery (and primary purposes of load balancing) is availability. In the simplest of terms, achieving availability is accomplished by putting two or more servers (virtual or iron) behind a load balancing device. If one of the servers fails, the Load balancer directs users to the remaining server, ensuring the application being served from that server remains available.

The question then is this: how does the load balancer know when an application is not available? The answer is: health monitoring.

Every load balancer (and clustering solution) can do this at some level. It may be just an ICMP ping or a TCP three-way handshake or determining whether the HTTP and application response received are correct. It may be a combination of a variety of health monitoring options. Regardless of what the health check is doing, it’s getting done and an individual server may be taken out of rotation in the event that its health check response indicates a problem.

Now, interestingly enough there is more than one way to perform a health check. As you might have guessed the first way is to communicate out-of-band with the server and/or application. Every <user configured> time interval, the load balancer performs a check and then acts or doesn’t act upon the response. The advantage of this is that the load balancer can respond very quickly to problems provided the time interval is of sufficiently granular value. The disadvantage of this approach is that it takes up resources on the load balancer, the network, and the server. In a service-provider or cloud computing environment, the resources consumed by out-of-band health checks can be devastating to network performance and may well impact capacity of the server.

What else is there? 

INBAND and PASSIVE MONITORING

While inband monitoring is relatively new, passive monitoring was pioneered by F5 many years ago. In fact, leveraging passive monitoring and inband monitoring together provides the means to more quickly address problems as they occur.

Inband monitoring was introduced in BIG-IP v10. Inband monitors can be used with either a Standard or a Performance (Layer 4) type virtual server, and as a bonus can also be used with active monitors. What inband monitoring does is basically eavesdrop on the conversation between a client and the server to determine availability. The monitor, upon an attempt by a client to connect to a pool member, behaves as follows:

• If the pool member does not respond to a connection request after a user-specified number of tries within a user-specified time period, the monitor marks the pool member as down.
• After the monitor has marked the pool member as down , and after a user-specified amount of time has passed, the monitor tries again to connect to the pool member (if so configured).

Passive monitoring is actually a technique that leverages network-side scripting (in our case F5 iRules) to inspect the content returned by an application and determine whether it is valid or not. If it is not valid, iRules affords the ability to mark the node down and/or resend the request to another (hopefully correctly working) application instance.  Here’s a brief example that can mark the server down after three successive failures, otherwise attempts to “retry” the request:

   1: rule count_server_down {

   2:    when HTTP_REQUEST {

   3:    if { not [info exists orig_request]} {

   4:       set orig_request [HTTP::request]

   5:    }

   6: }

   7: when HTTP_RESPONSE {

   8:    if { [HTTP::status] >= 500 } {

   9:       set failures [session lookup dest_addr [LB::server addr]]

  10:       if { $failures >= 3 } {

  11:          LB::down

  12:       } else {

  13:          session add dest_addr [LB::server addr] [incr failures]

  14:          LB::detach

  15:          HTTP::retry $orig_request

  16:       }

  17:    }

  18: }

  19: } 

Passive monitoring is real-time, it’s looking at real requests to determine actual availability and correctness of response. This is even more useful when you start considering how you might respond. The robust nature of iRules allows you to do some interesting manipulation of content and communication channel, so if you can think it up you can probably get it done with an iRule.

By combining inband with passive monitoring you end up with “inband passive monitoring”. This solution eliminates the overhead of active monitoring by eavesdropping on client-server conversations and ensures application availability by inspecting content.

For a great discussion of inband passive monitoring and a detailed scenario of how it might work in conjunction with a real application, check out Alan Murphy’s post on the subject, “BIG-IP v10: Passive Application Monitoring”.

WILS: Load Balancing and Ephemeral Port Exhaustion

Understanding the relationship between SNAT and connection limitations in full proxy intermediaries.

If you’ve previously delved into the world of SNAT (which is becoming increasingly important in large-scale implementations, such as those in the service provider world) you remember that SNAT essentially provides an IP address from which a full-proxy intermediary can communicate with server-side resources and maintain control over the return routing path.

There is an interesting relationship between intermediaries that leverage two separate TCP stacks (such as full-proxies) and SNAT in terms of concurrent (open) connections that can be supported by any given “virtual” server (or virtual IP address, as they are often referred to in the industry). The number of ephemeral ports that can be used by any client IP address is 65535. Programmer types will recognize that as a natural limitation imposed by the use of an unsigned short integer (16 bits) in many programming languages.

Now, what that means is that for each SNAT address assigned to a virtual IP address, a theoretical total of 65535 connections can be open at any other single address at any given time. This is because in a full-proxy architecture the intermediary is acting as a client and while servers use well-known ports for communication, clients do not. They use ephemeral (temporary) ports, the value of which is communicated to the server in the source port field in the request. Each additional SNAT address available increases the total number of connections by some portion of that space. As you should never use ephemeral ports in the privileged range (port numbers under 1024 are traditionally reserved for firewall and other sanity checkers - see /etc/services on any Unix box) that number can be as many as 64512 available ports between the SNAT address and any other IP address. For example, if a server pool (virtual or iron) has 24 members and assuming the SNAT address is configured to use ephemeral ports in the range of 1024-65535, then a single SNAT address results in a total of 24 x 48k = 1,152k concurrent connections to the pool. If the SNAT is assigned to a virtual server that is targeting a single address (like another virtual server or another intermediate device) then the total connections is 1 x 48k = 48k connections.

Obviously this has a rather profound impact on scalability and capacity planning. If you only have one SNAT address available and you need the capabilities of a full-proxy (such as payload inspection inbound and out) you can only support a limited number of connections (and by extension, users). Some solutions provide the means by which these limitations can be mitigated, such as the ability to configure a SNAT pool (a set of dedicated IP addresses) from which SNAT addresses can be automatically pulled and used to automatically increase the number of available ephemeral ports.

Running out of ephemeral ports is known as “ephemeral port exhaustion” as you have exhausted the ports available from which a connection to the server resource can be made. In practice the number of ephemeral ports available for any given IP address can be limited by operating system implementations and is always much lower than the 65535 available per IP address. For example, the IANA official suggestion is that ephemeral ports use 49152 through 65535, which means a limitation of 16383 open connections per address. Any full-proxy intermediary that has adopted this suggestion would necessarily require more SNAT addresses to scale an application to more concurrent connections.

One of the advantages of a solution implementing a custom TCP/IP stack, then, is that they can ignore the suggestion on ephemeral port assignment typically imposed at the operating system or underlying software layer and increase the range to the full 65535 if desired. Another major advantage is making aggressive use of TIME-WAIT recycling. Normal TCP stacks hold on to the ephemeral port for seconds to minutes after a connection closes. This leads to odd bursting behavior. With proper use of TCP timestamps you can recycle that ephemeral port almost immediately.

Regardless, it is an important relationship to remember, especially if it appears that the Load balancer (intermediary) is suddenly the bottleneck when demand increases. It may be that you don’t have enough IP addresses and thus ports available to handle the load.

WILS: Write It Like Seth. Seth Godin always gets his point across with brevity and wit. WILS is an ATTEMPT TO BE concise about application delivery TOPICS AND just get straight to the point. NO DILLY DALLYING AROUND.

The World Doesn’t Care About APIs

The Battle of Economy of Scale versus Control and Flexibility

When strategies are formed it quickly becomes obvious that cloud computing is more about balance than anything else.

At a time when you’d think cloud computing would be the primary “go to” strategy for managing scale and rapid growth multiple well-known and demanding organizations are building their own data centers instead.

With all the hype around cloud being faster, cheaper, and more efficient these folks must be crazy, right?

Not at all. In fact, these moves illustrate the growing friction between the economy of scale offered by cloud computing and the control and flexibility that is part and parcel of owning one’s own data center.

In April Twitter announced plans to build a data center of its own. On Wednesday it provided additional details on the Twitter Engineering blog.

“Later this year, Twitter is moving our technical operations infrastructure into a new, custom-built data center in the Salt Lake City area,” wrote Twitter’s Jean-Paul Cozzatti, who said having dedicated data centers will provide more capacity to accommodate growth of 300,000 new users per day. “Keeping pace with these users and their Twitter activity presents some unique and complex engineering challenges. Importantly, having our own data center will give us the flexibility to more quickly make adjustments as our infrastructure needs change.”

-- Data Center Knowledge, “Twitter Picks Utah for New Data Center”

Twitter isn’t the only Web 2.0 savvy organization moving to their own data center. Facebook earlier this year announced it, too, was also investing in building out its own data center.

BUT CLOUD AUTO-SCALES and STUFF!

It’s not all about scalability. I know that sounds nearly heretical, but it’s not. And it’s not a new mantra, either. Scalability is certainly a factor in why one would choose cloud computing over a localized deployment, but also important are control and flexibility. 

Another consideration is the ability to customize your data center infrastructure to provide more granular control of operations. “That control gives us a ton of flexibility, and we can build new things without having to wait for our partner,” said Heiliger [Jonathan Heiliger, Facebook’s VP of Technical Operations]

-- Data Center Knowledge, “Data Centers: For When The Cloud is Not Enough”

If I’ve said it once I’ve said it a thousand times: control is a huge factor in the decision making process and something that isn’t effectively offered by today’s public cloud computing offerings. Remember the Information week analytics Cloud computing survey in 2009?

Even though security remains concern number one, control and configurability are on the top of the list, as well. The issue of control has almost always gone hand in hand with cloud adoption inhibitors, but it always takes a back seat to the more glamorous and scary “security” issue. These are not minor stumbling blocks in many cases, and the inability to rapidly adapt an infrastructure to meet growth and scale and make architectural changes, if necessary, are paramount to success. If cloud computing cannot provide the agility necessary to meet these challenges then it is logical to assume that organizations will either (a) stay in the local data center or (b) move to a local data center from the cloud when it becomes obvious the environment is inhibiting forward momentum.

Current adoption patterns indicate that this is not an anomaly, but will instead likely become the norm for organizations. Applications that are initially deployed “in the cloud” will, upon becoming a critical business application or growing beyond the meager means of control and flexibility offered by the cloud, will migrate to the data center, where control and agility are provided by the simple fact that the organization can change at will any piece of the infrastructure – from its physical implementation to its logical organization – at will. This is evident in the percentage of organizations using cloud for “dev and test” but not for production. Clearly the economy of scale and rapidity of deployment makes the cloud a perfect environment for development and testing but not necessarily production.

ECONOMY of SCALE MAY be INHIBITING SCALE

The irony is that the economy of scale offered by cloud may well be biting cloud in the proverbial derrière as it becomes the inhibitor to effective scale by limiting or making extremely difficult the architectural changes necessary for an application to scale in a cloud environment.

At some point scalability can become not about the application but about its infrastructure and the way in which that infrastructure interacts. It can become about the network and its components and how applications end up interacting with and through that infrastructure. In a cloud computing environment it is rarely the case that a customer can impact that infrastructure and, when it can, it is then limited by other factors such as underlying virtualization technology and the physical server infrastructure on which the application is ultimately deployed. If the answer to a scalability obstacle is more bandwidth and higher throughput, you can’t really add another NIC to a server in the cloud. That’s not your call. But it is if you’re in the data center, and it is virtualization – not cloud - that ultimately provides the agility to make such a change and rapidly propagate that change across the application deployment.

It isn’t always about costs. Well, okay, it is about cost but in IT it’s about cost as it relates to performance, or flexibility, or other operational functionality required to successfully meet data center and business goals. When spending less on infrastructure results in higher operational costs, the organization really hasn’t saved money at all. Savvy CIO and CTOs understand that it’s not a battle, but a balancing act. It’s not about achieving the highest economy of scale, but the best economy of scale given the specific operational and business needs.

F5 Friday: Beyond the VPN to VAN

Web 2.0 and cloud computing have naturally pushed all things toward application-centric views, why not the VPN?

When SSL VPNs were first introduced they were a welcome alternative to the traditional IPSEC VPN because they reduced the complexity involved with providing robust, secure remote access to corporate resources for externally located employees.

Early on SSL VPNs were fairly simple – allowing access to just about everything on the corporate network to authenticated users. It soon became apparent this was not acceptable for several reasons, most prominently standing out the risk of infection by remote employees who might have been using personal technology to work from home. While most organizations have no issue with any employee working a few extra hours at home, those few extra hours of productivity can be easily offset by the need to clean up after a virus or bot entering the corporate network from an unsecured, non-validated remote source. This was especially true as one of the selling points for SSL VPN was (and still is) that it could be used from any endpoint. The “clientless” nature of SSL VPN made it possible to use a public kiosk to log-in to corporate resources via an SSL VPN without fear that the ability to do so would be “left behind.” I’m not really all that sure this option was ever widely used, but it was an option.

Then SSL VPNs got more intelligent. They were able to provide endpoint security and policies such that an “endpoint”, whether employee or corporate owned, had to meet certain criteria – including being “clean” – before it was allowed access to any corporate resource. This went hand in hand with the implementation of graded authentication, which determined access rights and authorization levels based on context: location, device, method of access, etc… That’s where we sat for a number of years. There were updates and upgrades and additions to functionality but nothing major about the solution changed.

Until recently. See, the advent of cloud computing and the increasing number of folks who would  like to “work from home” if not as a matter of course then as a benefit occasionally has been driving all manner of solutions toward a more application-centric approach and a more normalized view of access to those applications. As more and more applications have become “webified” it’s made less sense over time to focus on securing remote access to the corporate network and more sense to focus on access to corporate applications – wherever they might be deployed.

THE NEXT GENERATION of ACCESS CONTROL

That change in focus has led to what should be the next step in the evolution of remote access – from SSL VPN to secure access management, to managing application access by policy across all users regardless of where they might be located.

Similarly, it shouldn’t matter whether corporate applications are “in the cloud” or “in the data center”. A consistent method of managing access to applications across all deployment locations and all users reduces the complexity inherent in managing both sides of the equation.

We might even call this a Virtual Application Network (VAN) instead of a Virtual Private Network (VPN) because what I’m suggesting is that we create a “network” of applications that is secured by a combination of transport layer security (SSL) and controlled by context-based access management at the application layer. Whether a user is on the corporate LAN or dialed-in from some remote location that has yet to see deployment of broadband access shouldn’t matter. The pre-access validation that the accessing system is “clean” is just as important today when the system is local as if it were remote; viruses and bots and malware don’t make the distinction between them, why should you?

By centralizing application access across users and locations, such secure access methodologies can be used to extend control over applications that may be deployed in a cloud computing environment as well. Part of F5’s position on cloud computing is that many of the solutions that will be required to make cloud-deployed applications viable is that the control that exists today over locally deployed applications must be extended somehow to those remote applications as a means to normalize management and security as well as controlling the costs of leveraging what is supposed to be a reduced cost environment.

That’s part of the promise of F5’s BIG-IP Edge Gateway. It’s the next step in secure remote access that combines years of SSL VPN (FirePass) experience with our inherent application-aware delivery infrastructure. It provides the means by which access to corporate applications can be normalized across users and application environments without compromising on security and control. And it’s context-aware because it’s integrated into F5’s core enabling technology platform, TMOS, upon which almost all other application delivery functionality is based and deployed.

I highly encourage a quick read of George Watkin’s latest blog on the topic, Securing the Corporate Intranet with Access Policy Manager, in which he details the solution and some good reasons behind why you’d want to do such a thing (in case I’m not convincing enough for you). You may also enjoy a dive into a solution presented in a previous F5 Friday, “F5 Friday: Never Outsource Control”, that describes an architectural approach to extending normalized control of application access to the cloud.

Get Out Your Crayons. We Need to Color Us Some Bits

Those eight bits in the IP header aren’t doing much of anything these days, perhaps it’s time to put them to work

Back in the early days of bandwidth management, when quality of service and prioritization of traffic were on everyone’s minds because we were stuck with low throughput connectivity, there was a brief discussion about the use of IP’s TOS (Type of Service) bits as a means to meet specific application performance needs. I say brief because, well, it never really got anywhere. See, even though the creators of the IP specification had looked into the future and provided a technical solution to prioritization of traffic they couldn’t have looked into the future and seen the organizational roadblocks to leveraging such a simple but effective method of managing traffic.

The biggest problem was that you could ensure that TOS bits were honored within the organization, but once those packets passed through the organization’s boundary of control, i.e. onto the Internet, there was no guarantee or requirement that any other organization honor those bits. And because packets flow through many, many different routers and switches along its very long yet ironically very short travels from datacenter to client, if just one fails to honor the bits then the packets go gray and prioritization is lost.

Hence it was that bandwidth management moved up the stack, with queuing and rate shaping at the transport and application layers of the stack becoming the norm and the “coloring” of TOS bits fell into disuse, like childhood crayons set aside in favor of cool gel pens and clicky-mechanical pencils.

THE NEED STILL EXISTS

Interestingly enough, the need for prioritization and bandwidth management still exists and, in many cases, it could become of paramount importance to the successful implementation of a mature cloud.

It is easy to forget that when you look under the covers of a cloud computing environment that there still exist physical network connections that comprise that environment. Despite the magnitude of virtualization in use at all layers that abstracts the entire infrastructure from its physical implementations, that network is still there. It’s massive, it’s huge, and it’s a spider’s web of connectivity.

I am not the only one thinking about this, I’m fairly certain. An InformationWeek Analytics survey last year included a concern rarely seen in cloud computing surveys: “speed to activate new services/expand capacity.” And folks are apparently somewhat concerned about this. Less so than other “problem” areas of cloud but enough concerned that it made the chart.

The problem is that because everyone is still actually sharing a physical network connection all that traffic gets mixed up on the wire. At the physical layer, every packet is the same regardless of what payload it’s carrying. But the reality is that some packets are more “important” in the sense that some will be extremely time sensitive. A request to provision a service that’s under increasing load is more important than many application requests but both look the same to the routers and switches that get those packets from point A to point B within a cloud computing environment.

As cloud computing providers and enterprises get better at automating the provisioning and elastic scalability processes, the packets that make up requests for such operational tasks will become increasingly time sensitive and important to ensuring operational efficiency and success.

The problem, as once reared its ugly head before, is that all that traffic – customer and operational – is running over HTTP. REST, SOAP, whatever. The APIs that make it possible to provide “compute as a service” are almost unilaterally implemented using a combination of HTTP and REST or SOAP-based architectural principles. And so are the applications running in the cloud. Everything is running over HTTP and thus, as with bandwidth management challenges in the past, it becomes increasingly difficult to distinguish traffic without inspecting its payload.

And payload inspection always adds latency. It may be only microseconds but it’s still latency. And if every router has to do it, well, microseconds eventually add up to seconds. And when we’re talking about ensuring the order of operations in a provisioning process, those seconds can make a big difference.

BACK to BEGINNING

Sometimes the solution really is to go back to the beginning, to our roots, to the network.

It could be that the solution lies in out-of-band management networks. If not physically at least logically separated network channels that are by default prioritized and therefore will never get stuck in a traffic jam trying to get to location “C” because the application at location “A” is physically on the same network and heavily oversubscribed at the moment an important management request is trying to get to “C”. This would, however, add another layer of complexity to the management of not just the physical but the logical network. Complexity that is almost always translated into higher costs, which of course gets passed on to the customer.

It could be that DCSP (DiffServ or Differentiated Services Code Point (DSCP)) – what eventually became of TOS because it was basically unused - could become the solution precisely because a single provider “owns” the entire network. Because a provider has control over all the components comprising the underlying network infrastructure it could enforce the honoring of DSCP bits across its network and thus prioritize traffic at the IP layer.

DiffServ is concerned with classifying packets as they enter the local network. This classification then applies to Flow of traffic where a Flow is defined by 5 elements; Source IP address, Destination IP, Source port, Destination port and the transport protocol. A flow that has been classified or marked can then be acted upon by other QoS mechanisms. Multiple flows can therefore be dealt with in a multitude of ways depending on the requirements of each flow. Packets are first Classified according to their current DSCP. Then they are separated into queues where one queue may be routed via a marking mechanism and another queue may be examined more closely.

-- Quality of Service overview

If a provider segments by port “cloud management” traffic from “normal” traffic, this solution would be fairly easy to implement. If not, well, then we’re going to need something else. It’s the concept behind DiffServ and TOS that’s important to leverage in such a solution – the recognition that some traffic must be prioritized to ensure delivery in a timely fashion.

EIGHT BITS

What seems clear, to me at least, is that eventually we’re going to run into scenarios in which we need something akin to Operations Performance Management (OPM) to ensure that management and control messages are delivered in a timely fashion.

This is especially true as cloud computing matures and we start to see the dynamism inherent in infrastructure 2.0 components put into broader use. Real-time enforcement of security and delivery policies must be real-time. They can’t be near-time, they must happen now. When we start relying on existing open standards like HTTP and messaging hubs and event-driven networking architectures we have to remember we get the good and the bad from those existing standards and implementations. We get the ease of use and integration, the flexibility, and the ubiquity of support, but we also get the problems that have plagued quality of service implementations forever: when everything is delivered via HTTP then everything looks like HTTP. Differentiation is nice, but we need to have a way to do that that doesn’t impede performance in a cloud computing environment.

In this case, it seems wise to look down the stack and return to a perhaps less sophisticated but absolutely more elegant and simple means of distinguishing not only traffic but precedence and terms of service. DSCP or TOS or whatever we might decide to slap into those 8 bits in the IP header (perhaps there’s room for a new specification and use of those bits?) would be infinitely more scalable and easily supported in a cloud computing environment for distinguishing between the small subset of internal “traffic types” that need to be managed.

Let’s Face It: PaaS is Just SOA for Platforms Without the Baggage

At some point in the past few years SOA apparently became a four-letter word (as opposed to just a TLA that leaves a bad taste in your mouth) or folks are simply unwilling – or unable – to recognize the parallels between SOA and cloud computing .

This is mildly amusing given the heavy emphasis of services in all things now under the “cloud computing” moniker. Simeon Simeonov was compelled to pen an article for GigaOM on the evolution/migration of cloud computing toward PaaS after an experience playing around with some data from CrunchBase. He came to the conclusion that if only there were REST-based web services (note the use of the term “web services” here for later in the discussion) for both MongoDB and CrunchBase his life would have been a whole lot easier.

For an application developer, as opposed to an infrastructure developer, all these vestiges of decades-old operating system architecture add little value. In fact, they cause deployment and operational headaches—lots of them. If I had taken almost any other approach to the problem using the tools I’m familiar with I would have performed HTTP operations against the REST-based web services interface for CrunchBase and then used HTTP to send the data to MongoDB. My code would have never operated against a file or any other OS-level construct directly.

[…]

Most assume that server virtualization as we know it today is a fundamental enabler of the cloud, but it is only a crutch we need until cloud-based application platforms mature to the point where applications are built and deployed without any reference to current notions of servers and operating systems.

-- Simeon Simeonov  “The next reincarnation of cloud computing”

Now I’m certainly not going to disagree with Simeon on his point that REST-based web services for data sources would make life a whole lot easier for a whole lot of people. I’m not even going to disagree with his assertion that PaaS is where cloud is headed. What needs to be pointed out is what he (and a lot of other people) are describing is essentially SOA minus the standards baggage. You’ve got the notion of abstraction in the maturation of platforms removing the need for developers to reference servers or operating systems (and thus files). You’ve got ubiquity in a standards-based transport protocol (HTTP) through which such services are consumed. You’ve got everything except the standards baggage. You know them, the real four-letter words of SOA: SOAP, WSDL, WSIL and, of course, the stars of the “we hate SOA show”, WS-everything.

But the underlying principles that were the foundation and the vision of SOA – abstraction of interface from implementation, standards-based communication channels, discrete chunks of reusable logic – are all present in Simeon’s description. If they are not spelled out they are certainly implied by his frustration with a required interaction with file system constructs, desiring instead some higher level abstracted interface through which the underlying implementation is obscured from view.

CLOUDS AREN’T CALLED “as a SERVICE” for NOTHING

Whether we’re talking about compute, storage, platform, or infrastructure as a service the operative word is service.

It’s a services-based model, a service-oriented model. It’s a service-oriented architecture that’s merely moved down the stack a bit, into the underlying and foundational technologies upon which applications are built. Instead of building business services we’re talking about building developer services – messaging services, data services, provisioning services. Services, services, and more services.

Move down the stack again and when we talk about devops and automation or cloud and orchestration we’re talking about leveraging services – whether RESTful or SOAPy – to codify operational and datacenter level processes as a means to shift the burden of managing infrastructure from people to technology. Infrastructure services that can be provisioned on-demand, that can be managed on-demand, that can apply policies on-demand. PaaS is no different. It’s about leveraging services instead of libraries or adapters or connectors. It’s about platforms – data, application, messaging – as a service. 

And here’s where I’ll diverge from agreeing with Simeon, because it shouldn’t matter to PaaS how the underlying infrastructure is provisioned or managed, either. I agree that virtualization isn’t necessary to build a highly scalable, elastic and on-demand cloud computing environment. But whether that data services is running on bare-metal, or on a physical server supported by an operating system, or on a virtual server should not be the concern of the platform services. Whether elastic scalability of a RabbitMQ service is enabled via virtualization or not is irrelevant. It is exactly that level of abstraction that makes it possible to innovate at the next layer, for PaaS offerings to focus on platform services and not the underlying infrastructure, for developers to focus on application services and not the underlying platforms. Thus his musings on the migration of IaaS into PaaS are ignoring that for most people, “cloud” is essentially a step pyramid, with each “level” in that pyramid being founded upon a firm underlying layer that exposes itself as services.

SOA IS ALIVE and LIVING UNDER an ASSUMED NAME for ITS OWN PROTECTION

If we return to the early days of SOA you’ll find this is exactly the same prophetic message offered by proponents riding high on the “game changing” technology of that time.

SOA promised agility through abstraction, reuse through a services-oriented approach to composition, and relieving developers of the need to be concerned with how and where a services was implemented so they could focus instead on innovating new solutions. That’s the same thing that all the *aaS are trying to provide – and with many of the same promises. The “cloud” plays into the paradigm by introducing elastic scalability, multi-tenancy, and the notion of self-service for provisioning that brings the financial incentives to the table.

The only thing missing from the “as a service” paradigm is a plethora of standards and the bad taste they left in many a developer’s mouth. And it is that facet of SOA that is likely the impetus for refusing to say the “S” word in close proximity to cloud and *aaS. The conflict, the disagreement, the confusion, the difficulties, the lack of interoperability that nearly destroyed the interoperability designed in the first place – all the negatives associated with SOA come to the fore upon hearing that TLA instead of its underlying concepts and architectural premises. Premises which, if you look around hard enough, you’ll find still very much in use and successfully doing exactly what it promised to do.

Simeon himself does not appear to disagree with the SOA-aaS connection. In a Twitter conversation he said, “I still have scars from the early #SOA days. Shouldn't we start with something simpler for PaaS?”

To which I would now say “but we are”. After all it wasn’t – and isn’t - SOA that was so darn complex, it was its myriad complex and often competing standards. A rose by any other name, and all that. We can refuse to use the acronym, but that doesn’t change the fact that the core principles we’re applying (successfully, I might add) are, in fact, service-oriented.

HTTP Now Serving … Everything

You can’t assume anything about an application’s performance and delivery needs based on the fact that it rides on HTTP.

I read an interesting article during my daily perusal of most of the Internet (I’ve had to cut back because the Internet is growing faster than my ability to consume) on “Virtual Micro Networks.”

The VMN concept goes well beyond Virtual Local Area Networks (VLANs). Like VLANs or any other network, VMNs transport data from source to destination. But VMNs extend beyond transport to consider security, location, users, and applications. VMNs address:

Where is the information? The answer to this question used to be a physical server or storage device but application switching and server/storage virtualization makes this more dynamic and complex.

[…]

VMNs also must be aware of traffic type. For example, voice, video, and storage traffic is extremely latency-sensitive while HTTP traffic is not. Additionally, some network traffic may contain confidential information that should be encrypted or even blocked.

What are the specific characteristics of the information? Network-based applications may be made up of numerous services that come together at the user browser. How they get there isn’t always straightforward, thus the rise of vendors like Citrix NetScaler and F5 Networks. This is also where security comes into play as certain traffic may be especially sensitive, suspicious, or susceptible. [emphasis added]

What’s driving creation of Virtual Micro Networks

Okay, so first things first: the author really is using another term to describe what we’ve been calling for some time an application delivery network. That was cool in and of itself; not the emergence of yet another TLA but that the concept is apparently out there and rising. But what was even more interesting was the conversation this started on Twitter. If you don’t follow @csoandy (that’s the Twitternym of Andy Ellis of Akamai Networks) you might want to start. Andy pointed out that the statement “HTTP traffic is not latency-sensitive” is a bit too broad and went on to point out that it really depends on what you’re delivering. Live video, after all, is sensitive to latency no matter what protocol is transporting it.

Andy put it well in an off-line conversation when he said, “There's also the myth that HTTP isn't for low latency apps. HTTP lets you take advantage of optimizations done in TCP and HTTP to accelerate delivery.” A great point and very true. All the “built-in acceleration and optimization” of an application delivery controller’s TCP stack is free for HTTP because after all, HTTP rides on TCP. But ironically this is also where things get a bit wonky.

The reality is that the application data is sensitive, not the protocol. But because the data HTTP was initially designed to transport was not considered to be latency sensitive, you almost have to look at HTTP as though it is, which is why the broad statement bothered Andy in the first place. We wouldn’t say something like “TCP” or “UDP” is not sensitive to latency because these are transport layer protocols. We need to know about the data that’s being transported. Similarly, we can’t (anymore) say “HTTP isn’t sensitive to latency” because HTTP is the de facto transport protocol of web applications. As I remarked to Andy, the move to deliver everything via HTTP changes things significantly. “Things” being the entire realm of optimization, acceleration, and application delivery.

Context is Everything

As the initial blog post that started this conversation pointed out, and which nothing Andy and I discussed really changed, is that our nearly complete reliance on HTTP as the de facto transport protocol for everything means that the infrastructure really needs to be aware of the context in which requests and responses are handled.

When an HTTP GET and its associated response might in one case be a simple binary image and in another case it might be the initiation of a live video stream,  well, the infrastructure better be able to not only recognize the difference but handle them differently. HTTP doesn’t change regardless, but the delivery needs of the data do change.

This is the “application aware” mantra we (as in the entire application delivery industry) have been chanting for years. And now it’s becoming an imperative because HTTP no longer implies text and images, it implies nothing. The infrastructure responsible for delivering (securing, optimizing, accelerating, load balancing) the access to that application data cannot assume anything; not session length, not content length, not content type, not optimal network conditions. The policies that might ensure a secure, fast, and available web application are almost certainly not the same policies that will provide the same assurance for video, or audio, or even lengthy binary data. The policies that govern the delivery of a user-focused application are not the same ones that should govern the delivery of integration-driven applications like Web 2.0 and cloud computing APIs. These are different data types, different use cases, different needs.

Yet they are all delivered via the same protocol: HTTP.

Dynamic Infrastructure Needed

What makes this all even more complicated (yes, it does get worse as a matter of fact) is that not only is the same protocol used to deliver different types of data but in many cases it may be delivered to the same user in the same session.

A user might move from an article or blog to a video back to text all the while a stream of Twitter and Facebook updates are updating a gadget in that application. And the same infrastructure has to handle all of it. Simultaneously.

Wheeeeeeeeeee!

That HTTP can be extended (and has been, and will continue to be) to include broad advanced capabilities has been a blessing and a curse, for as we deliver more and more differenter content over the same protocol the infrastructure must be able to ascertain dynamically what type of data is being delivered and apply the appropriate actions dynamically. And it has to incorporate user information, as well, because applications highly sensitive to latency need special care and feeding when delivered over a congested, bandwidth constrained network as opposed to delivery via a high-speed, low latency LAN. The application delivery network, from user to application and back, must be context-aware and able to “turn on a dime” as it were, and adjust delivery policies based on conditions at the time of the request and subsequent responses. It’s got to by dynamic. 

Consider the comparison offered by Andy regarding video served via traditional protocols and HTTP:

Consider a live stream; say, Hope for Haiti. A user opens a browser, and has a small embedded video, with a button to expand to full screen. With most streaming protocols, to get a higher resolution stream, your player needs to either:

a) start grabbing a second, high res stream in the background, and guess when to splice them over. (now consider if the stream is too fat, and you need to downgrade)

b) pause (drop existing stream) and grab a new stream, exposing buffering to a user.

c) signal somehow to the streaming server that it should splice in new content (we built this. it's *hard* to get right. And you have to do it differently for each protocol).

With HTTP, instead what you see is:

a) Browser player is grabbing short (usually 2) second chunks of live streaming content. When it detects that it is fuller screen, and, inferring available bandwidth by how long it takes to download a chunk, ask for a higher resolution chunk for the next available piece.

Quite the difference, isn’t it? But underlying that simplicity is the ubiquity of HTTP and a highly dynamic, flexible infrastructure capable of adapting to the sensitivities specific not only to the protocol, to the data, but to the type of data being delivered.

So it turns that both Andy and I are both right, it just depends on how you’re looking at it. It isn’t that HTTP is sensitive to latency, it isn’t. But the data being delivered over HTTP most certainly is. But it is confusing to discuss HTTP in broad, general terms because you can’t assume anymore that what’s being delivered is text and images. We don’t talk in terms of TCP when we talk web applications, so maybe it’s time to stop generalizing about “HTTP” and start focusing on applications and data, about the content, because that’s where the real challenges surrounding performance and security are hiding.

Defeating Attacks Easier Than Detecting Them

Defeating modern attacks – even distributed ones – isn’t the problem. The problem is detecting them in the first place.

Last week researchers claimed they’ve discovered a way to exploit a basic security flaw that’s used in software that’s in high use by Web 2.0 applications to essentially support if not single-sign on then the next best thing – a single source of online identity.

The prevalence of OAuth and OpenID across the Web 2.0 application realm could potentially be impacted (and not in a good way) if the flaw were to be exploited. Apparently a similar flaw was used in the past to successfully exploit Microsoft’s Xbox 360. So the technique is possible and has been proven “in the wild.”

 The attacks are thought to be so difficult because they require very precise measurements. They crack passwords by measuring the time it takes for a computer to respond to a login request. On some login systems, the computer will check password characters one at a time, and kick back a "login failed" message as soon as it spots a bad character in the password. This means a computer returns a completely bad login attempt a tiny bit faster than a login where the first character in the password is correct.

[…]

But Internet developers have long assumed that there are too many other factors -- called network jitter -- that slow down or speed up response times and make it almost impossible to get the kind of precise results, where nanoseconds make a difference, required for a successful timing attack.

Those assumptions are wrong, according to Lawson, founder of the security consultancy Root Labs. He and Nelson tested attacks over the Internet, local-area networks and in cloud computing environments and found they were able to crack passwords in all the environments by using algorithms to weed out the network jitter.

Researchers: Password crack could affect millions

ComputerWorld, July 2010

Actually, after reading the first few paragraphs I’m surprised that this flaw wasn’t exploited a lot sooner than it was. The ability to measure fairly accurately the components that make up web application performance is not something that’s unknown, after all.  So the claim that an algorithm can correctly “weed out” network latency is not at all surprising.

But what if the performance was randomized by, say, an intermediary interjecting additional delays into the response? You can’t accurately account for something that’s randomly added (or not added, as the case may be) and as long as you seeded the random generation with something that cannot be derived from the context of the session there are few algorithms that could figure out what the random generation seed might be. That’s important because random number generation often isn’t and it can often be predicted based on knowing what was used to seed the generator. So we could defeat such an attack by simply injecting random amounts of delay into the response.

Or, because the attack depends on an observable difference in timing, simply normalizing response times for the login process would also defeat this attack. This is the solution pointed out in another article on the discovery, “OAuth and OpenID Vulnerable to Timing Attack”, in which it is reported developers of impacted libraries indicate that just six lines of code will solve this problem by normalizing response times. This, of course, illustrates a separate problem, which is the reliance on external sources to address security risks that millions may be vulnerable to now because while it’s a simple resolution, it may takes days, weeks, or more before it is available.

This particular attack would indeed be disastrous were it to be exploited given the reliance on these libraries by so many popular web sites. And though the solutions are fairly easy to implement, that isn’t the real problem. The real problem is how difficult such attacks are becoming to detect, especially in the face of the risk incurred by remaining vulnerable while solutions are developed and distributed.

DEFEATING is EASY. DETECTING is HARD.

The trick here, and what makes many modern attacks so dangerous is that it’s really really hard to detect them in the first place.

Any attack that could be distributed across multiple clients – clients smart enough to synchronize and communicate with one another – becomes difficult to detect, especially in a load balanced (elastic) environment in which those requests are likely spread across multiple application instances. The variability in where attacks are coming from makes it very difficult to see an attack occurring in real-time because no single stream exhibits the behavior most security-focused infrastructure watches for.

What’s needed to detect such an attack is to be more concerned with what is being targeted rather than by whom or from where. While you want to keep track of that data the trigger for such brute-force attacks is the target, not the client activity. Attackers are getting smart, they know that repeated attempts at X or Y will be detected and that more than likely they will find their client blacklisted for a period of time (if not permanently) so they’ve come up with alternative methods that “hide” and try to appear like normal requests and responses instead. In fact it could be postulated that it is not repeated attempts to login from a single location that are the problem today, but rather the attempt to repeatedly login from multiple locations across time that’s the problem. So what you have to look for is not necessarily (or only) repeated requests but also at repeated attempts to access specific resources, like a login page.

But a login page is going to see a lot of use so it’s not just the login page you need to be concerned with, but the credentials, as well. In any brute force account level attack there are going to be multiple requests to try to access a resource using the same credentials. That’s the trigger. It requires more context than your traditional connection or request based security triggers because you’re not just looking at IP address, or resource, you’re looking deeper and trying to infer from a combination of data and behavior what’s going on.

THIS SITUATION will BECOME the STATUS QUO

As we move forward into a new networking frontier, as applications and users are decoupled from IP addresses and distribution of both clients and applications across cloud computing environments becomes more common, the detection of attacks is going to get even more difficult without collaboration.

The new network, the integrated and collaborative network, the dynamic network, is going to become a requirement.  Network and application network and security infrastructure needs a new way of combating modern threats and their attack patterns. That new way is going to require context and the sharing of that context across all strategic points of control at which such attacks might be mitigated.

This becomes important because, as pointed out earlier, many web applications rely upon third-party solutions for functionality. Open source or not, it still takes time for developers to implement a solution and subsequently for organizations to incorporate and redeploy the “patched” application. That leaves users of such applications vulnerable to exploitation or identity theft in the interim. Security infrastructure must be able to detect such attacks in order to protect users and corporate resources (infrastructure and applications and data) from exploitation while they wait for such solutions. When is more important than where when it comes to addressing newly discovered vulnerabilities, but when is highly dependent upon the ability of infrastructure to detect an attack in the first place.

We’re going to need infrastructure that is context-aware.

F5 Friday: An On-Demand Turing Test

Detecting bots requires more than a simple USER_AGENT check today…

Anyone who’s taken an artificial intelligence class in college or grad school knows all about the Turing Test. If you aren’t familiar with the concept, it was a “test proposed by Alan Turing in his 1950 paper Computing Machinery and Intelligence, which opens with the words: "I propose to consider the question, 'Can machines think?'" Traditional Turing Tests always involve three players, and the goal is to fool a human interviewer such that the interviewer cannot determine which of the two players is human and which is a computer. There are variations on this theme, but they are almost always focused on “fooling” an interviewer regarding some aspect of the machine that it is attempting to imitate.

Common understanding has it that the purpose of the Turing Test is not specifically to determine whether a computer is able to fool an interrogator into believing that it is a human, but rather whether a computer could imitate a human.^[44] While there is some dispute whether this interpretation was intended by Turing — Sterrett believes that it was^[43] and thus conflates the second version with this one, while others, such as Traiger, do not^[41] — this has nevertheless led to what can be viewed as the "standard interpretation." In this version, player A is a computer and player B a person of either gender. The role of the interrogator is not to determine which is male and which is female, but which is a computer and which is a human.<sup>[45]

-- Wikipedia, Turing Test</sup>

Over the past decade, as the web has grown more connected and intelligent, so too have the bots that crawl its voluminous pages attempting to index the web and make it possible for search engines like Google and Bing to be useful. Simultaneously have come the evil bots, the scripts, the automated attempts at exploiting vulnerabilities and finding holes in software that enable malicious miscreants to access data and systems to which they are not authorized. While a web application firewall and secure software development lifecycle practices can detect an attempted exploit, neither are necessarily very good at determining whether the request is coming from a bot (machine) or a real user.

Given the very real threat posed by bots, it’s becoming increasingly important for organizations to detect and prevent these automated digital rodents from having access to web applications, especially business-critical applications. The trick is, however, to determine which requests are coming from bots and which ones are coming from real users. It’s a trick not only because this determination is difficult to make with a high degree of confidence in the result, but because it needs to be determined on-demand, in real-time.

What organizations need is a sort of “on-demand Turing test” that can sort out the bots from the not bots.

In a recent release of F5’s web application firewall, ASM (Application Security Manager), there was a very neat little “trick” that essentially provides just what we’ve described: an on-demand Turing test. This capability of ASM attempts to detect whether or not the client side is human by monitoring client-side interaction events. The assumption is, of course, that a bot will not be interacting with the page but a human must interact – clicking a mouse, typing – in order to interact with the application.

The intention behind this feature is to allow administrators to configure policies that act upon the determination. This is particularly helpful in preventing web scraping and other automated attack tools that make use of scripts and bots to attack sites.

How does it work?

Because ASM is deployed as part of a unified application delivery platform, it is able to take advantage of the systems’ underlying foundational technologies. In this case, TMOS – a full proxy that provides the capability to intercept, inspect, and transform application data. It is application aware, and can interact in real-time with the messages being exchanged between the client and the server. In much the same way as we leverage this capability to inject the Gomez Javascript code necessary to monitor any application for which BIG-IP provides application delivery services, we can inject Javascript into clients that provides the information necessary to determine whether the client is a human being or an automated system.

Based on the results of the determination, the administrator can take action. Rejecting the connection or redirecting to a honey-pot are common actions to take, but ASM like all TMOS-enabled solutions offered as part of the BIG-IP family of solutions the limits are really based on organizational need. iRules (network-side scripting) can be employed to provide additional custom and unique actions, logging, etc…or a pre-defined set of actions can be easily applied. This also provides some measure of future-proofing against modifications by miscreants to bots and automated scripts. Evolving the heuristics used to determine human from bot can be updated without requiring upgrades to the system, requiring modification of injected scripts. the ability to leverage iRules further allows organizations to develop their own methods that may or may not be based on ASM’s included functionality.

This is not a panacea. There is no web application security solution that is. But it is more than what is currently available in most web application security solutions today, and it’s something that isn’t generally included in the application itself, for a variety of reasons. It’s certainly not listed in the OWASP top ten methods for securing an application. There are legitimate uses for bots, after all, so the use of such a deterministic system to unilaterally deny or permit requests must be carefully balanced across the typical usage patterns and needs of the organization.

Regardless, it is another tool in the information security toolbox that can be applied to better secure applications against “users” of malicious intent, and it’s certainly moving web application security in the right direction by being as context-aware as possible when attempting to respond to a request for a resource.

Booth Babes Don’t Wear Glasses

Gidget or gadgets, that’s not really what the debate is all about anyway

An axiomatic truth of technology today is that women in technology are, few and far between. The recent debate over booth babes slides naturally into the question “how can we encourage young women to enter science, technology, engineering and mathematics (STEM) fields”? After all, the argument goes, what young woman would willingly enter a field where she’ll be assumed to be ignorant unless she proves otherwise? Where she’ll be admired not for her Masters degree but for her mastery of makeup?

STEM, and technology in particular, has an unclear career path that tends to put off young women and, unsurprisingly, young men, too. Young women, according to research, aren’t thinking about the difficulties that exist being a woman in a traditionally male-dominated field – and they do exist, there’s no denying that. They’re thinking “what the heck would I do with a degree in [insert STEM field here]?” [Scientific American, “U.S. Students say “yah for science””] Young women aren’t avoiding STEM because technology vendors at technology shows are hiring “booth babes.” In fact looking closer at the debate that’s risen regarding booth babes it really has nothing to do with them and everything to do with men and attitudes.

It isn’t the existence of booth babes that causes the treatment cited by Denise Dubie  in a recent post on this very debate over the practice, that is the result of a field that’s (1) still young compared to other STEM fields, (2) dominated by men, and (3) the fact that women in general haven’t been out en force  for all that long. It’s been less than 100 years since the right of women to vote was recognized, and less than 40 since we’ve really been accepted into the work force. Most of us are first and maybe second generation “working women”, professional women. We’re still paving the path for our daughters and their daughters. It’s no surprise that men in general, then, haven’t had a whole lot of time yet to adjust.

A recent post by Network World blogger Michael Morris caused a bit of discussion on that site as well as chatter across social media network Twitter. He held a contest immediately following Cisco Live to identify the hottest booth babe at the show.

[…]

The one thing I do note at these shows, being in a position required to approach and speak to attendees and being female, is that I am in the definitely in the minority there. And often when I approach male attendees, they seem a bit shocked that I am talking technology with them. And when sitting in on sessions and looking around the room, I am always elated to find another female in attendance. But I have found that some of those women work for the vendor doing the presentation and aren’t IT professionals attending the show. Still I have to admit in the 10+ years of going to shows, the number of women in attendance has increased, based on my informal opinion. [emphasis added]

The ‘booth babe’ debate

Denise Dubie, NetworkWorld

It would likely be difficult to find a woman in technology that hasn’t had this same experience. The shock, the surprise, the change in tone and demeanor that comes from male counterparts upon realizing that the woman they’re talking to knows what she’s talking about. In some cases, she knows a heck of a lot more than they do. But that “shock and awe” isn’t restricted to trade shows, and it’s really got nothing to do with booth babes. It’s the preconceived notions many  men carry along with them to trade shows. Notice that these opinions didn’t originate at a trade show replete with lusty booth babes, they were carried along. That shock and awe isn’t peculiar to trade show environments, it is everywhere. On the phone, on a webinar, in a lab. In environments where there are no booth babes. Hence, they are not really part of the “problem” at all.

Go ahead, take the booth babe out of the picture. The assumptions and pre-conceived notions held by many in IT will still be there. The front line in any booth is assumed to be staffed by both male and female “talent” or employees that are not necessarily very technical. Period. Whether they’re male or female, that “screening line” isn’t going to be able to describe in detail, for example, the anatomy of an SSL handshake. Go ahead, ask. Ask a man, ask a woman. Doesn’t matter. Front line booth staff are almost unilaterally there to screen and direct. They exist to welcome you to the booth, scan your badge, and to help you get to the right person if you’re interested in a discussion. They are there because they are outgoing and friendly and enjoy engaging with people, for which those of us who are hiding “in the booth” and are not outgoing are grateful.

Now many folks point out that using booth babes generates a lot of useless leads. Mostly true. But the same can be said for interesting swag, as well. Folks who visit a booth to snatch up the latest geegaw or enter a drawing to win an iPad aren’t exactly solid leads in the first place, so whether they’re attracted by a nice set of legs or a shiny toy isn’t really all that relevant now, is it? It’s not as if the guy who stopped by for a shiny, sparkly ball for his kids is any more interested in your product than the one who was attracted by a beautiful set of … legs. 

It is more than expected that women in technology want the respect they deserve based on their knowledge and expertise, not their choice in clothing or shoes. But how do you know who they are? And for that matter, how do you which men are the ones that have the knowledge you’re looking for? You can’t assume (or shouldn’t) that any given man at a technology show is any more technically astute than another because there are a variety of folks who attend those shows: from the IT manager who may not have 1337 technical skillz to the router jockey that does. Despite the attempts of Hollywood to the contrary, you can’t tell who’s is or isn’t a geek by looking at them. You simply can’t judge the competency and knowledge of an IT worker from their outward appearance or their title or even their degree.

On the other hand, you can be relatively sure that a booth babe is just that: a booth babe. Their proximity to the booth and uniform of short skirts, tight tops, and heavily glossed lips is hard to miss. I am not likely to be mistaken for a booth babe and conversely they are unlikely to be mistaken for someone with a clue. I will, however, almost certainly run into someone who is very surprised that I know a thing or two. Why? Because I’m a woman. In technology. And as much as I’d like to at times, I don’t carry a 2x4 wrapped in copies of my master’s degree in computer science, so there’s no obvious clue other than the fact that I’m not dressed like a booth babe. 

The debate over “booth babes” isn’t really about booth babes. It’s not even really about the inability to distinguish the geeks from the rest of the crowd. It’s about the reaction of men to a technically competent woman, no matter what she’s wearing. It’s the surprise and the shock in their expression and voice. It’s about the assumption that no woman is technically competent – at a trade show or on a conference call. That’s at the heart of this debate and others like it, and whether booth babes are present or not is unlikely to have an impact on those assumptions.    

NOTE: It’s been mentioned by “those in the know” that the event that started this whole brouhaha, CiscoLive, will be “booth babe” free in the future.

Out, Damn’d Bot! Out, I Say!

Exorcising your digital demons

Most people are familiar with Shakespeare’s The Tragedy of Macbeth. Of particularly common usage is the famous line uttered repeatedly by Lady Macbeth, “Out, damn’d spot! Out, I say” as she tries to wash imaginary bloodstains from her hands, wracked with the guilt of the many murders of innocent men, women, and children she and her husband have committed.

It might be no surprise to find a similar situation in the datacenter, late at night. With the background of humming servers and cozily blinking lights shedding a soft glow upon the floor, you might hear some of your infosecurity staff roaming the racks and crying out “Out, damn’d bot! Out I say!” as they try to exorcise digital demons from their applications and infrastructure. 

Because once those bots get in, they tend to take up a permanent residence. Getting rid of them is harder than you’d think because like Lady Macbeth’s imaginary bloodstains, they just keep coming back – until you address the source.

One of the ways in which a bot can end up in your datacenter wreaking havoc and driving your infosec and ops teams insane is through web application vulnerabilities. These vulnerabilities in both the underlying language and server software as well as the web application itself, are generally exploited through XSS (Cross-site scripting). While we tend to associate XSS with attempts to corrupt a data source and subsequently use it as distribution channel for malware, a second use of XSS is to enable the means by which a bot can be loaded onto an internal resource. From there the bot can spread, perform DoS attacks on the local network, be used as a SPAM bot, or join in a larger bot network as part of a DDoS.

The uses of such deposited bots is myriad and always malevolent.

Getting rid of one is easy. It’s keeping it gone that’s the problem. If it entered via a web application it is imperative that the vulnerability be found and patched. And it can’t wait for days, because the bot that likely exploited that vulnerability and managed to deploy the bot inside your network is probably coming back. In a few hours. You can remove the one that’s there now, but unless the hole is closed, it’ll be back – sooner rather than later.

According to an ARS Technica report, you may already know this pain as “almost all Fortune 500 companies show Zeus botnet activity”:

Up to 88% of Fortune 500 companies may have been affected by the Zeus trojan, according to research by RSA's FraudAction Anti-Trojan division, part of EMC.

The trojan installs keystroke loggers to steal login credentials to banking, social networking, and e-mail accounts.

This is one of those cases in which when is more important than where the vulnerability is patched initially. Yes, you want to close the hole in the application, but the reality is that it takes far longer to accomplish that then it will take for attackers to redeposit their bot. According to WhiteHat Security’s Fall 2009 Website Security Statistics Report a website is 66 percent likely to be vulnerable to an XSS exploit, and it will take an average of 67 days for that vulnerability to be patched.

That’s 67 days in which the ops and infosec guys will be battling with the bot left behind – cleaning it up and waiting for it to show up again. Washing their hands, repeatedly, day and night, in the hopes that the stain will go away.

The best answer to keeping staff sane is to employ a security solution that’s just a bit more agile than the patching process. There are several options, after all, that can be implemented nearly immediately to plug the hole and prevent the re-infection of the datacenter while the development team is implementing a more permanent solution.

• Virtual patching provides a nearly automated means of plugging vulnerabilities by combining vulnerability assessment services with a web application firewall to virtually patch, in the network, a vulnerability while providing information necessary to development teams to resolve the issue permanently.
• Network-side scripting can provide the means by which a vulnerability can be almost immediately addressed manually. This option provides a platform on which vulnerabilities that may be newly discovered and have not yet been identified by vulnerability assessment solutions can be addressed.
• Web application firewall can be manually instructed to discover and implement policies to prevent exploitation of vulnerabilities. Such policies generally target specific pages and parameters and allow operations to target protection at specific points of potential exploitation.

All three options can be used together or individually. All three options can be used as permanent solutions or temporary stop-gap measures. All three options are just that, options, that make it possible to address vulnerabilities immediately rather than forcing the organization to continually battle the results of a discovered vulnerability until developers can address it themselves.

More important than how is when, and as important as when is that the organization have in place a strategy that includes a tactical solution for protection of exploited vulnerabilities between discovery and resolution. It’s not enough to have a strategy that says “we find vulnerability. we fix. ugh.” That’s a prehistoric attitude that is inflexible and inherently dangerous given the rapid evolution of botnets over the past decade and will definitely not be acceptable in the next one.

John Pescatore, vice president and distinguished analyst, speaking about the enterprise threat landscape Tuesday at the Gartner Security and

Risk Management Summit, said rapid changes are taking place in the way IT organizations deliver services, largely via virtualization and cloud computing , and the way businesses consume them, which create infrastructure breakage points ripe for exploit.

"Ninety percent of attacks are exploiting vulnerabilities we already knew about, by missing patches, deciding not to patch, or uses of technology in which we made the decision to deploy without putting security controls on it," Pescatore said. "Less than 1% are zero-day attacks; the other 99% are exploited configurations and unpatched machines that the simplest vulnerability scan would've found."

And since attackers have had so much success using bots to take advantage of these vulnerabilities, Pescatore said that trend is likely to continue through 2012, when the ongoing cat-and-mouse game between attackers and enterprise defenders leads to the next threat delivery breakthrough.

With IT and security resources always at a premium, Pescatore said enterprises must prioritize their defensive efforts simply by asking themselves what an attacker would want to get from them; in most cases it's personally identifiable information (PII) in the form of credit card and Social Security numbers, or sensitive intellectual property. [emphasis added]

Gartner: Enterprises must learn to detect botnet threats 

It’s going to get harder and harder to exorcise those digital demons. That means the best strategy is to be proactive; to ensure the proper protections are in place before you go live. To provide developers with the training and tools to ensure that known vulnerabilities are addressed, and that the infrastructure or application is sanitizing inbound data and scrubbing outbound data.

Protecting the organization is no longer about which group or which solution should or shouldn’t be responsible for web application security. When web application vulnerabilities took legs and began to be a problem for the entire datacenter and the business, it became more about enabling solutions and less about who or how or what.

Patch the vulnerability first, in whatever way can make that happen immediately. Then you can discuss and debate about what’s the proper long-term solution for your organization. After all, no one would suggest that the little Dutch boy who stuck his finger in the dike should stand there forever, holding back the flood of water. But no one would suggest he was wrong to do it, either.

Does This Application Make My Browser Look Fat?

Web applications that count on the advantage of not having a bloated desktop footprint need to keep one eye on the scale…

A recent article on CloudAve that brought back the “browser versus native app” debate caught my eye last week. After reading it, the author is really focusing on that piece of the debate which dismisses SaaS and browser-based applications in general based on the disparity in functionality between them and their “bloated desktop” cousins.

Why do I have to spend money on powerful devices when I can get an experience almost similar to what I get from native apps? More importantly, the rate of innovation on browser based apps is much higher than what we see in the traditional desktop software.

[…]

Yes, today's SaaS applications still can't do everything a desktop application can do for us. But there is a higher rate of innovation on the SaaS side and it is just a matter of time before they catch up with desktop applications on the user experience angle.

When You Can Innovate With browser, Why Do You Need Native Apps?, Krishnan Subramanian 

I don’t disagree with this assessment and Krishnan’s article is a good one – a reminder that when you move from one paradigm to another it takes time to “catch up”. This is true with cloud computing in general. We’re only at the early stages of maturity, after all, so comparing the “infrastructure services” available from today’s cloud computing implementations with well-established datacenters is a bit unfair.

But while I don’t disagree with Krishnan, his discussion reminded me that there’s another piece of this debate that needs to be examined – especially in light of the impact on the entire application delivery infrastructure as browser (and browser-based applications) capabilities to reproduce a desktop experience mature. At what point do we stop looking at browser-based applications as “thin” clients and start viewing them for what they must certainly become to support the kind of user-experience we want from cloud and web applications: bloated desktop applications.

The core fallacy here is that SaaS (or any other “cloud” or web application) is not a desktop application. It is. Make no mistake. The technology that makes that interface interactive and integrated is almost all enabled on the desktop, via the use of a whole lot of client-side scripting. Just because it’s loaded and executing from within a browser doesn’t mean it isn’t executing on the desktop. It is. It’s using your desktop’s compute resources and your desktop’s network connection and it’s a whole lot more bloated than anything Tim Berners-Lee could have envisioned back in the early days of HTML and the “World Wide Web.” In fact, I’d argue that with the move to Web 2.0 and a heavy reliance on client-side scripting to implement what is presentation-layer logic that the term “web application” became a misnomer. Previously, when the interface and functionality relied solely on HTML and was assembled completely on the web-side of the equation, these were correctly called “web” applications. Today? Today they’re very nearly a perfected client-server, three-tiered architectural implementation. Presentation layer on the client, application and data on the server. That the network is the Internet instead of the LAN changes nothing; it simply introduces additional challenges into the delivery chain.

But Lori! Desktop applications require installation! They require maintenance, and support! They are more costly! They are evil! An ancient horror that we have worked hard to eradicate from the face of laptops and desktops everywhere!

Let’s get a grip on reality, shall we?

As more and more of that presentation layer logic moves into the client – the browser, the desktop – those same issues are going to arise. Perhaps not on the same scale, but they will arise. When you start relying on libraries – frameworks like Prototype and XAJAX and the like – to provide the underlying framework through which all that interactivity and cool functionality is achieved, you are basically installing software. It’s just as hidden from the end-user as much of the desktop software, at least in an enterprise setting. It still requires maintenance and support and oh how problematic is the support! When an AJAX call goes “bad” it’s not uncommon at all for a web application to fail to provide the means by which the end-user is notified or that updating can be “restarted”. At some point that’s going to generate a support call to someone, and troubleshooting a browser-based application has got to be the most miserable experience on the face of the earth. You don’t have – can’t have – an exact duplicate of the user’s environment because, well, you may not have the same plug-ins installed, or the same version of the plug-ins. You may not be accessing the application over the same network with the same underlying infrastructure that may or may not be causing issues. You may not even be running on the same operating system – or device. But you’re still going to have to troubleshoot it and figure out what’s causing the problem somehow.

Good luck with that.

I’m not saying the evolution of the browser into the modern-day desktop – and that is what is happening – is a bad thing. But I am pointing out that at some point the same issues that once made us look upon browser-based applications as a godsend will crop up and cause the same kind of headaches for IT and end-users and in some cases those headaches will be migraines that never existed when IT had control over the environment.

The days of SaaS and “web applications” claiming an almost moral superiority over “bloated desktop applications” are over. They went the way of “Web 1.0” – lost to memory and saved only in the “wayback machine” to be called forth and viewed with awe and disbelief by digital millennials that any one ever used applications so simple.

Today’s “web” applications are anything but simple, and the environment within which they execute is anything but thin. 

F5 Friday: Would You Like Some Transaction Integrity with Your Automation?

If you thought the integration and collaboration required new networking capabilities, you ain’t seen nothing yet.

Anyone who has ever configured a network anything or worked with any of a number of cloud provider’s API to configure “auto-scaling” via a load balancing service recognizes that it isn’t simply point, click, and configure. Certain steps need to be configured in a certain order (based entirely on the solution and completely non-standardized across the industry) and it’s always a pain to handle errors and exceptions because if you want to “do over” you have to backtrack through the completed steps or leave the system cluttered or worse – unstable.

Developers and system operators have long understood the importance of a “transaction” in databases and in systems where a series of commands (processed in a “batch”) are “all or nothing”. Concepts like two-phase commit and transaction integrity are nothing new to developers and sysops and probably not to network folks, either. It’s just that the latter has never had that kind of support and have thus had to engineer some, shall we say, innovative solutions to recreating this concept.

Infrastructure 2.0 and cloud computing are pushing to the fore the need for transactional integrity and idempotent configuration management. Automation, which is similar to the early concepts of “batch” processing, requires that a series of commands be executed that individually configure the many moving pieces and parts of an overarching architecture that are required in order to “make the cloud go” and provide the basic support necessary to enable auto-scaling.

Because it is possible that one command in a sequence of ten, twenty, or more commands that make up an “automation” could fail, you need to handle it. You can catch it and try again, of course, but if it’s a problem that isn’t easily handled you’d wind up wanting to “roll back” the entire transaction until the situation could be dealt with, perhaps even manually. One way to accomplish this is to enable the ability to package up a set commands as a transaction and, if any command fails the transaction management system automagically hits the “undo” button and rolls back each command to the very beginning, making it like it never happened.

We’re not there yet. If you were waiting for me to pronounce “we have arrived” I’m sorry. We haven’t, but we are at least recognizing that this is a problem that needs solutions. In order for the above scenario to be reality every system, every device, every component that is (or could be) part of the transaction must be enabled with such functionality.

Transactions are essential to maintaining the integrity of components across an orchestration in complex systems. That basically means most – if not all – datacenters will need transactions eventually if they go the route of automation to achieve operational efficiency. This is another one of those “dev” meets “ops” topics in which dev has an advantage over ops merely due to the exigencies of being in development and working with integration and databases and other development-focused systems that require transactional support. Basically we’re going to (one day) end up with layers of transactions: transactions at the orchestration layer that is comprised of individual transactions at the component layer and it will be imperative that both layers of transactions are able to handle errors, be idempotent (to handle restarts), and to provide integrity of the implied shared systems that make up a cloud computing environment (that’s fault isolation, by the way).

The necessity of transactional support for infrastructure 2.0 components is going to become more important as larger and more business critical systems become “cloudified”. The new network has to be more dynamic and that means it must be able to adapt its operating conditions to meet the challenges associated with integration and the dependencies between devices and applications that creates.

For F5, we’ve taken a few of the first steps down the road to Tipperary (and the eventually fully transactional-based automation implementation required to get there) by enabling our BIG-IP shell scripting solution (TMSH) with some basic transactional support. It isn’t the full kitchen sink, it’s just a start, and as you can guess you should “watch this space” for updates and improvements and broader support for this concept across the entire BIG-IP management space.

If you weren’t aware, back in a prior release (we called it Park City, you might know it as v10) there was a capability added called TMSH (TMOS SHell) which is an action/object based scripting shell for F5 BIG-IP that allows administrators to script the configuration and management of a BIG-IP in much the same way as they might develop system scripts in BASH and KORN or PowerShell or whatever their favorite shell scripting language may be.

What was really cool was the addition of the ability to wrap transactions around a series of TMSH commands.  What transactions allow you to do is bundle a list of commands to be executed as a single atomic operation. Now, we’re not anywhere near the “two phase commit” capabilities of mature batch processing systems like IBM and databases might have, but we are a far sight further along than perhaps people realized. When you’re “in” a transaction, for example, the command is only validated for syntax. It isn’t executed until the transaction is submitted for execution, you can’t undo a submitted transaction, and right now you can’t create “named” transactions that you can queue up for approval or save to execute at a later date or what have you.  You can, however, recreate a transaction you have deleted by using the cli history component.

   1: root@bigip(Active)(tmos)# create transaction

   2: [batch mode]root@bigip(Active)(tmos)# create ltm pool a_pool members add { 10.2.27.1:80 }

   3: Command successfully added to the current transaction

   4: [batch mode]root@bigip(Active)(tmos)# create ltm pool a_pool members add { 10.2.16.1:80 }

   5: Command successfully added to the current transaction

   6: [batch mode]root@bigip(Active)(tmos)# submit transaction 

   7: 01020066:3: The requested pool (a_pool) already exists in partition Common

   8: [batch mode]root@bigip(Active)(tmos)# list ltm pool a_pool

   9: 01020036:3: The requested pool (a_pool) was not found.

  10: [batch mode]root@bigip(Active)(tmos)# list transaction 

  11: 1: (tmos)# create ltm pool a_pool members add { 10.2.27.1:80 }

  12: 2: (tmos)# create ltm pool a_pool members add { 10.2.16.1:80 }

  13: [batch mode]root@bigip(Active)(tmos)# modify transaction delete 1

  14: [batch mode]root@bigip(Active)(tmos)# submit transaction 

  15: root@bigip(Active)(tmos)# list ltm pool a_pool

  16: pool a_pool {

  17:     members {

  18:         10.2.16.1:http { }

  19:     }

  20: }

This is only one (small) piece of a much larger puzzle: the automated, dynamic infrastructure (a la infrastructure 2.0). And it’s a piece that’s not even wholly complete yet. But it’s a start on what’s going to be necessary if we’re going to automate the data center and realize the potential gains in efficiency and move even further into the “automated self-healing, self-optimizing” networks of the (further out) future. We can’t get even to a true implementation of “cloud” until we can automate the basic processes needed to implement elastic scalability because otherwise we’d just be shifting man hours from racking servers to tapping out commands on routers and switches and load balancers.

But it’s a step forward on the road that every one is going to have travel.

The New Distribution of The 3-Tiered Architecture Changes Everything

As the majority of an application’s presentation layer logic moves to the client it induces changes that impact the entire application delivery ecosystem

The increase in mobile clients, in demand for rich, interactive web applications, and the introduction of the API as one of the primary means by which information and content is shared across applications on the web is slowly but surely forcing a change back toward a traditional three-tiered architecture, if not in practice then in theory. This change will have a profound impact on the security, delivery, and scalability of the application but it also forces changes in the underlying network and application network infrastructure to support what is essentially a very different delivery model.

What began with Web 2.0 – AJAX, primarily – is continuing to push in what seems a backward direction in architecture as a means to move web applications forward. In the old days the architecture was three-tiered, yes, but those tiers were maintained almost exclusive on the server-side of the architecture, with the browser acting only as the interpreter of the presentation layer data that was assembled on the  server. Early AJAX applications continued using this model, leveraging the out-of-band (asynchronous) access provided by the XMLHTTPRequest object in major  browsers as a means to dynamically assemble smaller pieces of the presentation layer. The browser was still relegated primarily to providing little more than rendering support.

Enter Web 2.0 and RESTful APIs and a subtle change occurred. These APIs returned not presentation layer fragments, but data. The presentation layer logic required to display that data in a meaningful way based on the application became the responsibility of the browser. This was actually a necessary evolution in web application architecture to support the increasingly diverse set of end-user devices being used to access web applications. Very few people would vote for maintaining the separation of presentation layer logic used to support mobile devices and richer, desktop clients like browsers. By forcing the client to assemble and maintain the presentation layer that complexity on the server side is removed and a single, unified set of application logic resources can be delivered to every device without concern for cross-browser, cross-device support being “built in” to the presentation layer logic. 

This has a significant impact on the ability to rapidly support emerging clients – mobile and otherwise – that may not support the same robust set of capabilities available on a traditional browser. By reducing the presentation layer assembly on the server side to little more than layout – if that – the responsibility for assembling all the components and their display and routing data to the proper component is laid on the client. This means one server-side application truly can support both mobile and desktop clients with very little modification. It means an API provided by a web application can not only be used by the provider of that API to build its own presentation layer (client) but third-party developers can also leverage that API and the data it provides in whatever way it needs/chooses/desires.

This is essentially the point to which we are almost at today.

This change on the surface may appear to be innocuous but for network and application delivery network and security professionals it should ring some alarm bells. For network and application delivery network folks this model changes the usage patterns of TCP and HTTP. It is no longer about HTML per se but about smaller, more abstract fragments of HTML and then little more than data packaged up in XML or JSON or CSV. Where once there was a great disparity between the size of a request and the size of a response now there is more equality. Where once there was a predictable number of request-response pairs per user connection, now there is a variable number. Where once there was a specific time-frame in which those request-response pairs would be sent and received, now the time-frame for interaction between the client and the server is indeterminable, based solely on the whim of the user and not on the protocol.

Data is now returned in multiple, schema-less formats that might be served by the same API invocation. Data is returned instead of HTML, for which intermediaries providing security (web application firewall) and load balancing and data leak prevention have long had capabilities to protect but now find it more difficult to provide. Caching of HTML objects is rendered less useful as there are fewer HTML objects being sent to clients because they are now being assembled on the client and data is now transmitted instead. TCP sessions are held open longer, consuming resources on intermediaries and servers, and over which are sent more requests more frequently.

The changes in the application development and deployment architecture significantly impact the entire application delivery infrastructure, forcing more focus on TCP and HTTP optimization and less on caching and compression. It incurs a higher rate of throughput because requests and replies are more frequent, and imposes a higher burden on the network due to the smaller size of both.

The redistribution of the three-tiered architecture to rely upon the client as the presentation layer is necessary to remove the complexity inherent in supporting multiple clients with varying presentation capabilities on the server, and makes more efficient the server-side processing. It moves that complexity into the presentation layer, onto the client, but this is operationally more efficient as the burden of execution becomes the client’s and not the server’s. The costs, then, associated with the processing necessary to determine the type of client and assemble the presentation layer are eliminated and transferred to the end-client.

But this redistribution imposes new burdens and introduces new challenges to the infrastructure that support it. It can no longer rely upon the HTML specification as a means to enforce security or acceleration or optimization. It must be able to adapt and support the same capabilities for schema-less data exchanges and be capable of modifying its those capabilities based on real-time usage patterns that are likely to be unpredictable across time. This is why the role of devops is emerging – because the very deployment model of an application changes by nature its behavior and how it uses its supporting infrastructure. Network and application network infrastructure tuning and optimization becomes less about bandwidth and more about behavior.  Ops needs to understand the application, and dev needs to understand the impact of TCP and longer sessions and smaller data on the network and its application servers. The two disciplines must communicate and merge to be able to deliver such applications in the most efficient manner possible.

Welcome to the brave new web.