It seems like I blinked and 2009 went by, but in that time I've been working on so many interesting projects at F5, I have a backlog of information to share with the community.  The first post this year is about the long distance VMotion with VMWare's ESX system.  This is a solution that enables the movement of live running virtual machine hosts from one data center to another.

The main problems in routing VMotion between data centers are latency, bandwidth, client traffic and security.  In BIG-IP 10.1 we have a solution that compresses, encrypts and shields the ESX servers from prevailing WAN conditions, to enable long distance motion of running hosts.  Take a look at the following screencast to see how this works:

 In the chart below are some of the typical improvement times we see with long distance VMotion with BIG-IP.  When latency goes up, VMotion is often not possible without BIG-IP in place.  For example, with 100 ms of round-trip latency, on an OC3, a virtual machine that has one gigabyte of active RAM memory, takes roughly three and a half minutes to migrate across the WAN.  If you were to try the same VMotion without BIG-IP in place, it would take more than 13 minutes and only succeed about half the time.

I'm excited about the types of architectures that can be enabled with this kind of solution in place.  F5 is laying the ground work to make some exciting infrastructures possible

Have a look at the F5 deployment guide which describes how to set this solution up and how to architect new solutions across your data centers: http://www.f5.com/pdf/deployment-guides/vmware-vmotion-dg.pdff

 Are there ways that F5 Networks and SAP can make your SAP operations simpler, more elegant and more automated, with appropriate controls in place to detect abnormalities?

With V10, F5 Networks is delivering three things: first, template based configuration support for SAP Enterprise Portal and Web Based SAP ECC Instances (so that a few questions and one submit button perfectly configures your instances).  Second, integration with SAP's ASLR (described below) to automatically detect and help configure all available SAP instances (so that your hunt for SAP instance numbers and port numbers during configuration is over). And third, a complete monitor that individually logs into each configured SAP Instance and checks that instance's unique health status (and then reacts appropriately).

SAP and F5 Networks's partnership manifests itself in exceptional ways and participating in the SAP Enterprise Services Community, part of SAP's Communities of Innovation was one of these opportunities.  SAP's idea was elegant and came with a very practical goal: SAP reached out to all of the networking partners and asked us how we can simplify SAP NetWeaver operations.

The initial presentation from SAP asked this high level question and presented a series of APIs that could be used to build these solutions.  SAP challenged us with the analog of the fly wheel governor (pictured above); could we invent a system that controls SAP based on the current working conditions. Through a series of working sessions with SAP we went to work at F5 Networks to solve this problem and the results of our work will be presented at the Americas SAP User Group Meeting (ASUG) in May.

Our solution has three components, detection of SAP instances, configuration of SAP web instances (Portal or other web based ECC instances) for high availability, web acceleration and security and monitoring of SAP web instances from login to database for a complete picture of what is up and what is down.  These steps are performed with the aid of the SAP Application Server List Retrieval (ASLR) API which is part of the SAP Message Server (standard on all SAP NetWeaver installations out-of-the-box).

Configuration - Support for SAP Enterprise Portal and all your Web Based SAP ECC instances.

With F5 Networks BIG-IP V10 templates for SAP ERP Portal and SAP ECC instances, configuring BIG-IP with advanced application delivery controls is accomplished with just a few questions to be answered on one single page.  Below, you can see a screen shot of my BIG-IP and the templates we have available today, especially for SAP, Enterprise Portal and a more generic SAP ECC template for the installation of any additional SAP Web Based instances.

 

 

Detection - Integration with SAP ASLR to automatically configure all SAP instances.  The hunt for SAP instance numbers and port numbers is over.

With the coming addition of SAP ASLR integration (planned for after the upcoming ASUG meeting), we will have the message server integration pictured below.  While this may look like a very small number of questions, the template takes care of all of the aspects of configuring SAP Portal. Of note to me is how easy it was previously for customers to miss important optimizations even though we detail and clearly document these in our deployment guides (for example the SAP Deployment Guide for V9).  With BIG-IP V10 SAP application delivery will be the fastest possible through the network, every time.

Now, by providing the SAP Message Server IP Address and port number, BIG-IP automatically retrieves and populates the SAP instances in the load balancing portion of the questionnaire. The hunt for instance numbers and port numbers is over, with the cooperation of ASLR and BIG IP Application Templates.

Monitoring - A complete monitor that individually logs into each SAP Instance and checks that instance's unique health status (and then reacts appropriately)

One of the shortcomings that F5 Networks has found with the APIs as they stand today is that although graceful shutdown is well detected by the ASLR API, more is needed to address unforeseen outages.  We hope that with the community involvement this will be addressed in coming versions of the API.

To solve this issue now, our SAP templates install a series of health monitors to cover the state of SAP Web Instances up the entire stack.  We begin with automatic configuration of ping and port monitors (which indicates the individual server or VM instance is up).  We then configure a monitor which checks for HTTP/1.1 presence (which indicates that SAP Dispatcher is up) and finally we now will recommend the addition of a health monitor to log into the SAP Web Application portal and check for a specified piece of validation text (we then log out, of course).  None of this configuration requires any command line interaction and can be delegated to SAP NetWeaver or Basis administrators.  Below you can see that we allow the configuration of the login username, password and validation text all via the UI.

For more on this project from our wonderful partners at SAP I highly recommend checking out Joerg Nalik's post at the SAP Community Network: Catching Up with Deploying and Operations Automation.

Analyze acquisition costs, maintenance costs, stability, performance, range of functions, high availability, security, configuration complexity, integration into your existing landscape and content caching when deciding between software and hardware load balancing.

F5 and SAP have a close and important partnership that involves engineers and business folks working to deliver the best customer experience for any given SAP landscape.  One of the questions that comes up over and over is how a company can make the decision between software load balancing and hardware high availability.  We recently published a document (that's available by emailing me ) about the logistics and considerations of making the choice or making the switch.

I will provide a summary of the document here, but please contact me if you need the document itself. 

In the most basic SAP Landscape SAP clients (either SAP GUI or web browser) make direct connections to SAP instances.  For example, a user wishing to file her travel expenses might click on the link from her company's portal, log into SAP (or be signed on automatically using single sign-on) and then she would fill out her expenses and submit them.

You can probably tell by the above scenario alone that this isn't a very reliable or scalable architecture.  Connecting directly to the instance means there's no load balancing.  All the users will end up at the same link.  What if our hypothetical user was part of a convention and there were thousands of users just like her trying to file the expenses on Monday morning?  The system would probably fail or come to a halt because of the load of traffic.

SAP with Web Dispatcher is the software load balancing solution to this problem.  In a software load balanced SAP Landscape, high availability is achieved when SAP clients connect directly to Web Dispatcher and then are load balanced to the back-end SAP servers.  Web Dispatchers can be installed either on the Central Instance or on its own server.  Web Dispatcher communicates with SAP Message Server to get health information about the SAP instances and Web Dispatcher also analyzes web browser cookies to create persistent connections (sending users to the same instances).

This is a big improvement over scenario one.  However, in a software load balanced SAP Landscape, high availability is achieved only as long as the SAP Message Server is aware of a failure in the Dialogue Instance  (Portal, for example) or if there is a graceful shutdown.  Many failures are, unfortunately, not detected by SAP Message Server today, creating a potential for Web Dispatcher to send traffic to Dialogue Instances that are down.  There is also the issue of Web Dispatcher as a single point of failure itself.  There are no out-of-the-box solutions for making software load balancers highly available.

To summarize, web Dispatcher supports HTTP, cookie load balancing and persistence as well as static round robin.  Web Dispatcher can also make decisions on routing based on the client's IP Address and finally, it can offload SSL.

SAP with BIG-IP Local Traffic Manager is the hardware high availability and performance enhancing solution to the SAP scaling problem.  In a hardware load balanced SAP Landscape, high availability, security, acceleration, health monitoring and intelligent session persistence can be achieved.  When an SAP client makes a direct connection with F5's BIG-IP Local Traffic Manager, the user is intelligently load balanced to only the back-end services that are available at that very moment through a health monitor that passively tests login and database.

BIG-IP Local Traffic Manager, out of the box, can also be configured to optimize the transport protocol between the client and the server, it can make intelligent persistence decisions based on cookies, it can off-load SSL, and most importantly, using web application server health monitoring, traffic can be guaranteed to be delivered.  With advanced scripting abilities, for e-commerce or other "five nines" environments, requests can even be re-submitted if a server crashes mid-transaction, without causing the user to re-submit their data or to lose any work. BIG-IP also has a variety of out-of-the-box high availability solutions that remove it as a single point of failure.

Both software and hardware solutions provide significant enhancements in contrast to a direct deployment with no load balancing or high availability.  When making the decision between hardware and software, SAP and F5 recommend that you frame the decision in the following way.  Ask about acquisition costs, maintenance costs, stability, performance, range of functions, high availability, security, configuration complexity, integration into your existing landscape and content caching.

I was excited to see our press release coverage today about F5 Networks and Bytemobile's ability to scale T-Mobile's 3G Network Capacity with our VIPRION product.  The article mentions the "explosive" growth of data  on mobile networks and how data network capacity has to be scaled to handle the tremendous growth.  As a sentence, this all makes sense, and most networking professionals wouldn't give it a second thought; it's a data network and it needs to be scaled.

On the other side of this datacenter-centric story is the world we see around us from day-to-day.  Most professional have some sort of mobile data device these days, from Apple iPhones, to Google Android to Blackberry devices, and no one is shy about using the mobile data services they come with.  The missing connection for many people I talk to casually about mobile data is where exactly their data goes, in other words, what is the path of the data and where does it run into limits? 

At home and work, our email, web pages and videos all take generally the same path.  This is not so true with mobile networks.  On mobile networks, Text messages (SMS) ride an out-of-band portion of the voice network (thus the 160 byte limitation), Blackberry email messages involve additional hops through RIM servers and involve their protocols as well, data such as web pages and other TCP content have yet another path.  The over-the-air part is just one small part of this path.

I see the light-bulb usually go off when I ask people "Where do you think the data goes from your phone?" and they say "oh, I've never thought about that".   To take a very high level view, when we make a request for a web page or a YouTube video from our 3G device, let's consider the path our data has to take. 

The request goes from the mobile device via over-the-air radio frequency to a local cell site.  This is the 3G network, and this is only the first "hop".  Your mobile request has traversed the over-the-air network in the fastest available way today, but now what?  It doesn't go straight to YouTube.

From the cell site it's either relayed to another cell site or relayed back via high-speed lines to the carrier's network.   At this point, depending on the network, there might be another conversion process to get to the TCP network and the request finally enters the "data network".  In most networks, there has to be an authentication process to authorize the connection.  Once the request has entered the carrier's network, it gets processed by all the standard mechanisms (DNS requests, proxy servers, caches, etc). This datacenter is the single point that all data communication go through and carriers work day-and-night to optimize the speed and reliability of these datacenters.

Once you realize that every single subscriber to your carrier's network is going through that same network, you realize just how important it is, certainly equally important to the 3G network, if not more.  Finally, once the request is processed it returns to the mobile phone via that great and speedy 3G network. 

And this is why today's news is so exciting to me.  Seeing investments being made at the Mobile Carrier's datacenter is good news for everybody.  It's these investments that will ultimately enable the "feature XYZ" or background applications or many of the other wish list items we crave on our mobile devices.  When people see the performance numbers of our VIPRION, they sometimes ask who could possibly push that much traffic, in the mobile providers we have one of the many examples of an answer to that question.

A funny thing happened at the F5 booth during SAP's TechED 2008 last week (6,000+ SAP Developers in Las Vegas, Nevada for the entire week of September 8th, 2008), several groups separately asked me how they should handle the politics of administration and system management: "how do I grow my local traffic management solution, add web acceleration but not give up the control I have with my current software based solution?"

It's a great question for me because I'd drafted this post about User Roles and Partitions last month and I didn't know exactly how relevant it was to our users.  As I came to find out, it's quite relevant.

The concept of a User Role and Partition will be familiar to anyone who has worked with delegated administration.   Entitlement of responsibilities are restricted by job function, thus a developer can have one level of access and a network administrator, another.  F5 extends this paradigm to the concept of Partitions, which are segmentations based on application function.  For example, in a company with both SAP Portal and Microsoft Sharepoint, an SAP Portal group would have one partition and the Microsoft Sharepoint group would have another, neither with the ability to change the other's configuration.  All of this is done through the standard F5 web-based interface.

Agility is maintained because the subject matter experts on a particular application are allowed to execute their portion of the tasks directly and without having to delegate their expertise to another set of administrators (network admins).  Network administrators on the other hand can sleep well at night knowing that system administrators can't take their boxes down.

In the case of SAP, the world's leading provider of business software solutions, the question arises from customers who want to augment or move away from SAP's Web Dispatcher (SAP's software based technology for server selection and load balancing).  Web Dispatcher is often managed by application administrators and introducing F5 equipment seems like a loss of control and ultimately agility for these administrators that are used to doing load balancing configuration for themselves.

For those not familiar with SAP, the software is all about speed and time-to-delivery in the application world.  In many cases a new SAP application can be developed and deployed in just a few days.  But the network administration portion of it can take weeks to complete in the traditional IT organization where requests have to be filed, meetings held and configurations vetted out.  There are many good reasons for this traditional model, including preserving the overall uptime and stability of the network and not impacting other applications, but it's not the only way to achieve those goals.

To summarize all of this, let's walk through an example.  Here we have an organization with BIG-IP Local Traffic Manager (LTM)  and three stakeholder groups, network administrators, SAP Portal developers/administrators and SAP Business Intelligence developers/administrators.

Step 1 - Network administrators configure and prepare the network to suit the needs and requirements of the business.  Fail-over, overall monitoring and other critical functions are squared away by the network administrators who retain full administrator rights on the box and can view, edit and control any aspect of the units.

Step 2 - Two partitions are created, one for SAP Portal and one for SAP Business Intelligence, by the network administrators.

Step 3 - User accounts are created on BIG-IP LTM for each of the SAP development and administration groups and assigned to the appropriate partition.  Each group presents the list of individuals that need access.  A good rule of thumb is that for each role, there is one primary person and one backup.  Generic accounts, not tied to an individual, should be avoided for auditing and accountability reasons.

In this example, 8 accounts are created in total, 4 for each group. For Portal and Business Intelligence each, a Manager primary and secondary and an Operator primary and secondary.

Step 4 - The creation and negotiation of a Service Level Agreement (SLA) is highly recommended.  This step is critical to ensure standard operating practices across all three groups.  The agreement must spell out who is responsible for each part of the system, how each person can be reached and what the remedies will be for any outages.

Step 5 - Users are provided training through the use of Deployment Guides (e.g., SAP NetWeaver and Enterprise SOA (BIG-IP LTM, WebAccelerator, FirePass, WANJet)), through hands-on training by network administrators or through on-site training.  In reality, learning how to configure load balancing on BIG-IP LTM will come easily to any administrator who is already familiar with application concepts and/or is already running SAP's Web Dispatcher.

Finally, each user then takes over responsibility for the their own systems.  As a best practice, shared documentation should be available to all groups, to post experiences, schedule system-wide downtime and benefit from each other's learning processes.

Roles are well documented on the http://www.askf5.com site, as well as http://DevCentral.f5.com, but I will include a summary of the roles here.  Look to DevCentral particularly for great articles on iControl and Roles and Partitions. The roles are: 

Administrator - Complete access to all objects and configuration synchronization on redundant system pairs.

Manager - Create, modify and delete virtual servers, pools, pool members, nodes, custom profiles, custom monitors and iRules(tm); view all objects.

Application Editor - Modify nodes, pools, members and monitors; view all objects.

Application Security Policy Editor - Complete control of the Application Security Manager; view all objects (analogous to administrator for BigIP systems with ASM installed).

Operator - Enable and disable nodes and pool members; view all objects.

Guest - View all objects.

First post! (Okay, I had to get that cliche out of the way). 

I'm pleased to be joining the illustrious group of bloggers at F5 Networks.  I'm looking forward to adding information from a former customer perspective and also information about SAP and Adobe, which will be my focus at F5 Networks.

As a new employee and former customer, my viewpoint is a bit like a kid's in a candy store right now.  I've had the opportunity to delve into many of the nooks and crannies of our application delivery networking offerings and I'm just now seeing how so many of my previous projects could have been streamlined, accelerated, been deployed faster and cost thousands less with the right mix of application delivery networking.

Big-IP can do some amazing things, but as a Sys Admin I rarely had the time or opportunity to delve into all of the things it can do.  I hope to help break down some of this time barrier for those of you that are Sys Admins out there by sharing some real world experiences.  For the system architects out there, I hope that this information will help create solid solutions that cost a fraction of other alternatives, for your upcoming projects.

For me, on several projects, the ability to control and initiate URL rewriting at the load balancing layer created the ability to launch some projects on an amazingly accelerated timeframe.  In one project (not on Big-IP unfortunately), our outside analysts had put a timeframe of several years on one migration project; with the help of layer 7 networking, we were able to cut that down to six months.  That project was more than five years ago and the amazing thing is that the technology we used to pull that off looks downright primitive next to the capabilities of Big-IP Local Traffic Manager (LTM) and the modules that can reside on Big-IP LTM.

There are already piles of real-world implementation and performance data located in our solution center (link to the performance data section in particular:  http://www.f5.com/solution-center/white-papers/#app-performance).  I'm going to share some of my personal experiences with these results and implementations, especially around SAP and Adobe, but also around the Linux, Apache, MySQL, PHP/Python/Perl (LAMP) stack world. 

Not every system administrator and network engineer who falls in love with the full proxy architecture of Big-IP LTM can come join F5 Networks, immerse themselves in the technology and learn all the inner workings, hopefully my experiences can bridge that gap.

Some of the topics I'll be delving into will include:

* Bridging the gaps between system administrators and network engineers through the use of Big-IP LTM partitions and  profiles (delegated administration).  And more importantly, why network engineers can safely balance the risks of handing over management access to people outside their group,

* Using Big-IP LTM and Web Accelerator to reduce system administration tasks on web servers,

* Building a cheaper disaster recovery solution using Big-IP and solutions such as link load balancing, global traffic management (GTM) and WAN Acceleration,

* Utilizing iControl to automate system administration tasks,

* Utilizing iRules to reduce complex web site monitoring tasks such a Omniture tracking, or measuring page load speeds.

The list goes on, but the theme will remain the same: getting system administrators and network engineers to bridge the gap, for each to give a bit of control to ultimately create a much more efficient infrastructure with somewhat limitless architectural possibilities.