iControl Ruby Library Updated to v11.0.0.1

With BIG-IP version 11 now out in the wild, it is time for another iControl Ruby Library update. This new version includes all the hooks you’ll need to take advantage of v11’s awesome new features. Head on over to the iControl Ruby Library forum and pick it up in the Downloads section.

Changelog

• WSDLs updated for version 111
• New examples now packaged with the library
• Tested for backward compatibility with version 10

iControl Ruby Library Updated to v10.2.0.2

Our Ruby iControl Library has been cooking here on DevCentral for six months now. We’ve received a lot of feedback on our first release from the DevCentral community. We’ve taken what we’ve learned from the alpha release, made a few improvements, and are now releasing our first stable version: 10.2.0.2.

Here are a few of the fixes and enhancements we made with this release:

• annoying SSL cert-depth message is now gone for good; you should no longer see the ‘at depth 0 - 18: self signed certificate’  error
• fixed installation issues associated with letters in the 'verson' section of gemspec
• updated iControl WSDLs

The new version should take precedence within Ruby Gems, but you probably want to remove the previous version just to prevent confusion. Here are the steps to update your version:

1. Download the iControl Ruby Library Gem
2. Uninstall the previous version

   gem uninstall f5-icontrol

3. Install the new version

   gem install f5-icontrol-10.2.0.2.gem

4. Verify the installed version

   % gem list -d | grep f5

   => f5-icontrol (10.2.0.2)
         Homepage: http://devcentral.f5.com

5. Test your new installation by running the example code

   cd /var/lib/gems/1.8/gems/f5-icontrol-10.2.0.2/examples (location may vary based on system)

   ./get-version.rb 192.168.1.245 admin admin

      => BIG-IP_v10.2.1

If everything went as planned, you should see your BIG-IP version without any SSL certificate errors. Another good test is to run the SSL certificate report (ssl-certificate-report.rb) against your BIG-IP. It is non-invasive and will test a number of compatibilities. Until next time, happy coding.

Installing LTM VE on VMWare ESXi

We’ve had a few requests for a tutorial detailing the installation of F5’s new Local Traffic Manager Virtual Edition on VMWare vSphere Hypervisor (ESXi). Now we have one, sort of. We decided the best way to do it was to record it as a video tutorial. If you can stand my stuttering for ten minutes, you should have a working LTM VE instance in no time.

First things first, you’ll need to get ESXi up and running before you can move on to the LTM VE portion. We’re not the experts on ESXi, so I thought it best to hand this over to VMWare. Their documentation and hardware compatibility guides can be found here. You’ll need to set up a free account with VMWare and download ESXi here.

Once you’ve installed ESXi, you’ve got the vSphere Client up and connected to ESXi, you’re ready to proceed with the LTM VE installation. Watch the video below (switch to HD in order to read the text) for a walkthrough of the LTM VE install:

At the completion of this procedure, you should have a fully functional LTM VE instance. You may run into a few snags depending on how your network interfaces are configured, but we’ve got you covered. Jason did a fantastic job of detailing how ESXi networks are configured in this blog post. Spend the time to read the post, it will be well worth your time and will explain many of the nuances of configuring ESXi’s networks.

We hope you enjoyed this video tutorial and it helps you get LTM VE running in your environment. Be sure and stop by our F5/VMWare Solutions forum for more dialog on F5 and VMWare integration.

A Few Of My Favorite (Not So Common) *nix tools

Every *nix engineer has a set of tools in their box. They make our lives and occupation easier, more efficient, and hopefully more enjoyable. We all use the big ones: Vim/Emacs, tcpdump, top, netstat, dig, telnet, SSH, etc., but there are a few others that are newer or don’t receive the love that the staples do. A lot of the commands that wind up in repertoire have to do with our mentor(s) and what they used, but we all discover new tools to facilitate or simplify a task. Here’s my list:

htop

I originally came across htop in an issue of the Linux Journal a few years ago. I fell in love with this little tool almost immediately. It takes the venerable, but dated top command and jazzes it up. Htop makes use of the ncurses library, which provides a terminal-based user interface that is far superior to that of the classic top. Htop provides the ability to sort by metric, kill processes, switch to “tree” views, customize available meters and their order, and change the color scheme to name a few. Htop is a welcome departure from the top command.

dstat

My first introduction to Dag Wieers was his immense repository of custom-built Red Hat packages. I was supporting Red Hat Enterprise Linux at a university at the time. If we needed anything other than what Red Hat supported, there were only a few options: built the package myself, yum (which ironically enough was installed using packages built by Dag), or get the .rpm from Dag. At the end of the day, Dag’s packages were the best of the available options for us.

A few years passed and one of my coworkers introduced me to dstat. Dstat combines the functionalities of vmstat, iostat, and ifstat. It provides the ability to compare any system resource to another. If you would like to see how network traffic is affecting interrupts or disk I/O, they can be displayed next to each other, color-coded to reflect their intensity, and updated at a given time interval. Custom resource statistics can be output to a text file for later review or archiving if need be. While it would seem that htop and dstat would overlap, I use them for different purposes and both have their own place in my toolbox.

stat

I’m sure you’re asking, “Stat? Why stat?” Yes, stat is about as ubiquitous as an command in the *nix world, but it was one of those useful commands that I never started using until much later in my career. I was unaware of stat until someone pointed out to me that I was wasting time by parsing directory listings in my scripts and there was a better way to obtain this information. I now use stat all the time for querying information about files or filesystems, sometimes directly in place of ls. 

bc

Bc is my go-to command line calculator. Once again, bc is as old as I am, but I still love it nonetheless. It can do everything the usual GUI-based scientific calculators can do and more. It has a syntax very similar to that of C and can perform basic arbitrary precision calculations out the door with the ability to load additional math libraries as needed. If you’re not already a bc user, give it a shot, but remember that length and scale are not the same.

irb

Last, but not least, is a shameless plug for my beloved Ruby: irb, the Interactive Ruby Shell. I’ll say this again, I love Ruby. I love it’s simplicity, syntax, and extensibility. Irb provides a valuable resource for experimenting, developing, and testing code. Those of us that are familiar with Ruby and use it regularly can also do complex systems administration tasks on the fly without having to use other Unix shells. If you’re evaluating or already using Ruby, but haven’t tried irb, I highly recommend it.

Clean Up Those WebAccelerator Performance Reports

I’ve participated in and inherited quite a few WA (WebAccelerator) deployments during my tenure at F5. One pet peeve I have and I see it again and again: dirty performance reports! Take a look at the image below and you’ll notice that solid red line section of the bar graph at the bottom of the chart. Those are errors and they are too consistent to be user produced.

A small baseline of errors is expected as there is always someone out there trying to forcefully browse the site or possibly a broken link, etc. These non-user-produced errors in the chart above however are almost always caused by an incorrect HTTP monitor. In a lot of cases, an administrator will point a plain HTTP monitor from their LTM at a WA virtual server and walk away. We all know that WA uses hostnames to determine the appropriate acceleration policy to apply to the incoming traffic. Without a host header the WA will respond with a HTTP 400 (Bad Request) and the monitor will be marked up, because it did receive a “valid response”. This has two consequences: not knowing the true health of our application and our performance reports will be marred by lots of errors.

In order to remedy this, we need to build a more intelligent HTTP monitor that provides a valid host header. I would also recommend a more intelligent monitor than a plain HTTP, but more on that next time. In order to do that, we need to login to our LTM user interface and navigate to “Local Traffic > Monitors > Create”. Name your monitor something intuitive (mysite.example.com_http), you can change the interval and timeout parameters to suit your needs, now here’s where the magic happens: the send string.

GET /index.html HTTP/1.1\r\nHost: mysite.example.com\r\nConnection: Close\r\n\r\n

You can set the receive string to something you expect to receive from index.html every time or leave it blank. Keep in mind that by leaving it blank, you still have the same case as mentioned previously whereby any HTTP response is considered valid. Either way, the errors will be removed from the performance charts making things a lot prettier. If all went well, your performance reports should start to look more like the one below.

BitTorrent For Distributed Deployments

About 3 years ago I was working at a small startup and had a conversation with one of my coworkers about using BitTorrent for distributed deployments in our datacenter. We got on the whiteboard, drew up some preliminary ideas, and then got our boss in the room to show him our idea. Being the “dark days” of BitTorrent, most of you can probably guess what came next. He said absolutely not, we are a legitimate company doing legitimate business and if anyone found out we were running BitTorrent internally it could tarnish our image. While that blocked us from rolling it out in our environment, we honestly didn’t really need that much throughput. We had a very heterogeneous environment where there were only double digits of any particular server class therefore we stayed with the central repository distribution model.

Fast forward a few years, BitTorrent becomes a staple in the open source software distribution arena. Almost any Linux distribution imaginable can be had via BitTorrent these days and at a fraction of the cost of what it would cost to host them centrally. I would call this the “transitional period” when BitTorrent started to receive something other than negative press.

I hadn’t really heard of anyone using BitTorrent in the capacity that we had originally discussed until a few weeks ago when Twitter’s Engineering group posted a blog on their implementation. They were using a single Git server to host all of their software packages and then instructing their application servers to all download from this one server. This was sufficient in the beginning, but as we all know Twitter has grown by leaps and bounds since its inception in 2006. Hitting a single Git server with thousands of application servers just didn’t work. Enter their new system of distributed deployments: Murder.

Murder has nothing to do with the nightly news, it is also defined as a “flock of crows,” which segues nicely into Twitter’s bird theme. It was written by Larry Gagea who is an infrastructure engineer for Twitter. Murder is deployed using Python and Capistrano. Python doing the heavy lifting for the BitTorrent traffic and Capistrano instructing the application servers. Given that BitTorrent was originally designed to run on the Internet with limited throughput and relatively high latencies, there had to be some modifications to the standard BitTorrent options. They decreased the timeouts on chunk transfers in order to not have machines hang waiting for a chunk that may not be there. Encryption was not needed to bypass ISP gateway, so it was disabled to reduce the CPU overhead. Distributed Hash Tables were also turned off in order to encourage a more linear distribution, which is discussed in length in Larry’s presentation. Lastly, UPnP was disabled as it was not needed for NAT traversal and makes traffic patterns less predictable.

If you are interested in playing with Murder, it can be downloaded from GitHub: http://github.com/lg. If you have the time, I would also encourage you to watch Larry’s half hour talk on the system. He outlines why they did what they did and what tools are available to build a similar distributed deployment system that isn’t Ruby or Python-centric. It is very cool to see such a neat and innovative protocol finally get some good press after all these years.

Securing the Corporate Intranet with Access Policy Manager

Fire doors are used to minimize damage to a structure during a fire. In the event of a fire, a central monitoring system will trigger the release of the doors isolating flames and smoke at the epicenter of the disaster while protecting adjacent sectors. While most of our office buildings are built in this fashion, corporate IT environments are largely built like multi-story stick-built mansions with no segregation mechanisms. Someone playing with fire in the tool shed can bring the entire house to the foundation. Does your infrastructure have any such safety net?

When engineers hear ‘VPN’ or access management, they think of a device that sits on the edge between the corporate intranet and the Internet. All too often, internal corporate traffic is allowed to bypass these barriers and access corporate resources directly. While there may be authentication mechanisms protecting these services, it is almost impossible to secure them all without placing some barrier between them and the users. Enter APM.

APM is well suited as a demarcation point not only for access from the Internet, but also from the corporate user space. In any organization, users are inherently your largest security risk. Whether a user is accessing the intranet from a non-corporate sponsored machine from their house or a workstation in their cubicle, they should be expected to adhere to the same security standards.

In a “flat” security model, all corporate users can see any resource (even if it is just a login page) whether or not they meet the authorization requirements. Is there really any reason why John in product development should ever be able to see the corporate payroll system? Even if he can’t access the information, he can still see it. That means if John’s workstation is compromised, an attacker may be able to use his access to execute an exploit against this system thereby gaining access to precious corporate information. This may be a far-fetched example, but such things have happened and in many cases they’ve made major news headlines. This is not the kind of press any large corporation wants.

In terms of physical security, we can use APM to ensure that every packet that is exchanged between our user subnets and the secure corporate space is encrypted. This protects any insecure data from being compromised by an eaves dropping attack and does so at a fraction of the cost of fiber.

If properly implemented, Access Policy Manager should not present any hurdles to the end user. In fact, with single sign-on (SSO) credential mapping features, APM should actually improve user experience. This means when Joan in human resources logs into her Edge Client in the morning and receives her session for the day, APM will cache her credentials. She won’t have to type her credentials every time she access the payroll system because APM will enter them for her.

The end result of securing your corporate resources with APM is increased security for your environment and improved accessibility for your users. While nothing can protect against every potential security breach, APM can go a long way in ensuring that a single attack will not bring your organization to its knees.

Thoughts On Hostname Nomenclature

There are few topics as controversial in the world of systems administration as hostname nomenclature. I'm not talking about the merits of using Disney characters versus the names of stars. I'm discussing how to structure a host's name in such a way that it requires a minimal amount of effort to decipher its purpose. I am going to present a naming convention that has worked very well for me in the past in both Windows and *nix environments. This nomenclature should reflect four key pieces of information about our host: location, environment, purpose, and a unique identifier. This hostname will be 15 characters in length with each field consisting of 3 letters or numbers separated by hyphens: loc-env-pur-001.

This is not just an issue for the systems administrator, this is also a matter of department policy. It is easy to follow a naming convention when there are one or two administrators in agreement, but when there are 50 in your organization it gets a little more tricky. Left to their own vices, the installers will adhere to whatever names they find convenient. If a hostname convention isn't actively reviewed by upper management and required to go through an approval process, it will just turn out to be a mess. There needn't be hours of meetings on this topic, but a basic thumbs up or down from a director can make everyone's life easier in the long run.

A hostname should reflect the physical location of a piece of hardware. This element is often omitted altogether in smaller environments that have a single office. It may seem redundant to include this if you only operate one location, but will be a pain to go back and correct when you open another office. Do yourself a favor by adding this from the beginning. You'll be a lot happier when your manager brings you the good news about the new office you are opening in London.

As a rule of thumb, I use the nearest airport's three letter code. This is easy to stick to and minimizes arguing as to how some localities should be abbreviated. Seattle-Tacoma International has a three letter abbreviation of 'SEA', so the first three letters of a machine name located in Seattle would be 'sea'. If we opened another datacenter in Portland, those hostnames would start with 'pdx'.

We use the environment field to indicate whether this database server is the MySQL server in corporate, development, test, etc. By placing the environment in the hostname we can reduce mistakes where we "thought we were doing that in test." We've all done it and it can turn a normal morning into a firestorm.

Some of the more common environments I have encountered in my tenure have been: corporate, production (also seen it called delivery), development, test, stage, stress, and operations. With that in mind, all of my production hardware in Portland will start with 'pdx-prd'. As the name starts to take shape, you should notice how easy it is to grep a zone file and give your boss a list of all the test machines you have at the satellite development office in Dublin.

Whether the machine will be used to host a web server, an e-mail relay, or a corporate database, it needs to be displayed in the hostname. I want to know what the machine is doing when I look at my Remote Desktop toolbar or my *nix prompt. I don't want to have to look at the process list, then search through the service's configuration to figure out what it is doing.

The list of possible purposes or services is too long to list here, but we can take a common service to demonstrate my point. We are getting ready to roll out some web servers at our new production datacenter in Boston. Our closest airport in Logan International and Director Tom says we're using 'prd' and 'web' for production web servers. Our new web servers are all going to start with the same prefix, 'bos-prd-web'. Done deal, there's not too much to argue about here.

This is self-explanatory, but we need a way to differentiate all of our web servers in Boston. I do this by using a base 10 integer padded with zeros. This works well unless we have more than 999 of a particular host type in one environment at a single location. The easy fix for this would be to substitute a hexadecimal integer in place of the decimal integer. This will allow you to have up to 4095 machines in that particular server class.

A My 40th development SQL server at our main campus in Seattle will be named 'sea-dev-sql-040'.

So now comes the question: "but we have all that information already broken out by subdomain, why would I put it in the hostname?" The default number of searchable domains on a Linux host is 6. This can be changed to a different value in resolv.h, but is an unacceptable expectation for the casual user. On the flip side, as an administrator I don't want to spend all day typing the fully qualified domain names of my servers. Searching one or a couple domains makes life a lot easier.

As for the 15 character maximum length, this is a limitation of NetBIOS. NetBios usage ceased with the releases of Vista, Server 2008, and all subsequent releases of Windows. While this is becoming less of a limitation with newer operating systems, I still like to provide backward compatibility for those users who may not be using the most current version.

This is by no means the definitive guide to hostname nomenclature. This is merely my personal opinion. There are few things that irritate me more than walking into a new position where someone has named everything from switches to web clusters after Star Wars characters. Don't get me wrong, I like Star Wars as much as the next guy, but please don't name a database server 'Chewie'.

I hope you found this informative and will take some of it to heart while architecting your infrastructure. Future systems administrators will thank you.

UNetbootin

One of my favorite new tools is UNetbootin. UNetbootin is a standalone binary, which runs on both Linux and Windows, that will create a bootable USB drive just about anything from FreeBSD to Gentoo to Ubuntu. It will even go out and retrieve the ISO images from the respective repository and build the bootable drive on the fly. Just select ‘Ubuntu – 10.04_Live’, go grab a cup of coffee, and you’ll be ready to rock and roll when you get back.

Aside from convenience, the other reason I really like this little program is that I have stacks of CDs laying around that I used once to do an install or just needed a live CD, then never used them again. Being a person that hates waste and extra stuff on my desk, this is annoying. USB flash drives are dirt cheap now ($10 for 4GB at the time of this writing) making this a really attractive alternative to CD-Rs. Give it a shot and tell the folks over at UNetbootin to keep up the good work.

Hello World!

Hi everyone, my name is George Watkins and I am the newest member of the DevCentral team. I have been at F5 for the past two years in our IT department as a Unix Systems Administrator/BIG-IP Engineer. During that time period I worked extensively with the F5 product suite including, but not limited to LTM, GTM, ASM, WAM, and APM. Prior to joining F5, I was a Systems Administrator at my alma mater, The University of Oklahoma, and Zillow.com. I hope my experience as both a customer and administrator will be of benefit to the DevCentral community.

When I’m not heads down in the “geeky stuff” here at F5, I enjoy camping, hiking, brewing tasty beer, and like many other F5ers, cycling. If you’re ever looking for a killer IPA recipe or a good place to snowshoe in the northwest drop me a message and I’ll do my best to point you in the right direction.

I look forward to chatting with each and every one of you, so don’t be afraid to say hello. Until then… cheers!