Oh No! It’s Not An iPad! Enough of the Playbook Bashing Already.
It’s kind of funny the way the tech press will kick an incumbent around the block for perceived or imaginary shortcomings in their products. The Blackberry Playbook is a good example. You’d think that RIM went out and created a useless piece of garbage that was never going to see uptake no matter how large RIM’s enterprise market share was. The press and bloggers have leveled a whole slew of complaints against the Playbook that range from no out-of-the-box integration with mail servers to the power button not being convenient enough. Reading most of these articles and blogs leaves one with the distinct impression that the author is typing his article/post on his Mac while listening to his iPod and downloading pictures from his iPhone. Seriously. And I’m guessing for more than a few of the drooling naysayers that is exactly the case. Lori and I pre-ordered the 32 Gigabyte versions for each other for our anniversary, and they came a little over a week ago. By the time ours were in hand, people were making all sorts of ridiculous claims about how horrible the device was and why it was destined to fail. So we were naturally curious, since RIM doesn’t generally make a poor product, most of them are astounding. The device bears out as another quality RIM product, whining reviewers notwithstanding. The biggest complaint against the device is that it came without support for email built-in, which reviewers tried to pose as “not ready for prime time”. There’s a difference though, between a feature being left out to enhance overall experience and the entire device not being ready for prime time. A huge difference. And frankly, all this hand-wringing and screaming that the Playbook would never make it without this feature seems pretty silly in an age where a web browser gives you access to your email and calendar. The Apple Fanboys I know respond to that simple observation with “Yeah, but I like to be notified…” really? Did you intend to spend all of your time on a Playbook sized device? I doubt it, that sounds like a lame excuse to kick RIM. Other reviewers have said “it’s not intuitive! Put it in someone’s hands and they don’t know how to use it!” Well, if you try to use it like Windows that’s true, but it’s true of the iPad too – these devices have a different input paradigm than a full-blown computer does. It takes a minute or two and you figure out how to do everything you need to. Again, a lame attempt to undermine a solid product, in my opinion. I won’t grace any of the other flailing attempts to denigrate the device with an answer. I mean seriously, you’re complaining about the power button? It’s that thing with the industry standard on/off symbol on it, try pushing it. Not that it is all sunshine and unicorns… There are definitely things I don’t like about the Playbook. Like any other piece of complex equipment, it is better at some things than others. In the interest of giving you some real information instead of stammering attempts to undermine the product, here are the ugly bits. When both WLAN and Bluetooth are enabled, the battery doesn’t last as long as I’d like, but that’s one of those things you adjust to – my laptops, both work and home – don’t last as long as I’d like either. Tethering doesn’t bother me. I can get at mail and calendar in a browser, but wanted to try it out. I have a Blackberry Curve 8310, and I tried to tether the Playbook to it. No success. Now there were two possibilities here, I’m on AT&T who had some problems getting the tethering software for the Blackberry approved, and my Blackberry hadn’t been updated in forever. So I upgraded, then tried again. Nothing. I went out and found a work-around for AT&T’s network, and tried it. Nothing. So I went and did some research. It turns out that the version of the Blackberry OS required for the tethering software is not available on my phone… That was painful, the Playbook should just have a link you can click to find out if your phone can be used for tethering. There are a lot of applications out there, but alas, the cool, very productivity-enhancing applications like iPad’s “Cat Toy” and “Get me a beer” applications don’t exist yet for the Playbook. No doubt my work will suffer for the lack. Believe it or not, the spreadsheet included in the Playbook isn’t enough for me. I don’t need a full featured, astounding monstrosity like Excel, but I was hoping for a bit more than is supplied. Time will no doubt fix this and also provide integration with the major desktop spreadsheets, but for now, I can’t use it to do an inventory I was planning on because the spreadsheet application is weak, and the closest thing I’ve found to what I need is a database application that is more end-user than what I’m after. I think the Playbook is comparable to the other devices of its kind out there, and with Blackberry’s history of enterprise support it will no doubt continue to grow into exactly what I was looking for. I’m not too worried about all of the negative reviews because all they do is make the reviewer look like an idiot when they scream and holler about things that just aren’t mission-critical. What is a Playbook used for. Access when the laptop is a bit much to be booting up. Not for 100% of your workload. Maybe one day, but the entire space is too new for any decent percentage of workers to start using it as their primary tool. And F5, rocking our world yet again (we’re hiring… Check here), ordered us shiny new Blackberry Torches so that we can try out tethering. So maybe I’ll circle back if that turns out to be exceedingly cool or exceedingly painful. Yeah, I thought the iPad was very cool when it came out too… But we don’t allow Apple products into our house for personal reasons (much to our teenage daughter’s consternation). This is pretty much the same product, just with the polar opposite of press reviews.
Load Testing as a Service: A Look at Load Impact (beta)
I admit it. I’m a load / performance testing junkie. During my years with Network Computing I burned through any number of solutions designed to throw more traffic at products than money Congress is throwing at failed banks these days. And I do mean burned, as the last time I was in the lab there were no less than three non-functioning Spirent Avalanche systems that had given up the ghost after being forced to their absolute limits over years of use and abuse. When I received a note telling me about LoadImpact.com, a load testing as a service site, naturally I was intrigued. Generate load? From across the Internet? Inconceivable! That’s a no-no, after all, at least the kind of load that I’m used to generating. So I decided to check it out. Here’s the low down. PICK YOUR POISON Load Impact offers four service levels, with the lowest-capable load generating service being free (as in gratis). Pricing is per month with varying numbers of users, configuration and result management, and differences in what you can/cannot configure for any given test. I tested out the “Light” version, of course, but the site allows you to see all the configuration options – they’re just not enabled for modification if you aren’t using the right version. Load Impact does a nice job of laying out the various versions in an at-a-glance format. LOAD ‘EM UP First you’ve got to create a test. When configuring a test you have two options: Basic and Advanced. Basic requires the page you want to test. That’s it. Load Impact supports both HTTP and HTTPS (SSL) and can handle sites requiring HTTP Basic Auth. From there it uses defaults. The Advanced settings offer you a little more control in the Light version, allowing you to analyze the page and then, if you so desire, you can edit the “load script”. The load script is pretty simple: # Initial sleep to distribute clients sleep 500-5000 request 2 GET:http://www.nandgate.com/ request 2 GET:http://www.nandgate.com/base.css request 2 GET:http://www.nandgate.com/bigip/f5-logo.png request 2 GET:http://www.nandgate.com/donlori-w.jpg Don’t let the simplicity fool you. You can do quite a bit with the script to change the behavior of the clients and customize it to closely match the behavior of (or at least what you expect will be the behavior of) real visitors. The script format is documented in the FAQ. The format allows you to control the number of simultaneous connections that can be opened (in this case always 2 by default, which appears to try to emulate the behavior of IE6 and below) to better mimic the behavior of a “real” web browser. To simulate a POST method, you send parameters as though it was a GET request; it is assumed that the Load Impact client interprets the script and translates that into an HTTP compliant POST operation. The ability to edit the script allows you to simulate behavior rather than just the loading of a page as you can continue the script and have a client load multiple pages and sleep in between if desired. In more advanced (paid) versions you can also use a recorder to help you build out more complex scripts – a feature common in most commercial load testing products. Load Impact clients can also handle cookies, so if you’re testing a web application that uses cookies (and what web application today doesn’t?) you’re good to go. This is also good news for testing web applications that are load balanced and using persistence-based routing as such functionality is often implemented via cookies. The actual load balancing algorithms and functions for hosted and cloud-based applications are rarely made public (and too often no one asks) so supporting cookies ensures that the most common of those methods will be supported. You can change the number of clients and the “ramp up” (step) values, as long as they stay within the specified limits for the version you are using. The default for the Light (free) version is to start with 10, stop at 50, and ramp up by 10 clients. You cannot configure it to “burst” all the clients at one time, though you could start with 49, end at 50, and ramp by 1 to simulate a situation close enough to bursting. The load generators are using HTTP 1.1 and each test consists of several subtests: one per step. Clients use persistent connections throughout each subtest, but open new connections at the onset of each subtest. A nice option, though not configurable in the “Light” version, is “page view time”. In high-end load testing terms this is usually referred to as “think time”. “Page view time” is used to simulate the fact that a real client wouldn’t just load the page and leave. They’d visit for a while - hence the reason we call them visitors, right? You can also change the client bandwidth in paid versions, though a change affects all clients, not just a specified subset. Load Impact has plans to broaden this capability and allow more granular assignment of bandwidth in the future in a manner similar to “page view time” by allowing you to configure a range and then randomly assigning bandwidth values within that range to clients. An important note about Load Impact clients: (*) Note that Load Impact clients always behave as web browsers with an empty client-side cache. They will always load all objects on a page, never caching anything. This means that they are sometimes "heavier" than a real client would be. This means that if you were hoping to test the impact of a web acceleration solution that takes advantage of client-side caching (like F5’s WebAccelerator) that Load Impact will not be valuable in assessing the improvement over subsequent visits due to client-side caching. This also means Load Impact will not accurately test your web server configuration or help you assess the impact on the use of cache-control HTTP headers or HTML meta-tags. It will, however, help assess any improvements in performance that may be due to server-side caching or other optimization/acceleration techniques. This means Load Impact may be invaluable to you when assessing technology designed to help improve application performance before purchasing. The Light version will do well enough for assessing impact on performance and is free, so be sure to remember to include it on the list of options during a proof of concept run with new technology. WYSIWYG Once the test is running you’ll see the graph and data updated in real-time. A nice to have, as you don’t have visibility into the status of the clients, but not very interesting to watch as it tends to update slowly. The end result is a nice little graph depicting “Delay” in seconds as it relates to the number of clients. Initially you see only the summary, the total delay for the entire page. But you can drill into each of the file types and select up to three objects to be included in the graph. As you drill down and select different objects, a spreadsheet with delay as it relates to load is updated. In this case, choosing the base.css and two image files showed the delay in ms rather than in full seconds, as would be expected. You can export the data in CSV format, which should allow you to chart all the composite objects performance together to get a better view of what’s causing any problems you might see. When you’ve selected more than one object you can also play with the “min”, “max”, and “average” settings to dig around a bit more into the actual data that was generated. Load Impact saves one average delay value per client step, so if you ramped from 10-50 on a step of 10, you’ll see 5 distinct data points plotted, each representing the average of all clients at that step. Load Impact also notes that “Delay” is not the time it would take for a client (browser) to load the page. This value is the sum of all individual object load times. Please note that this is not the same as the time the page will load and render in a web browser. Because objects are loaded across several connections in parallel, the total sum of all individual load times will not be the same as the time a web browser takes to load a page. The current "test summary" metric is intended for optimization work, as it accurately shows changes in object load times. We have, however, realized that many people expect it to show page load times as experienced by a user, and therefore we are going to implement such a metric as well. It’s also important to note that the data collected is not normalized against intercontinental latency. Load Impact has plans to increase the number of load generators available in the future but right now there are two options: one located in Stockholm and one in Chicago. In the Light version all load is generated from Stockholm, which necessarily incurs intercontinental latency that may or may not be applicable to your application. So if you’re using the Light version and hosting somewhere other than Europe and expect most visitors to be outside Europe you may want to manually adjust the data collected by downloading the CSV version of the data. The next version of Load Impact, which is expected to be available in the next couple of weeks, will offer more granular performance metrics including “a new response time metric and a bandwidth usage metric.” Initially these will be summary only with per-object reporting in the future. Plans for TCP metrics like TTFB (Time to First Byte) and TTLB (Time to Last Byte) as well as IP metrics such as packet loss, delay, and jitter are also in the works. More data is always good for troubleshooting exactly where application performance problems are occurring: at the network or application layers. RUN IT AGAIN SAM You can save and re-run a number of tests based on the version you choose, which is a nice way to test sites under low traffic, medium traffic, and high traffic volumes. With the free service version the limit of 50 users makes the definitions of low-medium-high somewhat irrelevant, but even at 5000 concurrent users with the Advanced version Load Impact is definitely not going to be able to generate a high enough load to be considered a mainstay for high-volume site testing. You’ll also want to consider that the transactions generated by the client are not synthetic unless you use parameters you will recognize as synthetic and can account for them. The traffic generated by Load Impact is real, and therefore it will register as real hits for purposes of advertising, campaign tracking, and any other web site measurement based on visitors you may have in place. If you are using script-based analytics or advertising you can relax; while scripts are downloaded they are not executed or evaluated by the load generator clients. Be careful about the potential to trigger security mechanisms if you’re testing a hosted or cloud-based application with Load Impact, or even your own web application. At this time all clients appear to be arriving from the same IP address, and depending on the configuration of security systems this could trigger alarms and notifications of an attack and potentially blacklist (usually temporarily) the load generator’s IP address. Of course if you’re implementing a security solution or are trying to evaluate rate shaping/QoS functionality you may want to consider using Load Impact to generate some load to evaluate the solution’s effectiveness as well. To address the possibility of overwhelming third party sites hosting scripts commonly used by web applications Load Impact limits the number of times a resource can be loaded by the entire system in a 24 hour period. While this doesn’t completely mitigate the potential for tripping up security infrastructure, it is a good idea for Load Impact to impose such a limit in the interests of being a good Internet citizen. The FAQ and forums have a lot of good questions and answers as well as tips to help you build out a test scenario. While the help / information available within the actual application are a bit light, you should be able to find what you need in the forums. There aren’t enough knobs to make me truly happy yet (I am a load testing junkie, after all) but the roadmap for additional features and metrics is definitely on track to get me there. For most folks this is a good way to start out testing and for many it is probably more than enough than they’ll ever need. For developers trying out their application in a cloud or hosted environment, this is definitely a good way to get a handle on how it’s going to perform. Even for developers wondering how their locally hosted web application is performing might want to consider giving Load Impact a whirl; it provides the perspective of a user outside the local environment which is something not always available to developers using in-house solutions. It’s free to try out, so go on over and give it a run yourself.
