Bare Metal Blog. Mean Time Between Failures
#BareMetalBlog #F5 MTBF has meaning well beyond storage.
If you are new to the Bare Metal Blog series, find them all here
When assembling a model – any model, from a highly detailed functional replica of an engine to a mass produced plastic model of an airplane – there are several places where things can go wrong. The final product is only as good as the model kit, the glue used, the tools used, and the skill of the craftsman. I’ve seen the same exact model assembled and painted by two different people that look completely different, simply because of the array of variables and how they interact.
This is true of high tech equipment also, and like modeling, it is often overlooked. Interestingly, in my entire IT career, MTBF has only been a measure that meant a ton in two circumstances: When designing hardware and scoping the parts to go in it, and when talking about storage. In all other endeavors, MTBF if mentioned was a side note.
And yet it matters. It can matter a lot. Like most hardware companies (because we spec our own parts and monitor our own quality), we track MTBF both computed from the sum of the parts with average environmental considerations, and actual tracking based upon support cases involving hardware and RMAs. For us, knowing helps us improve quality. For customers, knowing helps gauge the bounds of useful life for the equipment being purchased. Of course, MTBF is a mean, not a fact, and it is entirely possible for a device to last much longer than its MTBF, in fact the fact that it is a mean kind of implies that roughly half of the devices out there will last longer. But it’s the mean, not the median, and most IT shops do not want to plan like a device will last well beyond its MTBF value. MTBF can offer a bit of guidance when it is fairly calculated, and another tool in the evaluation toolbox never hurt an IT shop.
As mentioned earlier in this series, F5 sets quality standards for suppliers to meet, if they wish to continue supplying. This allows a bit better control over MTBF than doing something like “lowest bidder” or similar procurement, simply because the standards set include the quality of parts used, which all rolls into the MTBF calculations – and more importantly for most IT shops, the MTBF reality. While MTBF is a complex set of equations, you can generalize to “the MTBF of a device is as low as or lower than the MTBF of its weakest part”. That means supplier quality standards matter in a very real way. I had a RAID array fail on me once – several drives down all at the same time. The array vendor had to count that as a failure, since RAID no longer worked (thank heavens for backups!), but the failure was on the part of one of their suppliers. That’s how it is in the manufacturing world whomevers’ name is on the box gets the bad rep for quality, regardless of whose handiwork was slipshod. That is why F5’s non-stop quality monitoring program (devices are tested from before release until EOL is announced) matters a lot. It’s also why quality standards for parts suppliers matter more then getting the absolute cheapest part, as some manufacturers are wont to do.
I will not replicate our entire knowledge base article here, if you have an ask.f5.com account, you can click here to read it. I’ll just summarize and pull bits out for the readers’ enjoyment.
F5 gear runs the gauntlet from entry level to massive blade systems. As such, MTBF varies from device to device. The worst calculated MTBF for an F5 device is over three years. And our quality team tells me that the calculated value is far lower than the real-life-experience value they get from watching returns and such. The best calculated MTBF is over 21 years. It’s a rare piece of computer gear that is used that long, but Lori and I have got some pretty old F5 gear that’s still clipping away like it was new, so no surprises there. Most F5 devices fall somewhere in between.
Why the large variance in MTBFs if we control for quality? A valid question. The fact is that it is not all about the quality of parts. Airflow inside the device, number of redundant parts, number of removable parts… there are a zillion other things that go into MTBF, and they all tend to get better as the device gets physically larger. Entry level devices are small, restricting airflow and cutting down on available space for redundant power supplies, etc. While the top end blade servers have room for all of that, and since cards are replaceable, tend to less failures. You will find a similar spread with any other vendor that covers such a wide range of hardware. And all of those numbers are likely to beat out a COTS server running a software product.
So when looking at any electronic gear, ask about MTBF. Alone it simply gives you insight into the priorities for the device you’re looking at, when combined with the MTBF numbers from several different devices (the same manufacturer or multiple), it gives you an idea of what you are buying in terms of quality. Of course with a large chunk of any given appliance handled in software, MTBF is not as meaningful as it once was, but it is still the underlying bedrock for that software to run on.