We run a lot of servers out of our house. Six publicly accessible web and mail servers and numerous private servers for things like file sharing, etc.

Since 1999 or 2000 we have been using a business class UPS to allow us to shut down the systems in our racks cleanly. It just makes sense to do so, we could lose dozens of hours putting things back together after an outage if we just let the machines drop.

But we don't have SMP Notifications turned on for the UPS. They were new for UPS systems at the time we put ours in, and it doesn't work real well, so we don't use it.

Friday night, a telephone pole was taken down by a driver that left the road, cutting power to our entire neighborhood. In the eerie silence that is the modern house in the country without power, the UPS alarm buzzer was loud, so I managed to get everything shut down cleanly.

When power came back up, everything came back on just fine. Problem resolved, right? One little niggle with one of our web servers and the BIG-IP (the routing table hadn't been saved after we put the machine behind the BIG-IP), and we were 100% functional.

But Sunday night, everything suddenly dropped. In the noise that is the modern house with system fans and environment fans, the hum of electronics, and music playing, we didn't notice the alarm from the UPS going off until our network went down.

The sites we host see a combined total of thousands of hits an hour. Downtime is not our friend.

So we went to investigate. At first, the UPS appeared dead. Having used it since 1999, and it having served us faithfully, this was painful, but not terribly surprising. But further research showed that while the breaker had been tripped, the UPS didn't appear dead, just juiced out. So we plugged something into the outlet (a printer), and it ran fine.

We have a home-made plug converter that we've been using since we got the UPS:

PowerSurge1

Nothing snazzy, just a power cord to an outlet so that the UPS could be run on a dedicated circuit with a "normal" plug on it.

Since this outlet allows us to plug pretty much anything in our racks into it, we plugged that same poor old HP Printer into it, and... Nothing. At least we had the problem. So we inventively applied power strips to the problem to get our servers back up and running - and again everything came up just fine.

We took the cable up stairs to our work area and took it apart to see what went wrong. Imagine our surprise when we found this...

PowerSurge3

One of the connectors took a surge so strong during the outage that it melted not only the plastic around the connector screw, but melted the wire clean through also.

What saved us? Our UPS. Without it, that circuit would have blasted this level of power direct to our Cisco switches, our BIG-IPs, and our servers. While we do backups, we do not back every little subdirectory up, meaning we'd have been down for months repairing things. As it was, the UPS not only took the shock, but appears to be in working order. Thank goodness we have a commercial grade UPS.

So I'll rebuild our little plug converter, and we'll move everything back to the UPS, and we'll be grateful that we lost absolutely nothing but a few hours while our picture NAS and our document SAN both rebuilt after the outage.

Until next time,

Don.