The debate on whether infrastructure devices, particularly those providing security, should fail open or closed is far from over.

One of our field system engineers, Aidan Clark, has some thoughts on scenarios in which you should fail open, and provides some compelling arguments for his view point. He's graciously allowed me to post his thoughts as his response seems to be irritating the comment gremlins.

The View from the Trenches


Now you already know this, but other readers might not. My standard disclaimer first: I am an F5 employee. I am an f5 Field Systems Engineer that specialises in security. I live in Australia (hence the linguistic fear of the letter “z” etc) Now that it is clear that I am not hiding anything, I am going to wax lyrical here for a moment and take you through some of the thoughts behind "fail open" scenarios that I have seen that are both valid and essential to successful security deployments for customers. I will also try, for the benefits of the readers, to map these reasons to the physical security analogies that illustrated your point so effectively in the original blog post.

It is important to understand that there is merit to both approaches and that there is no silver bullet here. All I seek to do here is illustrate some of the valid "fail open" cases that I have seen in my time both previously, when I was employed doing penetration testing, security consulting and auditing others' security against the illustrious "best practice" and now, in my role as a pre-sales engineer with f5.

Lets take the analogy of the shopfront: If the power goes out, do we want the door magically unlocking, and falling open. In the case of Bob's widgets, this is not likely the desired solution. Bob needs a security system that fails closed to protect his crystal widgets.

Lets now look at a different scenario:

I own an apartment building. My apartment building has hundreds of tenants living in it, and all of these tenants have apartments full of possessions that are extremely valuable to them. Now in this apartment building, there are elevators, stairs and doors that all provide security to the building. To access the lifts, you need a swipe pass, to open a door you need a valid electronic key, to access the car park, you need a pin number. This is a secure building, with a doorman and a guard. Now, lets look at what happens when a system fails. There is a fire, and the building power fails. Now when the power fails, do you want the fire doors failing into a locked position? Do you want the car park that holds the fire hydrant the fire brigade needs to attach their hoses to remain locked? The answer is definitely not.

What we have here, is a *valid* reason for these systems to fail open. When the security system was designed, it was designed to manage the risks that were greatest first. In this case, it defines that the *potential* cost of the human lives is *greater* than the cost of *all* of their combined possessions being stolen by looters.

We all look at this case, and can clearly see the need for a fail open solution.

If we map this scenario back to our IT Security configuration, there are valid reasons for some security solutions to fail open: when the risks of failing closed outweigh the benefits of the security the system provided to start with. Now I am not talking about scenarios where systems will allow unfettered access to millions of credit card numbers should a WAF fail, or where the personal information of the citizens of an entire city will be left lying unprotected should a firewall suffer a problem. I am talking about systems where the availability of a system outweighs *all* possible security risks in the event of a security system failure.

Think of the security of patient record systems in hospital where the risks of not being able to verify a patient's medication allergies might end up with a patient being made severely ill or worse, even dying if given the wrong medication. If the security product that protects the patient data systems fails, what is the fall-back scenario? Right now, hospitals typically have paper charts at the end of a patients bed or a sign on the wall over the patient with this critical information on it, but with the progression of IT systems into health care practices, will this always be the case?

Think of the security of an emergency services communications system. If the firewall for the IP Telephony system fails, do you really want your 000 operator having their phones go off the air? (Note: in Australia, 000 is what you call when need to contact police, fire or ambulance. I don't know what happens if you dial 911, but the Australian sense of humour in me thinks that you probably hear a recorded message that tells you that your parents must have let you watch too much American TV as a child before they divert the call to the real 000 operators to arrange a fast red truck full of water...)

Now the examples I have given have been severe ones. Everyone can see a clear requirement for availability over security when human lives are at risk. But the folks responsible for security and the folks responsible for the business don't always have such a clear factor shaping the decisions they make. Often, they are required to make risk based assessments on the likely impact of 2 issues, and decide which one is a greater risk overall. For many organisations, the rare chance that a critical component fails totally (you are deploying your WAF's in HA pairs aren't you?!) has a much lower level of risk to the business in the 4 hours it took for the vendor to get replacement units onsite than the impact to the business of a 4 hour outage on a key application. For this organisation, a fail closed approach would make the impact of such a failure *much* worse than the failure of the security solution, that was *only* installed in the first place to protect the business. For these companies, a solution that can get out of the way when it fails is a responsible security decision that can be made by rational and capable people whose job it is to run the business securely.

That's my $0.02 AUD anyway...


What do you think? Fail open? Fail closed? Situational?

Follow me on Twitter View Lori's profile on SlideShare AddThis Feed Button Bookmark and Share