Years ago I worked for a large-ish post production company. They had recently moved into a swanky new location and everything there was tailored to spec, including the server room. In norwegian we sometimes call a server room a ‘machine room’, this is relevant.
As a part of the server room spec, a dry fire suppression system was among the requirements.
The summer of the incident was particularly hot, and we experienced some trouble with our cooling, so a cooling technician was called to have a look. While he was working on the unit inside the server room, he made a mistake that caused all the cooling gas to dump into the room, triggering the fire extinguishers.
A dry fire system works by releasing an inert gas into a space to displace any oxygen, effectively choking any fire. I imagine this is usually done by some solenoids opening some canisters of gas and the room quickly, but gradually becomes oxygen free. Luckily, my boss at the time was present and he quickly got both himself and the tech to safety.
All good right? No. The contractor who constructed the new location had ordered and installed a system meant for maritime machine rooms, not the computer ‘machine room’ we had. In an environment filled with fuel and grease, you optimize towards filling the room with an inert gas as quickly as possible, and it turns out they use explosives to complete the task. In this room there were three canisters in the ceiling with fire shooting out of them, burning pellets to generate the inert gas. The gas and smoke from the canisters combined with the leaked cooling gas, and started condensing.
Into hydrochloric acid.
While all this was going on, all of the servers and workstations were happily humming along, sucking the now extremely corrosive atmosphere into themselves, making sure that every nook and cranny inside and outside got covered in a thin greasy film of acid.
The aftermath:
Mine and two colleagues’s summer break was cut short, as we were called in to do damage control. Ripping out and wiping hard drives clean was what we did all summer.
With external help we managed to recover all of the data.
One feature film was delayed a few weeks.
The insurance payout actually made the company a bit ahead financially.
As far as I know there’s still burn marks in the floor of the server room, from when flames shot out of the fire extinguishers.
Everyone involved now knows what a proper dry fire suppression system for a server room looks like.
The kicker is, the cooling was messed up because a fabric awning on the building had fallen down and was covering the air intake. If anyone had thought to check the roof this whole thing would have been avoided, and that server room would probably still have bombs attached to its ceiling.
I’m in awe about this. I work in compressed gasses and it’s pretty common knowledge in our industry that the environment dictates usage. I cannot believe they never consulted a gas specialist or used a completely inert gas that could have done the same thing.
Sounds perfectly normal for a construction/install team to me. “Maritime…doesn’t that mean like ocean or something?” “Hey the drawing says install it so I’m installing it.” “…yeah fair enough.”
Years ago I worked for a large-ish post production company. They had recently moved into a swanky new location and everything there was tailored to spec, including the server room. In norwegian we sometimes call a server room a ‘machine room’, this is relevant.
As a part of the server room spec, a dry fire suppression system was among the requirements.
The summer of the incident was particularly hot, and we experienced some trouble with our cooling, so a cooling technician was called to have a look. While he was working on the unit inside the server room, he made a mistake that caused all the cooling gas to dump into the room, triggering the fire extinguishers.
A dry fire system works by releasing an inert gas into a space to displace any oxygen, effectively choking any fire. I imagine this is usually done by some solenoids opening some canisters of gas and the room quickly, but gradually becomes oxygen free. Luckily, my boss at the time was present and he quickly got both himself and the tech to safety.
All good right? No. The contractor who constructed the new location had ordered and installed a system meant for maritime machine rooms, not the computer ‘machine room’ we had. In an environment filled with fuel and grease, you optimize towards filling the room with an inert gas as quickly as possible, and it turns out they use explosives to complete the task. In this room there were three canisters in the ceiling with fire shooting out of them, burning pellets to generate the inert gas. The gas and smoke from the canisters combined with the leaked cooling gas, and started condensing.
Into hydrochloric acid.
While all this was going on, all of the servers and workstations were happily humming along, sucking the now extremely corrosive atmosphere into themselves, making sure that every nook and cranny inside and outside got covered in a thin greasy film of acid.
The aftermath: Mine and two colleagues’s summer break was cut short, as we were called in to do damage control. Ripping out and wiping hard drives clean was what we did all summer. With external help we managed to recover all of the data. One feature film was delayed a few weeks. The insurance payout actually made the company a bit ahead financially. As far as I know there’s still burn marks in the floor of the server room, from when flames shot out of the fire extinguishers. Everyone involved now knows what a proper dry fire suppression system for a server room looks like.
The kicker is, the cooling was messed up because a fabric awning on the building had fallen down and was covering the air intake. If anyone had thought to check the roof this whole thing would have been avoided, and that server room would probably still have bombs attached to its ceiling.
I’m in awe about this. I work in compressed gasses and it’s pretty common knowledge in our industry that the environment dictates usage. I cannot believe they never consulted a gas specialist or used a completely inert gas that could have done the same thing.
Sounds perfectly normal for a construction/install team to me. “Maritime…doesn’t that mean like ocean or something?” “Hey the drawing says install it so I’m installing it.” “…yeah fair enough.”
Great story! Very well told. I can tell you must enjoy retelling it to newbies when they join the company :)
But wow, other than 2 summer breaks being cut short, it sounds like a good outcome. Especially considering no one was seriously hurt
I’m not with that company anymore, but given the right audience, ‘that time the server room blew up’ is a big hit.
It could have gone way worse. A stressful lesson and a good story is best case scenario outcome when stuff hits the fan.