Precautions for the IT serveroom
Hi All,
I know its been a long while since I posted in here, but I have been flat fucking stick with my new job and shit. Anyway, I took time out today to lecture on the importance of thinking through your battery backup systems for a server room, after an incident occured at work last night.
The power in our office neighborhood failed for some hours last night, but we have a UPS powered off another UPS, which provides ample runtime for just about everything in the server room. This is approximatley 25 - 30 machines of varying sizes, plus the secondary UPS itself. (The primary UPS is downstairs).
The UPS worked flawlessly, and kept all of the systems running for several hours before giving out. The problem was that no one was there to shut anything down gracefully and more importantly, A UPS DOES NOT RUN THE AIRCON...
When I came into the office (which is almost totally empty since everyone else has relocated), I could smell rotten eggs, which I assumed was some kind of sewerage or plumbing problem. As I got closer to the server room though, the smell got worse.
I noticed all of the backup drive lights were in a state that suggests they were not finished backup (bad sign) and the UPS was beeping with a red error light.
The situation was that the servers had continued to run for a couple of hours with no airconditioning, and the room had become so hot, the UPS inside the server room batteries had balooned out to a point where we had to pull apart the UPS chassis to remove the modules.. two of the batteries had split their casing and the smell was because of the acid... and let me tell you, it stank like crap.
The moral of the story is, make sure your bloody servers shut down when the UPS kicks in for more then a few minutes.. this could have gone a lot worse then it did.
Cheers,
Darrkon
I know its been a long while since I posted in here, but I have been flat fucking stick with my new job and shit. Anyway, I took time out today to lecture on the importance of thinking through your battery backup systems for a server room, after an incident occured at work last night.
The power in our office neighborhood failed for some hours last night, but we have a UPS powered off another UPS, which provides ample runtime for just about everything in the server room. This is approximatley 25 - 30 machines of varying sizes, plus the secondary UPS itself. (The primary UPS is downstairs).
The UPS worked flawlessly, and kept all of the systems running for several hours before giving out. The problem was that no one was there to shut anything down gracefully and more importantly, A UPS DOES NOT RUN THE AIRCON...
When I came into the office (which is almost totally empty since everyone else has relocated), I could smell rotten eggs, which I assumed was some kind of sewerage or plumbing problem. As I got closer to the server room though, the smell got worse.
I noticed all of the backup drive lights were in a state that suggests they were not finished backup (bad sign) and the UPS was beeping with a red error light.
The situation was that the servers had continued to run for a couple of hours with no airconditioning, and the room had become so hot, the UPS inside the server room batteries had balooned out to a point where we had to pull apart the UPS chassis to remove the modules.. two of the batteries had split their casing and the smell was because of the acid... and let me tell you, it stank like crap.
The moral of the story is, make sure your bloody servers shut down when the UPS kicks in for more then a few minutes.. this could have gone a lot worse then it did.
Cheers,
Darrkon
posted by Darrkon at 7:02 PM
1 Comments:
Oh oh, power outages + servers + 5 labs@21pcs each = major headache when the power comes back on and all the desktops boot up at the same time and blow all the fuses as well. Luckily the 6 servers had done an orderly shutdown in time, but turning off each pc, room by room, resetting all the switch boards then restarting one by one and shutting down properly took freaking hours.
No rotten eggs, though. I don't envy you that at all!
Post a Comment
<< Home