?

Log in

No account? Create an account

Previous Entry | Next Entry

my day...

so yeah, let see.

apparently the electricians who Sinai contracted for the ongoing infrastructure upgrade are complete and total wastes of space.

Our entire data center now runs on a building UPS - the juice should never go down.

Unless the main circuit for the failover is 60 amp, instead of 100 amp.

When that happens, the circuit breaker blows up like a Chinese new year party favor.

What happens then, is lots and lots of the data center has no power - servers crash, as do racks and racks of Cisco gear, since, of course, they no longer have individual UPS - we are on building wide now!

The problem is identified (by the ozone and smoke) - extra emergency wiring is used to bypass the fried circuit - supposedly 3 phase power will be restored in 20 minutes.

Instead of 3 phase power, we end up with 2 phase, over wiring which was not made to handle that kind of current* - result: wire that is so hot that the coating on it melt, and burns if you touch it.

Amazing we didn't have a fire - which would have been the end of things, since the block we were running from, apparently, was the backup grid for the fucking EMERGENCY POWER - y'know, the things like the fire suppression system, and backup lights.

Ftards.

So yeah, now we have a super hot server room that is running on melty wire - the juice is cut, new wires are brought in, they get the old circuit partially removed (now that it has gone from liquid to solid state once more).

Of course, the dipshits patch us in to a circuit that either has an open neutral or a faulty ground. Three machines die booting - the main web server goes down like Haley's comet.

Again juice is cut.

Damage assessment - three machines removed from racks by flashlight (folks, don't try this at home - its like playing twister with power tools as extra partners).

A secure circuit is found.

Power is restored.

The secure circuit collapses under load - not once, not twice, but thrice - each time, it comes back, but each time, it took another machine with it.

Net result:

2pm, running on emergency power circuit AND "secure" circuit.

5 servers down.

WWW on backup unit, last synch was 18 hours ago - clients cannot ftp - we don't even know where to start.

Replacement hardware is in the air - I got to see what a slagged raid array looks like.

Tape drives have been largely inaccessible, since one of the last crashes kicked the main backup DB into divide by 0 space, leaving us having to re-merge ~ 2 terabytes of data back into the db before we can figure out what we have in the deck to deal.

People are hot, angry, and shit is up, for now, but everyone is watching pagers warily.

I want to go home, I definitely don't make enough to deal with this shit.

Best estimates is that replacement box will be here by 7pm - we should have things restored by ~9ish IF we can find good tape.

I am supposed to stay "on call".

*edit - I have corrected my verbiage and description, props to bruteforcemethd for straightening me out.

Comments

( 3 comments — Leave a comment )
bruteforcemethd
Sep. 11th, 2003 12:05 pm (UTC)
for my personal edification
just to be picky..

all AC power delivered is 3 phase (I'm pretty sure anyway - I'm not an EPOW..). when they say you are off 3 phase, I think that means you are switched to DC, which would make sense for a UPS (rather than a generator) which I assume is a big battery - 'no phase'.

P.S. I want all of the breakers in my house loaded with confetti too! that would have given my parents heart attacks weekly given my childhood. =]
delascabezas
Sep. 11th, 2003 12:21 pm (UTC)
you are right
your explanation prompted an edit - and makes perfect sense - apparenttly the electricians thought we were going to try and restore power to the UPS rather than go into the main switch for the electrical components - when they nearly melted down the wire, they were hollering back and forth about 3 phase vs 2 - i assumed it meant current carried, which isin't quite on the money.

tx for the 411.
grimbil
Sep. 11th, 2003 02:59 pm (UTC)
Well
So, like, is my Linux box fixed yet or WHAT?





I kid, I kid!
( 3 comments — Leave a comment )

Profile

Steam Escaping!
delascabezas
The Son of the last of a long line of thinkers.
delascabezas.com

Latest Month

February 2017
S M T W T F S
   1234
567891011
12131415161718
19202122232425
262728    

Tags

Powered by LiveJournal.com
Designed by Tiffany Chow