Why a server farm would not have a permanent back-up power solution in place is crazy to me. The building that we are in is owned by a guy that has the rest of the building as a server center. Out back he has a natural gas generator that is slightly larger than a 3500 series extended van with a 3" gas line going to it. He says that he can have the whole room up and running in less than 15 minutes in a total power failure. Some of this may be due to the fact that he handles the storage for a number of the hospitals in the area.

Depending on the data centre, and the supported applications , you may or may not have sufficient backup generator capacity to supply the entire data centre. If this was a data centre for the banking, securities or financial industries you would most certainly have a back-up generator and electrical system to be able to support those systems. Obviously, in the cost sensitive web server business does not demand this type of electrical redundancy or is willing to pay for this level of service.

However, having worked in several data centres, I also find it hard to believe that they would have the transformer located inside the building in such a location that a catastrophic failure of the transformer would have caused the amount of physical damage that it did. These guys could use some lessons from the old school telephone companies where protection of the central office building was more important than the equipment (i.e. switches, servers, routers ...etc.) housed in the building. The school of thought was that you can always replace the equipment in short order, but reconstruction of the building takes a lot longer.

Fixing the faulty breaker on the generator powering H1 Phase 1 was not successful. we have located a second generator that is currently being delivered to the facility. It is expected to arrive this afternoon and we will provide additional information regarding the new generator at that time.

This generator thing is starting to sound like the arrival of the XP-8...

I would think the electric company could have wired in a new transformer by now..

Or at least Shifted the DNS and Hosting to GoDaddy or some else! What a cluster F#4K operation they have down there.


The reason they cannot just bring up a Backup generator and get the section going again immediately is because the power wires in the floor feeding the Rooms server and air conditioning is damaged.

I'm ready to drive to Houston and kick some a$$ to get them moving.

Ive got two newborns, a wife starting to feel postpartum, my mother-in-law, and a 3 y/o wound tight because her grandma is visiting, all waiting for me at home. If I dont get an IP fix soon I may f'ing snap!

Well, don't you think the "original standard" was'nt good enough?

Just like rebuilding the New Orleans levy system to it's "original standard", becuase the original standard worked out so well for them.

They are releasing a new transformer version in two weeks.

I've got the shakes! Heads starting to hurt! I don’t feel good!

Withdrawal sickness? :(

Serious cluster,
What data center with 3000 servers doesn't have a redundant facility or a minimum recovery agreement with another provider?
Something is seriously wrong over there, and I'd suspect that they may never come back online. If they do manage to prop it back up, I'd move to a new provider as soon as I could backup the data.

I can see how a fire could damage the transformer room to the point where even the redundant lines are damaged beyond repair. But it's not all that difficult to run a temporary line to the internal power system and get it back up and functional.

Any DC worth it's salt has suitable backup power in place and ready to roll.

Hope I'm wrong, But we may need to start a "get IP to a new provider" fund ASAP.

God damn The Pusher man

I can't find the vein!!!

Hey Greg, I got some IP screenshots I will sell you for $5 each.

Temperature rising, fever is high, can't see no future, can't see no sky, my body is aching, goose pimple bones, I wish I was a baby, I wish I was dead...Cold Turkey, has got me on the run...

John Lennon