Engine Yard

Early this morning, we experienced the first power disruption in our 19-month history at our Sacramento data center.

At 2:05am Pacific time Saturday, March 15th, we started losing connectivity to one of our shared clusters, ey01.

Our engineers worked with the data center and determined that power to ey01 had been reduced or cut entirely. Keep in mind that clusters have 2 power connections, each capable of handling the load alone.

Other data center customers and infrastructure were affected - not just Engine Yard. But at least only 1 of our clusters was affected.

A full cluster reboot takes time, but not usually a long time. In this case, the problem was greatly magnified because the outage was not planned nor orderly, so we needed to do a cold reboot and make sure everything restarted in a good state, including integrity checks on all databases on ey01. Some websites were down as long as 4.5 hours, depending on which resources they needed. ey01 was fully back online by 6:35am Pacific.

Customers on other Engine Yard clusters were not affected.

We take this very seriously and are in contact with our data center to figure out exactly what happened and what measures they're taking to prevent it from happening again.

We are very sorry to Engine Yard customers who were affected by this power outage.

Please contact us if you have any questions or comments.