Arik Hesseldahl
November 1, 2012 at 5:49 pm PT
The whole point about cloud computing services that live on servers that customers never see, never touch, never think about is exactly that. The physical infrastructure is out of sight, out of mind, and customers focus on running their applications and using the services they provide.
But when disaster hits, when power fails followed by backup power failing, things get serious and physical rather quickly. That has been the case with Squarespace, an upstart Web site creation and hosting company based here in New York.
It has hosted its infrastructure with Peer1, one of a handful of companies affected by the flooding at 75 Broad Street. The short version of the story is that after backup generators kicked in, flooding overwhelmed the fuel pump.
CEO Anthony Casalena lives nearby in Soho and headed down to 75 Broad to see if he could lend a hand. Within hours he was pitching in with the staff of Peer1, manually carrying fuel to the generators, bucket-brigade style up 17 flights of stairs to keep them running. Staff from another company, Fog Creek Software, was also involved. The picture is of the “bucket brigade” from last night.
The result is that they managed to keep the systems running by manually feeding the generators while searching desperately for another way to keep a steady flow of fuel until reliable commercial power is turned back on. The best guesses we’re hearing from ConEdison is that this won’t be until Friday or Saturday at the earliest, maybe longer. The entire saga has been chronicled on a status blog with updates like this from 11 pm ET yesterday:
The midnight bucket brigade has begun (photo below).
When tonight’s brigade is done, we estimate we have a conservative 10 hours of capacity, allowing our 8am shift to begin tomorrow with low stress and also a switch to 120 minute refill sessions every 8 hours, versus yesterday’s continuous huffing of fuel up the stairs. Yeah, we’re getting good at this.
The building has had crews pumping water out of the basement all day. Unfortunately, we have no ETA on the repairing of the basement fuel delivery systems until the building’s crews get down there. However, both the building and PEER1 have been putting together separate contingency fuel pumping systems. We hope one of these will be online in the next 24-36 hours so that fuel refills will involve very few people and simply take place via a truck at street level.
Relief finally arrived late this afternoon with the installation of a new pump and by early evening the generators were humming along smoothly.
Who knew the cloud could be such hard work?