Peer 1 remained online, thanks to a bucket brigade carrying fuel up 18 floors
When Hurricane Sandy hit New York this week, a lot of big-name sites were taken down, after their ISPs were flooded or lost power. But some smaller sites remained online, despite using an ISP located in the evacuation zone, thanks to luck, planning, and sometimes a bucket brigade that carried fuel.
Huffington Post and Gawker were among the high profile casualties, after the superstorm took out power, communications and 25 percent of the mobile towers in 10 states. Peer 1 Hosting rents co-location space to a variety of smaller sites within 75 Broad Street, a high rise building in Lower Manhattan. The building had been surrounded by water, debris and wrecked cars, but the provider stayed online.
With a little help from our friends
The hero of this story is engineer Mike Mazzei of Peer 1 who, along with four other staff, had remained at the data centre since late last week when the firm began preparations for Hurricane Sandy. While Mike struggles to keep the data centre running, Peer 1′s spokesman Robert Miggins takes up the story: “It’s a combination of proper planning and a couple of lucky bounces that has kept us live.”
There was time to prepare for the storm, so Peer 1 made sure that all of its East Coast data centres were topped up with fuel for their back-up diesel generators, and food for staff who might be staying there round the clock.
The 75 Broad Street is a long-established “telecom hotel” in which Peer 1 is just one of several tenants. The storm hit on Sunday, and the surroundings began to flood. The city’s power for the area shut down, and 75 Broad Street building failed over to shared diesel generators, provided by the owners.
Flooding in the basement
The problem here was that diesel generators’ fuel tanks were in the basement. The flood water reached four feet in the lobby, and the basement flooded completely with salt water. The fuel pumps failed, and all the companies in the building had to shut down as the diesel generators ground to a halt.
All except Peer 1. The company has its own diesel generator on the roof of the building, at the 17th storey level. That generator had its own “day tank” of diesel fuel, gravity fed from the 18th floor, so the host carried on.
Diesel generators can only go on as long as they have fuel, and Peer 1 knew it would run out at some stage, so it planned for a ”controlled shutdown”. According to Miggins, on 30 October, with only two hours of fuel left, the company offered its customers a choice: “Stay online and take a chance, run a back up, or move your data to the cloud.”
Then came two of the lucky bounces: first, as water drained from the streets, some Peer 1 customers based in New York – Squarespace and Fog Creek Software – came over to manually shut down their kit.
And then, the bridges to Manhattan re-opened, allowing a diesel delivery truck to reach Broad Street. Since then, regular deliveries of diesel fuel have been possible.
So Mike Mazzei had all the diesel he needed – in barrels on the sidewalk outside the building – and a gasping diesel generator 17 floors above. There were no pumps available in the building, and no way to connect the barrels to the fuel line going upstairs.
The solution was a typical brute-force New Yorker approach. Squarespace and Fog Creek pitched in, along with Mazzei and his staff, to form a bucket chain lifing the fuel up to where it was needed. A total of 25 people continually fed the tank using five gallon buckets filled from the barrels. “People are just figuring it out together,” said Miggins. “It is about perseverence and getting something done.”
“We have also taken measures to cut down on fuel consumption,” said Miggins. “We have shut down the CRAC and air conditioning units, and we are allowing the temperature in the data centre to rise slightly to extend the useful life of the fuel tank.” Peer 1 believes it can go up to 90F without its equipment failing.
Lessons for the future
Fog Creek had access to its servers throughout the crisis
Peer 1 believes it can fuel the tank faster than the generator burns the diesel, and staff have been able to take breaks from climbing the stairs.
Meanwhile, the building owners are pumping out the basement, but it will take time: “We are now removing one foot an hour, with about 15 feet left,” says the most recent post on Fog Creek’s blog.
Getting a replacement pump to get fuel up to Peer 1′s tank is the other task, and that’s not proving easy: “Given the situation in New York City right now, we’re in a wait-and-see posture,” said Squarespace’s blog. “Fuel- and water-pumps are in short supply.”
Despite that, both customers and Peer 1 itself are optimistic they can keep running.
It’s early days to learn any lessons for the future, says Miggins, but the company is likely to look for a permanent back-up fuel pump. It might do well to find ways to reduce its energy needs, so that its diesel will go further. Most data centre owners agree that data centres should not need air conditioning at this time of year in temperate conditions like New York.
Do you know Green IT and energy consumption? Try our quiz!