Yesterday I posted about the likelihood of Jesus & Venus going offline due to a hurricane-related power outage at the data center that hosts this and other Squarespace sites. So far that has not happened, due entirely to the courageous and tireless efforts of the Squarespace staff.
Here is a selection of updates from Squarespace’s running blog on the Hurricane Sandy power outage:
October 30, 2:07 p.m.
We have an understanding that when the fuel header on the roof stops refilling our tank, we will likely be in a window where shutdown will occur within 2 hours. That has not happened yet, which is good. We have fuel at the building, but cannot tap into the damaged building infrastructure yet.
October 30, 7:39 p.m.
We have been running hour by hour by manually carrying fuel to our generators (17 floors) with support from the building. As the night goes on, this is becoming a bit more difficult to sustain, as fuel trucks will appear more intermittently. For now, we remain online. Thank you all for your patience.
As before, we do not have a sustainable solution to the energy problem, but we persevere in our efforts to get more fuel on site and get a pump connected.
October 31, 1:02 a.m.
We continue to manually provide fuel to the generator, a plan which appears to be working, but is likely not sustainable. The building now has powerful pumps clearing out the basement, which we hope will expose the main pump lines — which would allow us fuel for days. Hopefully our manual efforts, joined with the building, can see us through. We have loaded enough fuel to carry to the morning, at which point we’ll provide another status update.
October 31, 12:40 p.m.
Good news and bad news.
We were able to give the crew 90 minutes off because the tank is full. That’s about 4 hours of power given daytime usage. We’ll start back up at 1:30. Our awesome teammates are hoofing lunch over the Brooklyn Bridge for us. As we’ve said before, this situation is untenable. We can’t keep manpower going 24/7 for days.
The building’s first attempt at an alternative method for pumping fuel to the 18th floor has failed, as the fuel pump wasn’t powerful enough. They believe they have sourced an alternate pump, but given the situation in New York City right now, we’re in a wait-and-see posture. Fuel- and water-pumps are in short supply.
The basement is not draining at all either, despite the large pumps that were brought in late last night. DEP and ConEd have been here for a few hours. They fear a water main has ruptured somewhere and is pushing water (and other stuff) into our basement as we pump it out. This is pure speculation at this point.
Terrific progress this afternoon.
October 31, 5 p.m.
We have a few hundred gallons of spare fuel on the roof along with a full tank, which will take us well into the night. On top of teams from Peer1, Fog Creek, and Squarespace, we were able to hire additional help from Brooklyn and Queens. Special thanks to Mike Mazzei and his team at Peer1 NYC for unprecedented dedication these past days.
We have a big crew teaming up tonight for a midnight bucket brigade, and another in the morning. Traffic is terrible with gridlock everywhere and no public transport, so we are scheduling carefully.
We are cautiously optimistic that fuel will be running to the roof by end of day tomorrow or the day after. Water is being successfully pumped from the basement at a rate of one foot per hour. Fifteen feet of water remain. We have increasing confidence that we may avoid downtime. Of course, there are no guarantees and we are working hour-by-hour.
You can also see some pictures of the Squarespace team in action.
I’m sure I speak for every other Squarespace customer when I say that this is going far above and beyond what anyone would have expected from them. No one could blame the company for allowing some “Act of God” downtime on their servers; people understand that this is one of the greatest natural disasters the country has ever seen.
Instead, though, they’ve worked themselves tirelessly in a so-far successful effort to provide uninterrupted service. Their fanatical willingness to carry buckets of fuel up 17 flights of stairs over and over around the clock just to keep our sites running exemplifies why I want Squarespace hosting Jesus & Venus.