Rack And Ruin

Years ago, when I was doing network operations, we got an alarm for a data center that had a problem with the sprinklers. These days, data center sprinkler systems are loaded with inert gases like argon or nitrogen, but back then, it was water. We called someone to head into the office to work on the issue.

After that, we had to reach out to the facilities manager. As I am in Richmond, Virginia, and the data center was in California, I was calling them at about 5 AM. When the guy picked up (VP level), he read me the riot act and hung up. 

I called him back and stopped him when he began to rant:

Me: “I just need to give you the message, and then you can hang up. Your data center is flooding, and water is pouring on the server racks.”

VP: “Holy s***!”

Me: “I have facilities on their way to the site, and we are pulling the appliance list to notify the application owners.”

VP: “I’m in a car with a bunch of friends on the way to a football game.”

It’s 5 am their time, but California’s traffic is legendary.

Me: “Would you like me to call the CISO (Chief Information Security Officer, his boss)?”

VP: “…Yeah.”

Me: “Will do. Enjoy the game.”

He later contacted our department to apologize to me and talked me up to my boss.