The Desert Can Play Tricks On An Otherwise Sound Mind

I am a network engineer working on an Air Force base. I am not an NCO (Non-commissioned Officer) or an officer; I am a contractor from an outside company. My assignment is to write new access control lists and do a bunch of other tasks to about 200 network switches to make sure they are in compliance with certain rules. It’s tens of thousands of changes overall, and I have been putting in a lot of (paid) overtime to try and get it done sooner.

It’s early in the day when one of the senior Airmen comes over to talk to me.

Senior Airman: “We have an outage over at [Site], and I’ve been asked to take you there in case any of the changes you made caused it to go down.”

I get the IP address from him and type it into my terminal. It does not come up in my session history, so that means it is the first time I have ever used this IP address. I attempt to access it and the connection fails, showing that the IP address is not found on the network.

Me: “I don’t have a session for that node saved in [Application]. It does not look like I made any changes to it.”

Senior Airman: “They still want you to go with me to check it out.”

So, I pack up my laptop and brave the heat of the Nevada desert in June, and he drives me out to a remote part of the base. This location is used to store missiles, so it is not someplace that even my senior Airman companion can enter unescorted. We have to wait for someone to drive us up and walk us into the affected area.

When I get there, I find that there are a number of small buildings, each with their own computer stations, printers, phones, etc. Only one of those buildings is affected by the outage, and no critical or security systems are down. All it means is that the handful of people there have to walk next door to get on the network or make a phone call. It’s annoying, sure, but not a critical issue.

When the Airman and I get to where the switch is, we immediately see the problem. There are no lights at all from the switch, and it is plugged directly into the wall! You never plug enterprise-class networking equipment directly into power outlets without something to insulate it from power surges.

Senior Airman: “Uh… where’s the uninterruptable power supply?”

I trace the power cables to confirm.

Me: “Why is your switch directly connected to your power outlets?”

The recruit who was left with us has no idea what we are talking about. I try to turn on the light in the area to see better, but it won’t turn on.

Recruit: “We had a power surge last night and that light doesn’t work anymore. Totally fried our printer, too.”

Me: “I think it fried more than that.”

Senior Airman: “That might have been something you guys should have mentioned in your call.”

I unplug the switch’s two power supplies and test the outlets with my laptop.

Me: “Looks like power on this circuit is still good, but your switch is dead.”

Recruit: “Can you fix it?”

Senior Airman: “No, we’re going to have to return it. We’ll see if we can get another one out here… and an UPS to make sure it is protected from power surges.”

So, that is half a day of my time wasted. When I am done for the day and it is late in the evening, I fill out my daily report — a report read by government representatives, a higher-ranking NCO on base, and three of my bosses — and I am so annoyed that I must admit I lose a touch of my usual formality and it contained the following bullet point in the middle.

Report: “I was taken out on site to examine a down switch that was suspected of being disabled due to the changes I have been making to the network. Upon examination, it was discovered that improper installation has allowed a power surge to cause the switch to be downgraded from ‘critical network access device’ to ‘$11,000 paperweight’.”

I get back to my hotel pretty late and go to bed. The next morning, I call into a scheduled phone meeting. The only other participant who has arrived so far is one of my bosses. We exchange pleasantries, and then…

Boss: *Poorly trying to contain his laughter* “So… $11,000 paperweight, huh?”