Servers, Servers Everywhere

After we had the Bad Boss, who reduced our college’s IT team and budget to nothing, we had the Good Boss, who was great. He wanted to improve things instead of just desperately duct taping them together. Very hands-on, he even went out in the field sometimes to see what we were doing.

When he arrived, the greater University was just gearing up to transition from Windows XP to 7. The discussion over how to do this got a little delayed, so then it became XP to 10 (much to our great relief). Our boss suggested we make an image for our college’s computers following University standards to push out to all the machines.

When we stopped laughing, we pointed out that this wasn’t going to happen. Our college’s computers weren’t networked in any real sense of the word beyond “most of them connect to the internet, somehow”. Our servers certainly didn’t talk to the University servers. Most of our servers didn’t talk to OUR servers. The best we could possibly do was use this upgrade to bring everything into cohesion.

Our new boss asked us, cradling his head in his hands:

Boss: “Wait a minute. Help me understand the scope of the problem. How many of our servers don’t talk to our other servers? How many servers do we actually have?”

We all looked at each other.

There were several servers in the room we were in; those were easy enough. There was an email server and a server for the printers on this floor. We also had—

Boss: “Wait. The print server is just for this floor? We have ten buildings and probably thirty floors between them all.”

Oh, no, we reassured him, some of the buildings had just one print server, and some even shared them. But some had a different print server per lab because the labs used to be owned by a different college and we inherited them, and in some cases, a professor had gotten a grant and bought their own print server.

Boss: “What? Why?”

Shrug. Who are we to question the wisdom of the faculty?

But back to the count. Everyone knew about the server next door because it was part of an international grant and the US government contacted us occasionally to ask why it was transmitting to Iran. (Answer: the professor was in Iran… hopefully doing normal things.) But no one knew what the server sitting on top of that one was for.

Actually, as we took our impromptu meeting into that room to poke around, we found four more servers that were definitely running and doing something. So that was seven, and those were just the ones in immediate proximity to us.

Our network guy, aka the one tech who knew something about networks, said that he had about thirty-six of them that he monitored. He could tell from traffic that there were definitely more, but he didn’t know where they were, exactly.

Were any of these servers backed up? Onto what, exactly? More servers?

Our new boss, looking older by the minute, gave us orders: any time we weren’t on a ticket, we were to go room by room in every building, looking for servers.

It was the Easter egg hunt from Hell. We found servers running under desks in storage closets, behind other servers, above ceiling tiles. One had been installed in a Facilities closet against a hot water intake pipe and had partially melted. I remember that one in particular because the tech who found it had to fill out an injury report after getting burned by the server/pipe hybrid. After that, the boss made sure we all learned what hot water pipes looked like, just in case.

Our boss also ventured out himself to help. One time, he found three servers just stacked on the floor. While ranting to the tech with him about the ideal closet he would have installed them in if HE had put them in the room, he opened the next door and found exactly the model of wiring closet he had just described, standing empty. He had to go have a lie-down.

Our end total?

One hundred sixty-eight servers.

I never got into networking, so I’m uninformed in this area, but they assured me this was not the correct number of servers for a workforce of about 1,000. I don’t know. Maybe it works better if everyone has their own print server.