This story happened when I first joined my current company, and while I was not the one who actually had to deal with the problem, I was a bystander and heard the juicy parts from my mentor himself.
Exactly two days before a major festive celebration, we got a call from a user who was panicking because one of his equipment had failed and production had come to a screeching halt. Now, I work in a company that services critical process equipment in a country with a distinct west half and east half, separated by the sea (important as we are based in the western half). The client was a major refining plant for the petroleum industry.
As we normally do, we went through the usual troubleshooting steps — did you turn this on, is this connection active, etc. — but the only answer coming from [User] was, “Yes, yes, yes,” with nothing seemingly wrong. This went on for about half an hour.
Then, suddenly, our boss came in. The client’s head of production had just called him and was apparently livid. It turns out the machine hadn’t been working for more than an hour, and the production was severely interrupted until the problem was fixed.
Now, everyone was in a panic, as every hour the production was interrupted, the client was losing money in the tens of thousands of dollars, and the client had the right to sue us for any damages that occurred as a result of equipment downtime. [Head Of Production] was not happy that their internal team was not able to fix the problem, and [User] was not making any headway in fixing the problem via phone.
To resolve the issue, [Head Of Production] demanded that support be performed immediately onsite. Coming back to my earlier points: one, it’s the festive season, and two, they were across the sea, and traveling was a bit of a problem. But [Head Of Production] said money was not an issue and they would pay anything for immediate onsite support.
Cue my mentor, who was handed the unsavory task of handling the emergency. Immediately, he grabbed his tools and sped off to the airport to grab the next available flight. At the same time, his wife had to pack some clothes for him from home and rushed to pass them to him at the airport. Due to the festive season, [Mentor] didn’t have choices for flights, so in the end, he had to take a 1,000 USD business class flight. Normally, flights to where the client is located cost around 80 USD; we’re a developing country.
Upon arriving, [Mentor] was whisked from the airport with a driver, sent immediately to the refinery, and granted immediate security clearance to enter the plant. (Anyone working in petroleum knows how big a deal this is.) By this time, a good six hours or so had passed since we received the call, and it was well into the night. Greeting him in front of the equipment were [Head Of Production], [User], and various other senior management personnel, all anxious to see what the problem was.
[Mentor] is a guy with no chill, and he was also the one originally speaking to [User] on the phone. He recounted this part, so I’m paraphrasing him.
Head Of Production: “So, what is the problem?”
Mentor: “Wait, let me take a look.”
He started to go through the normal troubleshooting checklists but stopped almost immediately.
Mentor: “[User], are you sure you checked everything I asked you to?”
User: “Yes! Everything, word for word!”
Mentor: “Are you absolutely sure?”
User: “Yes!”
Mentor: “Do you remember what the third thing was that I asked you to check over the phone?”
User: “Why does it matter? Just fix the g**d*** problem!”
Mentor: “The first thing we normally check is to make sure the PC is turned on.” *Points at the CPU LED indicator* “The second thing we check is to make sure the equipment is on.” *Points to the machine LED* “The third thing…”
He brought his hand to a gas control valve and rotated it, a loud hiss was heard as the gas line pressurized, and the equipment beeped.
Mentor: “…is to make sure the gas is on.”
User: “…”
Head Of Production: “…”
Everyone Else In The Room: “…”
Mentor: “I would like to go have dinner now.”
After more awkward silence, [Head Of Production] thanked [Mentor] for his effort and asked the driver to take [Mentor] somewhere for dinner.
You’d think the story ends here, but there’s more!
By the time [Mentor] had finished his dinner, it was well past midnight, so he checked himself into a hotel for the night. The next day, he went back to the airport and found out that all flights were completely sold out for the next four days due to the festive traveling. He called my boss to inform him that he was basically stranded, and my boss just coolly said:
Boss: “Well, [Mentor], consider this as having a free holiday paid for by [Client].”
So, [Mentor] checked into the most luxurious hotel in the area and spent the next four days basically on vacation before coming back to work.
In total, we billed the client for around 10,000 USD for the flights, hotel, emergency arrangements, allowances, etc., all for ten seconds to check LEDs and turn a valve. This didn’t include the losses from halting the production. It’s still one of our most memorable stories that we recount to new hires or clients in our industry. Sometimes we wonder what happened to [User], but he was transferred out of his role not too long after this incident.