Unfiltered Story #296180

One of the features of working with semi-automated computer systems is the rise of the “machine ID” a full directory entry, complete with email address for the automated process in question. If you dig into the directory record you can figure out which human owns the ID.

There are also automatically generated email distribution lists. So if you work for department XYZ, you can send an email to Department_XYZ_mail and it will reach everyone in the department. When your department changes in the company directory, the automated process moves your ID to the new list.

One day everyone in the division (thousands of people) got a strange email that looked like it came from a machine ID. As the info wasn’t relevant to any of us, several people replied to the ID asking to be removed from the list. This is the normal procedure: don’t hit reply all, just reply to the machine ID and get removed from the process distribution list. Problem was, it WASN’T a machine ID, it was a distribution list for a high level position in the org chart, so that everyone in the down line of that position got the email, and all the responses asking to be removed. In essence replying to the sender only was the equivalent of reply all.

This lead to the inevitable sequence of people hitting reply all to the removal requests asking that people not use reply all. Inboxes were quickly flooded and all real communication was lost in the noise. A few of us generated mail filters to dump everything from that sender to a spam folder and turned off our new mail notifications, but the servers themselves were overwhelmed so legitimate communication was often quite delayed.

Then the investigation began as we tried to figure out who owned the offending system and why it was using a distribution list instead of a machine ID to send out the initial message. Digging through the directory we found a conundrum: the distribution list belonged to an executive that was no longer with the company (which should have automatically disabled it), and it’s ownership had defaulted to a machine ID. The machine ID in question, owned by the missing executive, was assigned to the distribution list as the owner. Someone had gotten creative with assigning roles and backups, and the failover when the top of the pyramid left didn’t work as a result. There was no longer an assigned human in charge of any of this!

It took nearly half a day for someone in IT with sufficient access to disable the rogue distribution list. When I went to empty my spam folder, there were over 3000 messages from the chain.

All that because someone somewhere didn’t want to be bothered with properly managing IDs and assigned a machine to monitor the machines.