Sometimes fellow techs that makes you want to smash your keyboard. (all names have been changed to protect the incompetent).
I used to work in a Service-Desk/Triage capacity (lovingly dubbed Tier 1.9) where we would monitor, ticket, troubleshoot, and escalate as needed (we were expected to have a 90% self-solution return).
One thing that we never were allowed to touch was network issues. Our area telcom team had lost their extremely competent team lead some 18 months earlier and their team had quickly degraded to people who didn't really seem to care about their job. They were also secure in the fact that the telcom system had degraded enough to be sufficiently obfuscated enough where it would take considerable resources to figure it out by a competent replacement. This made them act like they were untouchable. As such, I had learned fast to start recording my phone calls to them (thank you Google Voice!) so they couldn't pin anything on us.
Anyway, one day around 6PM the network dropped out. As such, I called the Telcom On-Call.
Me Leaving Voicemail: Hi, this is Craysh from Tier 1.9. It looks like the network is having issues again. Internet, communicator, and email are all unresponsive. All three webfarms are unavailable. Trace route to the gateway has failed. Ticket is unavailable as the ticketing system is inaccessable due to the issue. Please call back to confirm your taking ownership of the incident.
15 minutes go by and SOP dictates that I call back before escalating to their manager. The on-call answers.
Tech: On-Call Phone, this is Steve
Me: Hi Steve, have you gotten my voicemail? We're having issues with the network.
Tech: No, I haven't. Tell me what's wrong.
Me: Repeats what I said above.
Tech: Ok. Wait 20 minutes and try again. Click
I had assumed that by 'Wait 20 minutes' he meant that he was taking a look at it and that it should be back up in 20 minutes. 20 minutes go by and the network is still having issues (The network has now been down for the last 40 minutes or so)
I call the On-Call phone again. No answer.
Me Leaving Voicemail: Hi, this is Craysh again. The issue is still ongoing. Please call me back as soon as possible.
15 Minutes go by so it's time to call again. No answer. Time to call the acting supervisor.
Supervisor: This is Aaron
Me: Hi Aaron, this is Craysh from Tier 1.9. We are having issues with the network and the On-Call has stopped answering their phone.
Supervisor: Stopped? You mean they initially answered?
Me: Yes, he said to wait 20 minutes.
Supervisor: OK, he's probably working on it. Call me back in an hour if the issue persists. click
At this point I'm calling my supervisor and apprising him of the situation.
For the next hour, I continue to call the On-Call every 15 minutes (as is procedure). No answer.
I call the supervisor again
Supervisor: This is Aaron
Me: Hi Aaron, this is Craysh from Tier 1.9 again. We are still having issues with the network and the On-Call still has not answered his phone.
Supervisor: OK, I'll take a look at it.
Turns out that Telcom had pushed a policy to the Junipers that had scrubbed the white lists (completely blocking all access to all ports).
All in all it took 5 hours to get everything back up and running. I documented the ticket extremely thoroughly once the system was available again.
The next day came the fall-out. Telcom was claiming that I had never paged them, and they said that a customer ended calling the supervisor to find out what was going on. They even showed zero calls on their cellphone (they had removed the calls and messages).
As such, we had a meeting to 'discuss Tier 1.9's failure to report a severe incident'.
I walked into the room filled with extremely ****ed off big wigs, including Aaron and and Steve. They demanded why I hadn't contacted the on-call, and I stated that I had and that they had lead me to believe that they had been working on the issue.
Nobody believed me because "The On-Call phone didn't have any messages or call logs from Tier 1.9" Their excuse for having a completely empty call log was that they cleared it daily so they could keep track of the calls made on the phone.
I calmly walked to the overhead projector, logged into Google Voice, and played each consecutive voicemail/conversation.
They no longer work here, and I've had three promotions since.
Edited by John S., 04 November 2012 - 12:10.