Wait 20 Minutes and Try Again or How I Almost Got Fired...


Recommended Posts

This is a copy / paste from Reddit, i thought it was a good read!

Sometimes fellow techs that makes you want to smash your keyboard. (all names have been changed to protect the incompetent).

I used to work in a Service-Desk/Triage capacity (lovingly dubbed Tier 1.9) where we would monitor, ticket, troubleshoot, and escalate as needed (we were expected to have a 90% self-solution return).

One thing that we never were allowed to touch was network issues. Our area telcom team had lost their extremely competent team lead some 18 months earlier and their team had quickly degraded to people who didn't really seem to care about their job. They were also secure in the fact that the telcom system had degraded enough to be sufficiently obfuscated enough where it would take considerable resources to figure it out by a competent replacement. This made them act like they were untouchable. As such, I had learned fast to start recording my phone calls to them (thank you Google Voice!) so they couldn't pin anything on us.

Anyway, one day around 6PM the network dropped out. As such, I called the Telcom On-Call.

Me Leaving Voicemail: Hi, this is Craysh from Tier 1.9. It looks like the network is having issues again. Internet, communicator, and email are all unresponsive. All three webfarms are unavailable. Trace route to the gateway has failed. Ticket is unavailable as the ticketing system is inaccessable due to the issue. Please call back to confirm your taking ownership of the incident.

15 minutes go by and SOP dictates that I call back before escalating to their manager. The on-call answers.

Tech: On-Call Phone, this is Steve

Me: Hi Steve, have you gotten my voicemail? We're having issues with the network.

Tech: No, I haven't. Tell me what's wrong.

Me: Repeats what I said above.

Tech: Ok. Wait 20 minutes and try again. Click

I had assumed that by 'Wait 20 minutes' he meant that he was taking a look at it and that it should be back up in 20 minutes. 20 minutes go by and the network is still having issues (The network has now been down for the last 40 minutes or so)

I call the On-Call phone again. No answer.

Me Leaving Voicemail: Hi, this is Craysh again. The issue is still ongoing. Please call me back as soon as possible.

15 Minutes go by so it's time to call again. No answer. Time to call the acting supervisor.

Supervisor: This is Aaron

Me: Hi Aaron, this is Craysh from Tier 1.9. We are having issues with the network and the On-Call has stopped answering their phone.

Supervisor: Stopped? You mean they initially answered?

Me: Yes, he said to wait 20 minutes.

Supervisor: OK, he's probably working on it. Call me back in an hour if the issue persists. click

At this point I'm calling my supervisor and apprising him of the situation.

For the next hour, I continue to call the On-Call every 15 minutes (as is procedure). No answer.

I call the supervisor again

Supervisor: This is Aaron

Me: Hi Aaron, this is Craysh from Tier 1.9 again. We are still having issues with the network and the On-Call still has not answered his phone.

Supervisor: OK, I'll take a look at it.

Turns out that Telcom had pushed a policy to the Junipers that had scrubbed the white lists (completely blocking all access to all ports).

All in all it took 5 hours to get everything back up and running. I documented the ticket extremely thoroughly once the system was available again.

The next day came the fall-out. Telcom was claiming that I had never paged them, and they said that a customer ended calling the supervisor to find out what was going on. They even showed zero calls on their cellphone (they had removed the calls and messages).

As such, we had a meeting to 'discuss Tier 1.9's failure to report a severe incident'.

I walked into the room filled with extremely ****ed off big wigs, including Aaron and and Steve. They demanded why I hadn't contacted the on-call, and I stated that I had and that they had lead me to believe that they had been working on the issue.

Nobody believed me because "The On-Call phone didn't have any messages or call logs from Tier 1.9" Their excuse for having a completely empty call log was that they cleared it daily so they could keep track of the calls made on the phone.

I calmly walked to the overhead projector, logged into Google Voice, and played each consecutive voicemail/conversation.

They no longer work here, and I've had three promotions since.

Edited by John S.
properly tagged
  • Like 1

Good stuff.

I was once on the phone to Vodafone on behalf of my girlfriends mother whose contract had finished but she was still getting charged.

I recorded everything they said as i knew it was going to be a pain in the arse.

I was completely right and a dozen phone calls later no one at the company admitted to any call logs of previous correspondence between myself and Vodafone.

After playing back part of my conversation to the Indian chap on the phone, i was forwarded to another Indian chap and then another one and eventually ending up with a well spoken English chap who solved all my issues, closed the account, paid back all money owed and compensated us to the tune of ?150 for wasted time.

Sad fact is i had to go through all that for a simple account closure. There is no accountability any more.

Wow. Awesome story with a good ending. Incompetent people should realize that eventually their foolishness will catch up and bite them in the butt. Good to hear you've been promoted since then too! It sounds like management know when they've got someone who can do good work.

This topic is now closed to further replies.
  • Posts

    • I was using searxng for about a year , self hosted, but results were starting to timeout and eventually it became unusable so I switched to degoog. Much better for my needs, more polished and add-ons like maps and calculations etc
    • Fake Superman doing the Anti-Trump PR for us, good man !
    • Hello, I am not as familiar with AMD CPUs as I am with Intel's, but as I understand it, that's a mid-range CPU from about three years ago.  I would think it to be fine for everyday casual-type use.  A larger SSD might be better, but with storage prices these days that's a decision that has to be carefully thought about. Regards, Aryeh Goretsky  
    • Ocenaudio 3.19.5 by Razvan Serea  Ocenaudio is a full featured, fast and easy to use audio and music editor. It is the ideal software for people who need to edit and analyze audio files without complications. Ocenaudio also has powerful features that will please more advanced users. To assist ocenaudio development, a powerful toolset of audio editing, analysis and manipulation called Ocen Framework was created. ocenaudio is also based on Qt framework, a well known library for cross-platform development. Cross-platform support ocenaudio is available for all major operating systems: Microsoft Windows, Mac OS X and Linux. Native applications are generated for each platform from a common source, in order to achieve excelent performance and seamless integration with the operating system. All versions of ocenaudio have a uniform set of features and the same graphical interface, so the skills you learn in one platform can be used in the others. VST plugins support Ocenaudio supports VST (Virtual Studio Technology) plugins, giving its users access to numerous effects. Like the native effects, VST effects can use real-time preview to aide configuration. Real-time preview of effects Applying effects such as EQ, gain and filtering is an important part of audio editing. However, it is very tricky to get the desired result by adjusting the controls configuration alone: you must listen the processed audio. To ease the configuration of audio effects, ocenaudio has a real time preview feature: you hear the processed signal while adjusting the controls. The effect configuration window also includes a miniature view of the selected audio signal. You can navigate on this miniature view in the same way as you do on the main interface, selecting parts that interest you and listening to the effect result in real time. Multiselection for delicate editions To speed up complex audio files editing, ocenaudio includes multi-selection. With this amazing tool, you can simultaneously select different portions of an audio file and listen, edit or even apply an effect to them. For example, if you want to normalize only the excerpts of an interview where the interviewee is talking, just select them and apply the effect. Eficient edition of large files With ocenaudio, there is no limit to the length or the quantity of the audio files you can edit. Using an advanced memory management system, the application keeps your files open without wasting any of your computer's memory. Even in files several hours long, common editing operations such as copy, cut or paste happen almost instantly. Fully featured spectrogram Besides offering an incredible waveform view of your audio files, ocenaudio has a powerful and complete spectrogram view. In this view, you can analyze the spectral content of your audio signal with maximum clarity. Advanced users will be surprised to find that the spectrogram settings are applied in real time. The display is updated immediately when altering features such as the number of frequency bands, window type and size and dynamic range of the display. Ocenaudio 3.19.5 changelog: Fixes crashes related to audio devices on Windows (DirectSound and ASIO) Fixes several crashes and memory corruption issues Fixes opening several headerless files at once, which previously dropped all but one Improves batch export by suggesting and remembering the destination folder Fixes accented and non-Latin characters in VST plug-in and compressed-archive file names Adds zstd compression support and updates the archive library Other bug fixes and improvements Download: Ocenaudio 64-bit | Portable | ~40.0 MB (Freeware) Download: Ocenaudio for Linux and Mac OS View: Ocenaudio Homepage | Screenshot Get alerted to all of our Software updates on Twitter at @NeowinSoftware
  • Recent Achievements

    • One Month Later
      Excellence2025 earned a badge
      One Month Later
    • Week One Done
      Excellence2025 earned a badge
      Week One Done
    • Week One Done
      flexorcist earned a badge
      Week One Done
    • One Month Later
      Woland13 earned a badge
      One Month Later
    • Week One Done
      Woland13 earned a badge
      Week One Done
  • Popular Contributors

    1. 1
      +primortal
      501
    2. 2
      +Edouard
      204
    3. 3
      PsYcHoKiLLa
      145
    4. 4
      Steven P.
      72
    5. 5
      FloatingFatMan
      68
  • Tell a friend

    Love Neowin? Tell a friend!