Wait 20 Minutes and Try Again or How I Almost Got Fired...


Recommended Posts

This is a copy / paste from Reddit, i thought it was a good read!

Sometimes fellow techs that makes you want to smash your keyboard. (all names have been changed to protect the incompetent).

I used to work in a Service-Desk/Triage capacity (lovingly dubbed Tier 1.9) where we would monitor, ticket, troubleshoot, and escalate as needed (we were expected to have a 90% self-solution return).

One thing that we never were allowed to touch was network issues. Our area telcom team had lost their extremely competent team lead some 18 months earlier and their team had quickly degraded to people who didn't really seem to care about their job. They were also secure in the fact that the telcom system had degraded enough to be sufficiently obfuscated enough where it would take considerable resources to figure it out by a competent replacement. This made them act like they were untouchable. As such, I had learned fast to start recording my phone calls to them (thank you Google Voice!) so they couldn't pin anything on us.

Anyway, one day around 6PM the network dropped out. As such, I called the Telcom On-Call.

Me Leaving Voicemail: Hi, this is Craysh from Tier 1.9. It looks like the network is having issues again. Internet, communicator, and email are all unresponsive. All three webfarms are unavailable. Trace route to the gateway has failed. Ticket is unavailable as the ticketing system is inaccessable due to the issue. Please call back to confirm your taking ownership of the incident.

15 minutes go by and SOP dictates that I call back before escalating to their manager. The on-call answers.

Tech: On-Call Phone, this is Steve

Me: Hi Steve, have you gotten my voicemail? We're having issues with the network.

Tech: No, I haven't. Tell me what's wrong.

Me: Repeats what I said above.

Tech: Ok. Wait 20 minutes and try again. Click

I had assumed that by 'Wait 20 minutes' he meant that he was taking a look at it and that it should be back up in 20 minutes. 20 minutes go by and the network is still having issues (The network has now been down for the last 40 minutes or so)

I call the On-Call phone again. No answer.

Me Leaving Voicemail: Hi, this is Craysh again. The issue is still ongoing. Please call me back as soon as possible.

15 Minutes go by so it's time to call again. No answer. Time to call the acting supervisor.

Supervisor: This is Aaron

Me: Hi Aaron, this is Craysh from Tier 1.9. We are having issues with the network and the On-Call has stopped answering their phone.

Supervisor: Stopped? You mean they initially answered?

Me: Yes, he said to wait 20 minutes.

Supervisor: OK, he's probably working on it. Call me back in an hour if the issue persists. click

At this point I'm calling my supervisor and apprising him of the situation.

For the next hour, I continue to call the On-Call every 15 minutes (as is procedure). No answer.

I call the supervisor again

Supervisor: This is Aaron

Me: Hi Aaron, this is Craysh from Tier 1.9 again. We are still having issues with the network and the On-Call still has not answered his phone.

Supervisor: OK, I'll take a look at it.

Turns out that Telcom had pushed a policy to the Junipers that had scrubbed the white lists (completely blocking all access to all ports).

All in all it took 5 hours to get everything back up and running. I documented the ticket extremely thoroughly once the system was available again.

The next day came the fall-out. Telcom was claiming that I had never paged them, and they said that a customer ended calling the supervisor to find out what was going on. They even showed zero calls on their cellphone (they had removed the calls and messages).

As such, we had a meeting to 'discuss Tier 1.9's failure to report a severe incident'.

I walked into the room filled with extremely ****ed off big wigs, including Aaron and and Steve. They demanded why I hadn't contacted the on-call, and I stated that I had and that they had lead me to believe that they had been working on the issue.

Nobody believed me because "The On-Call phone didn't have any messages or call logs from Tier 1.9" Their excuse for having a completely empty call log was that they cleared it daily so they could keep track of the calls made on the phone.

I calmly walked to the overhead projector, logged into Google Voice, and played each consecutive voicemail/conversation.

They no longer work here, and I've had three promotions since.

Edited by John S.
properly tagged
  • Like 1

Good stuff.

I was once on the phone to Vodafone on behalf of my girlfriends mother whose contract had finished but she was still getting charged.

I recorded everything they said as i knew it was going to be a pain in the arse.

I was completely right and a dozen phone calls later no one at the company admitted to any call logs of previous correspondence between myself and Vodafone.

After playing back part of my conversation to the Indian chap on the phone, i was forwarded to another Indian chap and then another one and eventually ending up with a well spoken English chap who solved all my issues, closed the account, paid back all money owed and compensated us to the tune of ?150 for wasted time.

Sad fact is i had to go through all that for a simple account closure. There is no accountability any more.

Wow. Awesome story with a good ending. Incompetent people should realize that eventually their foolishness will catch up and bite them in the butt. Good to hear you've been promoted since then too! It sounds like management know when they've got someone who can do good work.

This topic is now closed to further replies.
  • Posts

    • Personally, I’ve found that it’s usually worth investing in the infrastructure you don’t want to replace later, especially cabling. Running Cat6A (or better, depending on your needs) during an upgrade is relatively inexpensive compared to having to re-cable a few years down the road. For switches I try to balance current specs with realistic growth. If my budget allows it Ill choose switches with higher uplink speeds which leaves room for expanding later on, but I don’t necessarily overspend on access ports if the endpoints won’t benefit from them anytime soon. One lesson I’ve learned is that planning for scalability pays off. It’s much easier to add devices, VLANs, or higher-bandwidth workloads when your network infrastructure already supports it than to replace hardware later.  What is your budget like?
    • I hate the term, "future-proof." We saw it back in the 90's / 2000's, if not before. You cannot future-proof anything, since there is no definition of how far into the future you plan on prepping for. Best idea is to tell us what you currently have and what its use is at the moment, and we can then offer ideas about some areas that might need an upgrade and other areas that can be left alone.
    • I can agree that it is being used in a small capacity. I worked for a company where their engineers still used XP, and when asked why it was because their sensor software wasn't compatible with newer operating systems and the software was discontinued so they couldn't upgrade the software. Given that the sensors were still in use by companies, they had to continue using XP to support the sensor, otherwise the price to the company would have gone into the millions or billions. Our response was simple: Ok, you can keep the XP machine. But we're removing it from the network. "But then it can't access the Internet or folder shares!" Yup, kinda the point. If someone wants to continue using an unsecure OS they can do, I have no problem with that. But it should be isolated. Simple. I had a fight with a guy in the engineering department for weeks before he finally relented. But we digress.   What do I plan on doing to commemorate the anniversary? Nothing. I have fond memories of the OS, but at the end of the day it's just an OS. If I had some time I might see if I could install it on my Raspberry Pi for a laugh. But my reflex memory with today's OS ideas would probably get me frustrated and I'd uninstall it after 5 mins.
    • Shutter Encoder 20.2 by Razvan Serea Shutter Encoder is one of the best video converter software and image, audio available today. It has been designed by video editors in order to be as accessible and efficient as possible. It is one of the few free professional tools. Based on FFmpeg, it has the largest codec library available. You can thus convert your files into many different formats. Complete settings for the most advanced Shutter Encoder has a panel containing a large number of settings, in order to define your own choices based on your files and perfect your video or audio output. Well-thought-out settings, with parameters predefined to create files quickly and easily. List of functions Without conversion: Cut without re-encoding, Replace audio, Rewrap, Conform, Merge, Extract, Subtitling, Video inserts Sound conversions: WAV, AIFF, FLAC, MP3, AAC, AC3, OPUS, OGG Editing codecs: DNxHD, DNxHR, Apple ProRes, QT Animation, GoPro CineForm, Uncompressed YUV Output codecs: H.264, H.265, VP9, AV1, OGV Broadcast codecs XDCAM HD422, AVC-Intra 100, XAVC, HAP....and much more. Shutter Encoder 20.2 changelog: Added "Intel Quick Sync" hardware acceleration for Linux Added 'Identify speakers' option for "Audio transcription" function Improved installer package Improved video player performance Improved timecode display with drop-frame videos Improved naming convention for surround audio files Fixed splash screen freeze Fixed bug with file hanging Fixed bugs with presets loading Fixed bugs with video player's buffer Fixed bug with 'Total length of file' option Fixed bugs with 'Record screen/device' option Fixed bug with "XAVC" & "XAVC Long GOP" functions Rollback to XPDF tool for PDF conversion Removed unused binary architecture for Mac Various corrections Various improvements Download: Shutter Encoder 20.2 | 166.0 MB (Open Source) Download: Shutter Encoder Portable | 185.0 MB Links: Shutter Encoder Home Page | FAQ / Tips | macOS | Screenshot Get alerted to all of our Software updates on Twitter at @NeowinSoftware
    • did you give it permission to do so? its probably on the ToS. After that South Park episode I'm paying attention to them lol
  • Recent Achievements

    • Reacting Well
      NovaEdgeX earned a badge
      Reacting Well
    • Week One Done
      NovaEdgeX earned a badge
      Week One Done
    • One Year In
      BA the Curmudgeon earned a badge
      One Year In
    • Conversation Starter
      rosiecharles earned a badge
      Conversation Starter
    • First Post
      KMilenkoski1202 earned a badge
      First Post
  • Popular Contributors

    1. 1
      +primortal
      534
    2. 2
      +Edouard
      265
    3. 3
      PsYcHoKiLLa
      152
    4. 4
      Steven P.
      99
    5. 5
      macoman
      66
  • Tell a friend

    Love Neowin? Tell a friend!