Wait 20 Minutes and Try Again or How I Almost Got Fired...


Recommended Posts

This is a copy / paste from Reddit, i thought it was a good read!

Sometimes fellow techs that makes you want to smash your keyboard. (all names have been changed to protect the incompetent).

I used to work in a Service-Desk/Triage capacity (lovingly dubbed Tier 1.9) where we would monitor, ticket, troubleshoot, and escalate as needed (we were expected to have a 90% self-solution return).

One thing that we never were allowed to touch was network issues. Our area telcom team had lost their extremely competent team lead some 18 months earlier and their team had quickly degraded to people who didn't really seem to care about their job. They were also secure in the fact that the telcom system had degraded enough to be sufficiently obfuscated enough where it would take considerable resources to figure it out by a competent replacement. This made them act like they were untouchable. As such, I had learned fast to start recording my phone calls to them (thank you Google Voice!) so they couldn't pin anything on us.

Anyway, one day around 6PM the network dropped out. As such, I called the Telcom On-Call.

Me Leaving Voicemail: Hi, this is Craysh from Tier 1.9. It looks like the network is having issues again. Internet, communicator, and email are all unresponsive. All three webfarms are unavailable. Trace route to the gateway has failed. Ticket is unavailable as the ticketing system is inaccessable due to the issue. Please call back to confirm your taking ownership of the incident.

15 minutes go by and SOP dictates that I call back before escalating to their manager. The on-call answers.

Tech: On-Call Phone, this is Steve

Me: Hi Steve, have you gotten my voicemail? We're having issues with the network.

Tech: No, I haven't. Tell me what's wrong.

Me: Repeats what I said above.

Tech: Ok. Wait 20 minutes and try again. Click

I had assumed that by 'Wait 20 minutes' he meant that he was taking a look at it and that it should be back up in 20 minutes. 20 minutes go by and the network is still having issues (The network has now been down for the last 40 minutes or so)

I call the On-Call phone again. No answer.

Me Leaving Voicemail: Hi, this is Craysh again. The issue is still ongoing. Please call me back as soon as possible.

15 Minutes go by so it's time to call again. No answer. Time to call the acting supervisor.

Supervisor: This is Aaron

Me: Hi Aaron, this is Craysh from Tier 1.9. We are having issues with the network and the On-Call has stopped answering their phone.

Supervisor: Stopped? You mean they initially answered?

Me: Yes, he said to wait 20 minutes.

Supervisor: OK, he's probably working on it. Call me back in an hour if the issue persists. click

At this point I'm calling my supervisor and apprising him of the situation.

For the next hour, I continue to call the On-Call every 15 minutes (as is procedure). No answer.

I call the supervisor again

Supervisor: This is Aaron

Me: Hi Aaron, this is Craysh from Tier 1.9 again. We are still having issues with the network and the On-Call still has not answered his phone.

Supervisor: OK, I'll take a look at it.

Turns out that Telcom had pushed a policy to the Junipers that had scrubbed the white lists (completely blocking all access to all ports).

All in all it took 5 hours to get everything back up and running. I documented the ticket extremely thoroughly once the system was available again.

The next day came the fall-out. Telcom was claiming that I had never paged them, and they said that a customer ended calling the supervisor to find out what was going on. They even showed zero calls on their cellphone (they had removed the calls and messages).

As such, we had a meeting to 'discuss Tier 1.9's failure to report a severe incident'.

I walked into the room filled with extremely ****ed off big wigs, including Aaron and and Steve. They demanded why I hadn't contacted the on-call, and I stated that I had and that they had lead me to believe that they had been working on the issue.

Nobody believed me because "The On-Call phone didn't have any messages or call logs from Tier 1.9" Their excuse for having a completely empty call log was that they cleared it daily so they could keep track of the calls made on the phone.

I calmly walked to the overhead projector, logged into Google Voice, and played each consecutive voicemail/conversation.

They no longer work here, and I've had three promotions since.

Edited by John S.
properly tagged
  • Like 1

Good stuff.

I was once on the phone to Vodafone on behalf of my girlfriends mother whose contract had finished but she was still getting charged.

I recorded everything they said as i knew it was going to be a pain in the arse.

I was completely right and a dozen phone calls later no one at the company admitted to any call logs of previous correspondence between myself and Vodafone.

After playing back part of my conversation to the Indian chap on the phone, i was forwarded to another Indian chap and then another one and eventually ending up with a well spoken English chap who solved all my issues, closed the account, paid back all money owed and compensated us to the tune of ?150 for wasted time.

Sad fact is i had to go through all that for a simple account closure. There is no accountability any more.

Wow. Awesome story with a good ending. Incompetent people should realize that eventually their foolishness will catch up and bite them in the butt. Good to hear you've been promoted since then too! It sounds like management know when they've got someone who can do good work.

This topic is now closed to further replies.
  • Posts

    • Good think I still have SDRAM and FP RAM sitting around.
    • Fitbit Charge 6 fitness tracker with Google apps is now at its lowest price with 47% off by Fiza Ali Amazon is currently offering the Fitbit Charge 6 fitness tracker at its all-time low price with a 47% discount. The device features an AMOLED touchscreen display protected by Corning Gorilla Glass 3 that should offer improved scratch resistance and durability. The Charge 6 is equipped with a range of sensors including an optical heart rate sensor, a 3-axis accelerometer, built-in GPS with GLONASS support, red and infrared sensors for SpO2 monitoring, a skin temperature sensor, an ambient light sensor, a vibration motor, NFC, and multipurpose electrical sensors compatible with the ECG and EDA Scan apps. Heart rate is recorded every second during exercise tracking and every five seconds during normal daily use. The device requires the Google Health app for setup and synchronisation. Furthermore, Bluetooth provides wireless connectivity for syncing and communication with devices running Apple iOS 16.4 or later and Android 11.0 or later. The tracker stores up to 7 days of minute-by-minute activity data and retains daily activity totals for the previous 30 days. In terms of water resistance, the Fitbit Charge 6 has a 5 ATM rating that should make it suitable for swimming and water activities. The tracker operates in temperatures ranging from 14°F to 113°F and at altitudes of up to 28,000 feet. Moreover, the included Infinity band is made from a flexible silicone material and features a loop-and-peg fastening. The small band fits wrists measuring 5.1 to 6.7 inches, while the large band fits wrists measuring 6.7 to 8.3 inches. Both small and large bands are included in the box. When it comes to battery performance, the Fitbit Charge 6 should deliver up to 7 days of battery life under typical usage conditions. Features such as the Always-On Display, built-in GPS, and SpO2 monitoring increase power consumption and may require more frequent charging. The rechargeable lithium-polymer battery should take approximately two hours to charge from empty to full. Fitbit Charge 6 Fitness Tracker with Google Apps: $85.45 (Amazon US) - 47% off Good to know This Amazon deal is U.S. specific, and not available in other regions unless specified. We only use first-party seller links (at the time of article publishing); ensure that you purchase from a first-party seller link only. Check out Today's Deals on Amazon | or our recent tech deals. Become a Prime member (for Students or SNAP) via Neowin Get Prime Access - Prime for half price (for qualifying Medicaid, EBT, SNAP) Subscribe to Prime Video, Audible Plus, Music Unlimited or Kindle Unlimited via Neowin As an Amazon Associate, we earn from qualifying purchases.
    • google, meta, microsoft, true cancers of modern society
    • TeraCopy 4.0 Build 28 by Razvan Serea TeraCopy is a compact program designed to copy and move files at the maximum possible speed, also providing you with a lot of features. Copy files faster. TeraCopy uses dynamically adjusted buffers to reduce seek times. Asynchronous copy speeds up file transfer between two physical hard drives. Pause and resume transfers. Pause copy process at any time to free up system resources and continue with a single click. Error recovery. In case of copy error, TeraCopy will try several times and in the worse case just skips the file, not terminating the entire transfer. Interactive file list. TeraCopy shows failed file transfers and lets you fix the problem and recopy only problem files. Shell integration. TeraCopy can completely replace Explorer copy and move functions, allowing you work with files as usual. TeraCopy is free for non-commercial use only. For commercial use you need to buy a license. The paid version of the program includes the following features: Copy/move to your favorite folders. Save reports as HTML and CSV files. Select files with the same extension/folder. Remove the selected files from the copy queue. Features added since version 3.17: Enhanced speed graph. New multi-threaded copy engine. Support for copying to multiple targets. Queue system for managing multiple copy operations. Support for receiving files via the LocalSend protocol. TeraCopy entry in the modern Windows Explorer context menu. Integrated toolbar in the title bar. Why receive LocalSend transfers with TeraCopy? Handle file conflicts: Skip, overwrite, or rename files when a file with the same name already exists. LocalSend always creates another copy, which can waste time and disk space, especially when resuming an interrupted transfer. Filter unwanted files: Apply ignore lists or remove files manually before accepting a transfer, so unnecessary files are not downloaded. Better performance on fast networks: In tests over a 10 Gbps connection, TeraCopy received files several times faster than the standard LocalSend app on Windows. TeraCopy 4.0 Build 28 changelog: Fixed a bug where Overwrite behaved as Overwrite All during same-drive move operations. AdvancedInstaller fixed the installer’s security vulnerability: EXE Bootstrapper resolved the %appdata% location incorrectly for the System account. Download: TeraCopy 4.0 Build 28 | 14.6 MB (Freeware, paid upgrade available) View: TeraCopy Website | Screenshot Get alerted to all of our Software updates on Twitter at @NeowinSoftware
    • First exciting thing to come to Windows in a long time ! This is the kind of things they should focus on, instead of cramming as much AI as they can in everything.
  • Recent Achievements

    • Reacting Well
      NovaEdgeX earned a badge
      Reacting Well
    • Week One Done
      NovaEdgeX earned a badge
      Week One Done
    • One Year In
      BA the Curmudgeon earned a badge
      One Year In
    • Conversation Starter
      rosiecharles earned a badge
      Conversation Starter
    • First Post
      KMilenkoski1202 earned a badge
      First Post
  • Popular Contributors

    1. 1
      +primortal
      536
    2. 2
      +Edouard
      269
    3. 3
      PsYcHoKiLLa
      150
    4. 4
      Steven P.
      97
    5. 5
      macoman
      61
  • Tell a friend

    Love Neowin? Tell a friend!