Wait 20 Minutes and Try Again or How I Almost Got Fired...


Recommended Posts

This is a copy / paste from Reddit, i thought it was a good read!

Sometimes fellow techs that makes you want to smash your keyboard. (all names have been changed to protect the incompetent).

I used to work in a Service-Desk/Triage capacity (lovingly dubbed Tier 1.9) where we would monitor, ticket, troubleshoot, and escalate as needed (we were expected to have a 90% self-solution return).

One thing that we never were allowed to touch was network issues. Our area telcom team had lost their extremely competent team lead some 18 months earlier and their team had quickly degraded to people who didn't really seem to care about their job. They were also secure in the fact that the telcom system had degraded enough to be sufficiently obfuscated enough where it would take considerable resources to figure it out by a competent replacement. This made them act like they were untouchable. As such, I had learned fast to start recording my phone calls to them (thank you Google Voice!) so they couldn't pin anything on us.

Anyway, one day around 6PM the network dropped out. As such, I called the Telcom On-Call.

Me Leaving Voicemail: Hi, this is Craysh from Tier 1.9. It looks like the network is having issues again. Internet, communicator, and email are all unresponsive. All three webfarms are unavailable. Trace route to the gateway has failed. Ticket is unavailable as the ticketing system is inaccessable due to the issue. Please call back to confirm your taking ownership of the incident.

15 minutes go by and SOP dictates that I call back before escalating to their manager. The on-call answers.

Tech: On-Call Phone, this is Steve

Me: Hi Steve, have you gotten my voicemail? We're having issues with the network.

Tech: No, I haven't. Tell me what's wrong.

Me: Repeats what I said above.

Tech: Ok. Wait 20 minutes and try again. Click

I had assumed that by 'Wait 20 minutes' he meant that he was taking a look at it and that it should be back up in 20 minutes. 20 minutes go by and the network is still having issues (The network has now been down for the last 40 minutes or so)

I call the On-Call phone again. No answer.

Me Leaving Voicemail: Hi, this is Craysh again. The issue is still ongoing. Please call me back as soon as possible.

15 Minutes go by so it's time to call again. No answer. Time to call the acting supervisor.

Supervisor: This is Aaron

Me: Hi Aaron, this is Craysh from Tier 1.9. We are having issues with the network and the On-Call has stopped answering their phone.

Supervisor: Stopped? You mean they initially answered?

Me: Yes, he said to wait 20 minutes.

Supervisor: OK, he's probably working on it. Call me back in an hour if the issue persists. click

At this point I'm calling my supervisor and apprising him of the situation.

For the next hour, I continue to call the On-Call every 15 minutes (as is procedure). No answer.

I call the supervisor again

Supervisor: This is Aaron

Me: Hi Aaron, this is Craysh from Tier 1.9 again. We are still having issues with the network and the On-Call still has not answered his phone.

Supervisor: OK, I'll take a look at it.

Turns out that Telcom had pushed a policy to the Junipers that had scrubbed the white lists (completely blocking all access to all ports).

All in all it took 5 hours to get everything back up and running. I documented the ticket extremely thoroughly once the system was available again.

The next day came the fall-out. Telcom was claiming that I had never paged them, and they said that a customer ended calling the supervisor to find out what was going on. They even showed zero calls on their cellphone (they had removed the calls and messages).

As such, we had a meeting to 'discuss Tier 1.9's failure to report a severe incident'.

I walked into the room filled with extremely ****ed off big wigs, including Aaron and and Steve. They demanded why I hadn't contacted the on-call, and I stated that I had and that they had lead me to believe that they had been working on the issue.

Nobody believed me because "The On-Call phone didn't have any messages or call logs from Tier 1.9" Their excuse for having a completely empty call log was that they cleared it daily so they could keep track of the calls made on the phone.

I calmly walked to the overhead projector, logged into Google Voice, and played each consecutive voicemail/conversation.

They no longer work here, and I've had three promotions since.

Edited by John S.
properly tagged
  • Like 1

Good stuff.

I was once on the phone to Vodafone on behalf of my girlfriends mother whose contract had finished but she was still getting charged.

I recorded everything they said as i knew it was going to be a pain in the arse.

I was completely right and a dozen phone calls later no one at the company admitted to any call logs of previous correspondence between myself and Vodafone.

After playing back part of my conversation to the Indian chap on the phone, i was forwarded to another Indian chap and then another one and eventually ending up with a well spoken English chap who solved all my issues, closed the account, paid back all money owed and compensated us to the tune of ?150 for wasted time.

Sad fact is i had to go through all that for a simple account closure. There is no accountability any more.

Wow. Awesome story with a good ending. Incompetent people should realize that eventually their foolishness will catch up and bite them in the butt. Good to hear you've been promoted since then too! It sounds like management know when they've got someone who can do good work.

This topic is now closed to further replies.
  • Posts

    • Opinion. All you did was blame Democrats for everything. You offered nothing but a hit piece to support your pro Trump, anti union right wing ideology.
    • Excuse me for having an opinion, fella'... (Why am I not surprised?...) Congrats on your very informative post however...
    • By the sounds of that wall of Fox News propaganda gibberish attacking the Democratic Party you've already had plenty of "juices" flowing this morning. You've ruined what could have been a productive comment thread.
    • (Topic to get the juices flowing this Sunday morning!...) Actually, the situation has almost nothing to do with "lack of skills", especially since assembly-line skills can be taught to anyone, including Americans, certainly. Rather, the inadequacy-to-impossibility of large-scale tech manufacturing in America today, and the reasons why America finds tech manufacturing completely onerous in the 21st century, has to do with politically driven laws amid a plethora of non-scientific, utterly politicized "science-fact" that is patently false, punitive business taxation at every turn, an array of judicial fines of unimaginable scope and complexity, and, last but not least, American unionization strictures that serve to actually slay job creation and hobble all such manufacturing endeavors in America before they can get off the ground. Globalism emerged, they tell us, as the needed answer to American hubris and an unholy American drive to excel. Unless one is buried under mounds of political propaganda, it's easy to see the absurdity of labeling the employees of SpaceX, for instance, as "unskilled labor"... Etc. ad infinitum. At one time in the recent past, American manufacturing prowess was the envy of the world in a wide variety of technical fields! The current federal and state government roadblocks against America becoming competitive globally in tech manufacturing are considerable, it's true, as anyone with a working brain knows. But remarkably, that is only half the story! The other half of the story is, of course, the corporations themselves... Chinese tech manufacturing is simply unassailable in terms of profits, because the Chinese government wants to see its tech manufacturing second-to-none globally so that no companies/nations can compete in terms of ROI, and China has completely succeeded in that goal. Let's tic-off a few things: *Chinese tariff policies are set according to what is considered best for Chinese business, Chinese employees, and the Chinese people. Huge difference with how things are done with tariffs in the US--as the US government (SCOTUS in this case, Congress in others) plainly feels that tariffs are "unfair" for the limited number of citizens who may pay them, whereas nothing is "unfair" when Congress considers the Personal Income Tax rates to be infinitely hike-able, along with infinitely enlarging annual budget deficits. *The Chinese government boldly subsidizes Chinese companies to artificially amplify their profits. *The Chinese government deliberately refuses to avidly demonize Chinese businesses and does not consider Chinese businesses "the enemy", so very unlike American (D)s these days. *Chinese labor laws and businesses are allowed to set their own labor policies according to what Chinese companies consider is best for companies and their employees... Simply put, American workers in tech manufacturing are not allowed to set their own labor policies! * One additional problem corporations have that I also do not sympathize with is they don't want to pay to train their American employees. They could easily do so, but would rather not have to pay for it. I find that pathetic, actually. It is the height of hypocrisy for Americans to decry working conditions in China while simultaneously ensuring that American products are manufactured in China, not in the US, simply to maximize profits. There is nothing wrong with making a profit, of course, absolutely nothing. But there is plenty wrong with attempts to normalize hypocrisy of this kind! But rank hypocrisy and the (D) party in the US are longtime bedfellows... The current government in Washington is working overtime to see if it can toss out the horribly poor, failed economic policies of the past, while the (D)s still in Washington work very hard to bring back the stupidity whenever possible. With the right policies in place, America can be an infinitely competitive manufacturer.
  • Recent Achievements

    • Conversation Starter
      jessse3334 earned a badge
      Conversation Starter
    • Reacting Well
      JuvenileDelinquent earned a badge
      Reacting Well
    • One Month Later
      Excellence2025 earned a badge
      One Month Later
    • Week One Done
      Excellence2025 earned a badge
      Week One Done
    • Week One Done
      flexorcist earned a badge
      Week One Done
  • Popular Contributors

    1. 1
      +primortal
      508
    2. 2
      +Edouard
      198
    3. 3
      PsYcHoKiLLa
      152
    4. 4
      Steven P.
      73
    5. 5
      FloatingFatMan
      62
  • Tell a friend

    Love Neowin? Tell a friend!