main

Failed Windows XP Upgrade Downs 60,000 UK Gov't PCs

malebolgia   on 28 November 2004 - 03:57 · 62 comments & 6640 views

Advertisement (Why?)
Most of the desktop computers in the UK's Department for Work and Pensions were paralyzed for four days on Monday, when a failed upgrade took them offline. The outage, covering 75 percent to 80 percent of the DWP's 80,000 PCs, is one of the largest in the UK government's not entirely impressive IT history.

And possibly one of the most costly. According to staff reports, the outage occurred on Monday afternoon, disconnecting staff e-mail, benefits processing, and Internet and intranet connectivity. According to one, a limited network upgrade from Windows 2000 to Windows XP was taking place, but instead of this taking place on only a small number of the target machines, all the clients connected to the network received a partial, but fatal, "upgrade."

News source: eWeek


Build 11 24 November 2004

* Made code to start/stop programs and services multi-threaded for faster execution
* Added a few more descriptions for possible error messages when starting/stopping services
* Fixed bug that caused crash in GUI when loading a profile with a filename longer than 20 characters
* Added auto-arrange to the main profile window (icons will arrange themselves as the window is resized)

Post a comment · Send to friend Comments · There are 62 additional comments
#1 jagedEdge on 28 Nov 2004 - 04:00
I think some water just came out of my nose from laughing.
(3 replies) #2 figgy on 28 Nov 2004 - 04:01
I read somewhere eles that the support guys deployed Windows XP files on Windows 2000.
The patching systems replaced some of the Win2K dlls with XP dlls in effect bringing it down.
#2.1 configure on 28 Nov 2004 - 09:02
So I've read too but doesn't the patch installer supposed to check for the compatibility anyhow? :/
#2.2 nemesis89 on 28 Nov 2004 - 11:17
no it was on windows xp....any windows xp upgrade checks to see if your computer is running windows xp...or else it wont install...unless u open the exe using winrar and do a manual upgrade
#2.3 configure on 28 Nov 2004 - 12:45
But didn't the patch installer itself has compatibility check feature?
(2 replies) #3 virtorio on 28 Nov 2004 - 04:04
Sounds like the administrators should have been more careful. That'll teach them.

I don't think Microsoft can be held responsible for this, yet I got the feeling that the person who wrote the article wanted us to think that.
#3.1 thebuggerguy on 29 Nov 2004 - 07:07
Exactly -- human error. Stupidity, if you will.

Everyone always blames the computers when something goes wrong
#3.2 shao on 29 Nov 2004 - 09:35
i'm sure in this case they'll be blamin the outsourcing company.
(6 replies) #4 Sn1p3t on 28 Nov 2004 - 04:15
Yeah, this article was posted on Slashdot a little while ago. The engineers made the mistake.

Removed by author to reduce further flames. Sorry guys, didn't mean it!

Last edited by 44562 on 29 Nov 2004 - 09:03
#4.1 [Fosters] on 28 Nov 2004 - 04:18
You may be right or wrong.
But you are feeding forder to trolls.
#4.2 configure on 28 Nov 2004 - 04:43
Some of the bigger Linux distribution offers support solutions :/
#4.3 Darkinspiration on 28 Nov 2004 - 07:09
might i remind the << removed >> that 1: linux as many support service redhat, mandrake, suse just to name a few. 2: linux is not considered on this article it's was a admin error he forgot the golden rule always test the patch before deployment 3: the same mistake could have appened in linux 4: usualy the fan base linux support like forum and the like is friendly and useful you just have to go to good places ... like not here where your support is something like try XXX distro that's what i use .... etc ...




Last edited by 36818 on 28 Nov 2004 - 14:00
#4.4 Axon on 28 Nov 2004 - 08:50
By the sounds of if, they were trying to test it on a select few machines, but even the test errored.
#4.5 markjensen on 28 Nov 2004 - 14:06
Since Sn1p3t brought an unnecessary reference to Linux here (I think the term is trolling) with this comment:
I was telling my wannabe Linux fanboi that since they have a real company backing them (rather than downloading a Linux distro), they'll get real support, rather than having to post on a forum.
I might take the time to point out that as was pointed out before, Linux comes with as much support as Windows.

And, as a reminder: yes, you can get warned for trolling in the news section of Neowin, too.
#4.6 cheesegoduk on 28 Nov 2004 - 15:04
I've read that they tested the update out on a few machines, it didn't work and then some plank deployed the same update to every machine in the network.......
#5 the_atheist_prophet on 28 Nov 2004 - 04:28
It's not funny when you're relying on the benefits system!
(4 replies) #6 PseudoRandomDragon on 28 Nov 2004 - 04:35
Obviously this is the fault of stupid admins. Always, ALWAYS test an update before deploying it on that many computers!!!
#6.1 cheesegoduk on 28 Nov 2004 - 15:04
they did, The update didn't work but they deployed it to the whole department anyway :/
#6.2 PseudoRandomDragon on 28 Nov 2004 - 17:51
That is beyond dumb.
#6.3 thexfile on 28 Nov 2004 - 19:58
so much for the read me files...
#6.4 shao on 29 Nov 2004 - 09:37
this didn't work BECAUSE they rolled it out to the wrong pc's.
no doubt the 7 they were testing deployment to worked fine.
(8 replies) #7 dougkinzinger on 28 Nov 2004 - 04:35
Sounds like they didn't perform the quickie upgrade checklist: Ad-aware, BIOS update, antivirus update.

--DK
#7.1 configure on 28 Nov 2004 - 04:45
What you just listed is totally irrelevant to the issues that the UK's DWP are having.
#7.2 Knight' on 28 Nov 2004 - 10:35
DK that was an amazingly dumb answer.

I just wish our government IT systems weren't so dependant on Microsoft software, far too costly in my view, and updating kernel/librarys/software should not cause the system to crash!

Last edited by 20986 on 28 Nov 2004 - 10:41
#7.3 nemesis89 on 28 Nov 2004 - 11:16
read the god dam article! ITS NOT MICROSOFT FAULT. just cause the people don't know that installating an upgrade that is not even fully downloaded is not a smart thing to do, its their fault not microsofts!

QUOTE
all the clients connected to the network received a partial, but fatal, "upgrade."
#7.4 wildk on 28 Nov 2004 - 12:05
It is not microsofts fault, and wtf the has linux got to do with this, the blame soley lies with EDS who are utter icompetents, having dealt with them for a system we run at work, at they cocked the ugrade of that up as well, However it was left to my IT Dept to rectify the situation after they nuked the server then denied all responsibility for it, The governements problem is its over reliance on EDS who i would no trust to support a calculator!!!!
#7.5 markjensen on 28 Nov 2004 - 13:59
wildk, you were the only one to mention Linux in this thread. If your rage against Linux was about comments in thread #4, then your post should have gone there.
#7.6 wildk on 28 Nov 2004 - 21:02
Actually Mark, I did not rage about nor complain about it, I actually use Linux myself as well as windows, I am just tired of the constant bickering the moment any thing to do with windows is mentioned as in "OMG Windoze suxorz use l33t Linux".

QUOTE
I just wish our government IT systems weren't so dependant on Microsoft software, far too costly in my view, and updating kernel/librarys/software should not cause the system to crash!


If that was not an example of advocating the use of OSS software!!, Unfortunatly government departments WILL NOT switch to linux because it does not meet their requirments for security, adaptability and the support issues involved would be far greater than using micorsoft software. Yes i know that you can get support with one of the major distros, but when your IT Dept (7 technicians) looks after some 3000 pcs all over the country, we want the easiest solutions to provide support to our end users (i.e. Technet and the corporate support we have with microsoft). To me this is why Linux has not taken off with the home user nor OEM's as it would be a knight mare to support. So don't be to quick to jump on people just because linux was brought up (I believe you are a supporter of linux, was this a knee jerk because you thought i was knocking it, Far from it!) i was trying to point out that the fault lies with EDS as they have screwed up many of the contracts they have, look at the Child Suport Agency, Dept of Work & Pensions, the passport office, etc
#7.7 markjensen on 28 Nov 2004 - 22:02
My only comment was that you started asking what Linux had to do with it when it wasn't mentioned. Linux isn't the only OS with a kernel, some libraries, and software. Windows has their NTOS kernel, dll libraries and plenty of software, as does all major OSes I can think of.

I just thought your comment may have been sort of for section 4, not 7...
#7.8 Treefrog on 29 Nov 2004 - 23:37
QUOTE
Sounds like they didn't perform the quickie upgrade checklist: Ad-aware, BIOS update, antivirus update.


Somehow I don't think that would make a whole lot of difference when you roll an XP patch out on a win2k machine.
(1 reply) #8 imtoomuch on 28 Nov 2004 - 04:51
Sounds like it's time for a new IT staff.
#8.1 Samoa on 28 Nov 2004 - 22:16
That's what I'm thinking. These people are idiots...
#9 winmacguy on 28 Nov 2004 - 05:04
In any situation no matter what OS your using be it MS, Linux or OSX you ALWAYS makesure that your test machine is isolated from the rest of the network so that you can fully check out any upgrade or new install and test it before fully deploying it.
(5 replies) #10 Budious on 28 Nov 2004 - 05:20
What was the benefit of going to XP from 2000 for the average work place user?. Even if it had been successful upgrade it would have still been a waste of cash. Money better spent elsewhere...
#10.1 SquareSoft0 on 28 Nov 2004 - 05:53
Security , and I take no weight to the opinions of trolls. (For anyone responding)
#10.2 Darkinspiration on 28 Nov 2004 - 07:10
i don't think that this was supposed to be a complete upgrade rather a patch that whent really wrong
#10.3 Mando on 28 Nov 2004 - 13:21
err security, the fact that W2k support is DROPPED this year.

theres 2 valid reasons for it and besides the GOVT have an software assurance from MS (being a big customer).

The upgrade of licences from their W2k to XP doesnt cost them more than their annual subs to MS.

And for the rest who say oooh they should have tested it.
They did you knobbers if you read the actual article befor eposting it you would see that.

Techies did what they were asked , it was the boffins sitting in his office that decided do them all (prolly a Degree Grad in 1st week of post over long term techies, who have the practical experince hehe)


#10.4 Budious on 28 Nov 2004 - 17:16
Ok, but security wise 2000 is pretty up to date with SP4 and post updates, internet traffic should be limited and is probably on a AD system with 2003 servers, that should be cutting most of that potential danger off at the server level. So they have a special deal with MS that costs them very little in the long run to upgrade, alright but why not wait until 2000 is officially dropped before rolling out the software. Im just going by the base rule of if its not broken, don't fix it. What does my opinion have to do with me being a troll? I thought that title was reserved for MS sucks and IE sucks fanboys.
#10.5 SquareSoft0 on 29 Nov 2004 - 02:14
I didn't say you were a troll, I said that I wouldn't use any opinions expressed by trolls as input. Plain fact is, XP is more secure than 2000. Whether they've been hit with exploits or not it's just common sense to move to whichever version is more secure.

Last edited by 46870 on 30 Nov 2004 - 08:51
(2 replies) #11 DrIndianaJones on 28 Nov 2004 - 06:18
Perhaps the headline should be:
UK Gov't Admin Team Mistakenly Rolls-out Particial WinXP Upgrade Downing 60,000 PCs

The current headline makes it seem that it was Microsoft's fault...they had nothing to do with it.
#11.1 Samoa on 28 Nov 2004 - 22:18
I agee
#11.2 uceboyx on 28 Nov 2004 - 23:21
but it was caused by a microsoft's software anyway

it's a shame for UK though...
#12 greg098 on 28 Nov 2004 - 06:29
i read this earlier on another site...that is pretty bad...i bet some people are really pissed off
(3 replies) #13 leesmithg on 28 Nov 2004 - 10:29
Typical of the government plastic dummies waiting till this time of year to do such a stupid thing.

They should employ people with out a cabbage for a brain.

I don't understand why they need to go from 2k to xp. They still use nt4 on atm machines, what a joke.
#13.1 wildk on 28 Nov 2004 - 12:10
Read the article, IT WAS EDS ENGINEERS. EDS Carried out the upgrade and this is so typical of the way EDS Work, I know fro personal experience having had to deal with them. Next time atucally think about what you are posting, try to exercise the cabbage you got in the bargin basement.
#13.2 leesmithg on 28 Nov 2004 - 13:58
atucally what?
#13.3 wildk on 28 Nov 2004 - 21:06
Typing on an unfamiliar keyboard, That OK with you!!!

#14 deadmonkey on 28 Nov 2004 - 10:31
If people RTFA you would see it was a PAPERWORK problem. The admin's did everything correctly just they were asked to do the wrong thing. The plan was to upgrade 7 (or around that number) systems to Windows XP as part of a trial however someone sent paperwork to upgrade ALL systems to Windows XP. During the upgrade it was cancelled and the systems were left unable to boot so they couldn't, easily, restart the upgrade again to either roll it back or continue the upgrade.

Everyone is pointing the finger at EDS as they normally cock things up pretty good but this isn't really their fault. They were asked to upgrade the systems and they did, then they were asked to cancel it which they did (to be honest they shouldn't have done this really).
#15 CubanPete on 28 Nov 2004 - 11:08
Hehe, Why am i not surprised, our government always mess everything up!
#16 M2Ys4U on 28 Nov 2004 - 12:42
Nice one DWP!
#17 jivemastert on 28 Nov 2004 - 14:07
oh right, any IT guy should know that stopping an upgrade like that mid way though is going to trash the system. though they were told to cancel it, they shouldnt have, thats all there is to it. its better to have to fix a bunch of computers having problems with the upgrade than having to redo all of the computers that go destroyed by canceling the upgrade. time to break out the computer image cds!
#18 Shining Arcanine on 28 Nov 2004 - 15:47
I wish they would have the sucessful upgrades in the news. Anyway, I suppose this would be a good reason to fire their system administrators.
(1 reply) #19 Hills420 on 28 Nov 2004 - 15:51
Upgrade 2000 to XP? I'd probably just reformatted them and installed it from scratch. I've never trusted unattended installations.
#19.1 Samoa on 28 Nov 2004 - 22:19
it wasn't an upgrade really, rather an mispathcing of the files...
#20 EduardValencia on 28 Nov 2004 - 16:18
oh man,hopes we don't have hordes of trolls

it says that it WASN'T MICROSOFT FAULT
#21 nw_raptor on 28 Nov 2004 - 17:43
lol... way to go!
#22 bucko on 28 Nov 2004 - 22:41
Downs 60,000 that would not look good on your portfolio folder
(1 reply) #23 Daugirdas on 28 Nov 2004 - 22:56
Why the hell some PEOPLE are so STUPID, the ones who don't have any brain or the ones who employ 'em...?
#23.1 configure on 29 Nov 2004 - 08:10
People make mistakes, the quicker you can get over the fact that mistakes has happened, the quicker you can figure out what you can do to fix it.
#24 hardgiant on 29 Nov 2004 - 00:08
Pilot error
#25 Eversurf on 29 Nov 2004 - 02:58
Talk about a bad at work
#26 Xion on 29 Nov 2004 - 10:57
Yep, it was quite funny as I was in my local jobcenter the other day when they got affected. Not the first time BTW..
What I found hilarious is that I been out of work for the last year with 10+ years experience & an MCSA in the IT field.

Commenting has either been disabled on this article or you are not logged in. Click here to login or register, its free!

Note: Anonymous commenting is disabled in order to keep the quality of responses to a high standard.

Advertisement (Why?)