Whoops! Emory University server sent reformat request to all of its Windows 7 PCs

Sometimes, there are incidents that take place that remind people who use PCs to back up their files on a regular basis. Such an event happened earlier this week at Emory University in Atlanta, where an "accident" resulted in a server sending out a reformat request to all of the Windows 7 PCs at the school, including the server that sent out the request itself.

The incident, which happened sometime on Wednesday, was reported on the school's website, which added that efforts to restore all the school's PCs were taking much longer than anticipated due to the main server being reformatted. The site stated:

So, we were without our preferred methods for deploying images to desktops/laptops all yesterday and relied on older methods – USB + Ghost, LANDesk (we still had our old LANDesk server) + PXE. These methods required a lot of manual work plus our success was uneven with them.

The site has since posted a number of updates on the progress to restore all of the school's PCs. The last posts were made on Friday, which stated that the university's SCCM tool was now fully operational and as a result "things are moving at a much faster pace now" to get every Windows 7 computer restored. There's no word on how many PCs were affected, nor has a reason been issued on why the school's servers sent out the reformat request in the first place.

Source: Emory University via The Next Web | Image via Microsoft

Report a problem with article
Previous Story

Google buys company behind Word Lens photo translation app

Next Story

Microsoft now rolling out May Xbox One system update to all consoles

53 Comments

Commenting is disabled on this article.

Does anyone else find it ironic that they Emory has an IT "Service Management Competency Center?"

With great ease comes great responsibility. Proper network design is about the only way to technically prevent this from happening. Also, we make collections specifically for OS deployment and only deploy to those collections. Generally, our deployments only change a few times throughout the year. We just add machines as needed to our deployment collections and remove them when we are done.

It is a good practice in Config Man to use separate collections for different purposes where possible. For example, use separate collections for maintenance windows that are named to indicate what their purpose is and separate collections for deploying updates. One of the departments where I work (university with 20k students) refused to follow our advice and inadvertently rebooted all their user machines for two months in a row in the middle of the day while installing updates because they used multiple maintenance windows applied to multiple, non-descript collections. One maintenance window was supposed to be a one-time deployment of a critical out-of-band patch, but was actually set to re-occur every month and was never deleted. They essentially blamed me for two months as I work on the team that runs the config man environment for the university. I eventually had to create a report that showed all the collections a machine was a member of and then looked at each collection till I found the problem for them. Oh yeah, and double and sometimes triple check your work.

We have been using SCCM for image deployment for about 4 years with great success. We have about 3,000 devices. Interesting thing is that thru a simple setting, this could have been prevented. It would have caused a little more work on each deployment, but a lot less than what they are doing now. I set our boot disk within SCCM to only deploy to unknown computers. As you can see now, this is a very powerful feature. If the MAC address of the target computer is in the SCCM database, it will not get the task sequence. I created a seperate collection for each of our remote sites granting the local system admin at that site access to it. When they need to re-image a computer, they simply go to their collection and delete the comptuer record and they can then re-image. If they dont, then that computer does not get the task sequence. I find it hard to believe that this was not in place. Either that, or someone deleted all the comptuer accounts from the All Systems Collection. Either way, a pretty bonehead move.

Yep, I took the Microsoft SCCM course last month and the guy sitting next to me had exactly the same horror story only for a larger organisation: over 10,000 workstations, 900 servers, all got Windows 7. Too easy!

SCCM is a powerful tool, and I can easily see how this could happen. If configured properly (it takes a lot of planning, skill, time, testing, and effort) you can do zero touch deployments and people will never know their machine got reimaged. If you are not an SCCM and deployment expert, well its also easy to hit the wrong button and reimage everything. Oops. I managed one of these systems for a number of years and can say that it is VERY easy, almost TOO easy to send out an advertisement to the wrong group.

I'd love to see the task sequence they were using for this. A typical refresh task sequence should also use USMT to perform a system state backup to the State Migration Point. It would be easy enough to simply push a mandatory task sequence that immediately reboots endpoints into WinPE but if the task sequence actually works, it would apply the image. If that were the case, there would be no need to reimage machines using your old version of LANDesk, Ghost and so on. That suggests that this task sequence wasn't production ready. More than likely, they were trying to target a small subset of machines by filtering it out of the All Systems collection. Even when you select the target machine using the filter, there are two options in the right pane. One of them allows you to target the selected item but the other allows you to target the whole collection. Mandatory pushes that do not require end user intervention are terrifying for this very reason.

This is not all that uncommon.... I've seen other huge enterprises where admins have targeted the wrong collections using SCCM and pushed out images to entire sites when they only meant to push it to a small group of systems. Not very hard to make such a mistake if you don't understand the tool and the concepts of collections.

Zero touch deployments and upgrades of OS can be scripted via task sequences. So there won't be any admin/user interaction, it will just get scheduled to happen and when that time goes its going to do what its' been instructed to do on the systems that were targeted without asking any questions.

pupdawg21 said,
Zero touch deployments and upgrades of OS can be scripted via task sequences. So there won't be any admin/user interaction, it will just get scheduled to happen and when that time goes its going to do what its' been instructed to do on the systems that were targeted without asking any questions.

Yup, as the old saying goes, "It'll only do what you tell it to!"

this is why I always laugh whenever someone says "I'm a IT professional and I will never let my organization upgrade to Windows 8!!!" because the organization they're working for are probably already suffering a fate worse than Windows 8 anyway.

FalseAgent said,
this is why I always laugh whenever someone says "I'm a IT professional and I will never let my organization upgrade to Windows 8!!!" because the organization they're working for are probably already suffering a fate worse than Windows 8 anyway.

No fate was worse than Windows Malignant Edition!! Or Windows Multiple Errors! Windows ME to laymen :)

which IT pro would actually make their organization upgrade to Windows ME? Windows ME was meant for home use, and the enterprise was supposed to be running Windows 2k, which was built off Windows NT for stability and reliability.

Spoiler: Windows 2k was awesome.

FalseAgent said,
which IT pro would actually make their organization upgrade to Windows ME? Windows ME was meant for home use, and the enterprise was supposed to be running Windows 2k, which was built off Windows NT for stability and reliability.

Spoiler: Windows 2k was awesome.

I wasn't on about businesses running ME and am fully aware of 2000, used it for years as Advanced Server. My point was that ME is a MUCH more horrible fate than Win 8.

Didn't the program warn admins beforehand?:
'This procedure will wipe all physical drives on the network. Proceed? (y/n)' y
'AGAIN, DATA WILL BE LOST AND UNRECOVERABLE! PROCEED? (y/n)'

68k said,
Didn't the program warn admins beforehand?:
'This procedure will wipe all physical drives on the network. Proceed? (y/n)' y
'AGAIN, DATA WILL BE LOST AND UNRECOVERABLE! PROCEED? (y/n)'

You would think so, wouldn't you!

cork1958 said,

You would think so, wouldn't you!

I'm sure if CloneZilla can confirm two to three times before formatting drives, SCCM should be able to!

You should ONLY be able to hit your WDS environment from a deployment VLAN that requires the switch ports to be set specifically. This is some boneheaded #### on an unparalleled level. They need an architecture consultant to come in and recommend some sweeping infrastructure changes.

blaktron said,
You should ONLY be able to hit your WDS environment from a deployment VLAN that requires the switch ports to be set specifically. This is some boneheaded #### on an unparalleled level. They need an architecture consultant to come in and recommend some sweeping infrastructure changes.

While I agree with you for the most part, it would require an additional level or levels of admin to start VLANing the environment off, and depending on the complexity of the environment it might be something the school/college wants to avoid. If its SCCM, I am sure you know you can have PXE enabled without risk of your clients getting reimaged without admin sending the request from SCCM.

Still amusing though, although I would hate to be the guy to have to live that mistake down. At least the environment should be "Computers can be reimaged at any time" so users are told not to save to the local pc. Also, GPOs should redirect personal doc and desktop to network share and local disk locations hidden and locked down.

blaktron said,
You should ONLY be able to hit your WDS environment from a deployment VLAN that requires the switch ports to be set specifically. This is some boneheaded #### on an unparalleled level. They need an architecture consultant to come in and recommend some sweeping infrastructure changes.

Sounds like you have lots of experience from enviroments with more than one physical site. The ability to reinstall at the users office is necessary when you have many (ex. 10+) locations and a lot of devices. A deployment VLAN(as the only alternative) will become a great solution when switch management(changing VLAN) can easily be scripted by only supplying the mac-address of the device.

Most companies configure SCCM to re-image machines at their users location. With that amount of computers and remote locations you cannot narrow it down and keep it practicable.
There are many ways of preventing accidental deployments like this one. However, SCCM is a powerful tool, and allows you to do pretty much everything.
Just google "commonwealth bank australia sccm" and you can see where even in a professional environment a simple mistake can kill an entire bank within minutes.

Hmmm.. What does this button do.. This shiny red big button with the text 'DO NOT PRESS'... :D

Sounds like someone made a booboo.. servers do not 'just'go about telling all desktops to format and then format themselves.

c.grz said,
And that is why we disable PXE boot on all critical servers and workstations!

SCCM doesn't require PXE to work. This deployment can start from Windows and switch to WinPE, no PXE necessary.

siah1214 said,

SCCM doesn't require PXE to work. This deployment can start from Windows and switch to WinPE, no PXE necessary.

It does sound a lot like an SCCM scenario, it did the thing you didn't want it to do with a very high success rate.

I don't enable PXE on server, but workstations, sure. You can set a lab to image over the weekend with no intervention from anyone. That's what PXE is for.

Buendia said,
How is that even possible for a computer to format itself?

very easy, depending on how the request was sent out, the server could have sent itself a reboot command and proceeded to pxe boot to the deployment image.

I can only laugh. I went to Emory and their IT department was generally good. For something to happen like this is embarrassing, but for a person who hasn't been there in over ten years, rather funny too lol.

macrosslover said,
I can only laugh. I went to Emory and their IT department was generally good. For something to happen like this is embarrassing, but for a person who hasn't been there in over ten years, rather funny too lol.

it's more embarrassing that they don't seem to have a clue that it's very possible to restore formated partition and it's usually much faster than reinstalling everything from scratch, not to mention all the data returns in it's place.

x.iso said,

it's more embarrassing that they don't seem to have a clue that it's very possible to restore formated partition and it's usually much faster than reinstalling everything from scratch, not to mention all the data returns in it's place.

Hahaha. You speak like someone that's never worked in an enterprise environment.

siah1214 said,

Hahaha. You speak like someone that's never worked in an enterprise environment.

I never worked as admin in enterprise environment. But I did recover formated or lost partitions many times with different circumstances. Usually the trick is just about recreating MBR.

I think that you're referring to the fact that it may took more time to recover partitions from PC to PC than mass deploy a new OS via Server, but they at least could recover the server partitions and then the question is just how valuable could be the data that is lost.

Edited by x.iso, May 17 2014, 5:50pm :

x.iso said,

I never worked as admin in enterprise environment. But I did recover formated or lost partitions many times with different circumstances. Usually the trick is just about recreating MBR.

I think that you're referring to the fact that it may took more time to recover partitions from PC to PC than mass deploy a new OS via Server, but they at least could recover the server partitions and then the question is just how valuable could be the data that is lost.


You cannot just 'recover a partition', it depends entirely upon how it was deleted. You can only recover a partition if just the MBR and/or partition table was erased, anything more than that and you'd have to load in a specialist recovery program to recover individual file/folder contents.

If they had a server image then they could restore it. Otherwise, a clean install is needed, with a multiple external drive backup system where one of the backup drives is always offline.

n_K said,

You cannot just 'recover a partition', it depends entirely upon how it was deleted. You can only recover a partition if just the MBR and/or partition table was erased, anything more than that and you'd have to load in a specialist recovery program to recover individual file/folder contents.

Yes it does depend, but usually if it's just a format, then usually only MBR is affected and it can be reconstructed. And of course you need a specialized recovery tools for any type of data recovery.

x.iso said,

Yes it does depend, but usually if it's just a format, then usually only MBR is affected and it can be reconstructed. And of course you need a specialized recovery tools for any type of data recovery.

There's no mention of what software was used for the format, and I'd imagine that most enterprises (and large educational institutes) would be using secure erase programs, nothing quick/sloppy.
You don't need specialised tools for all recoveries, I can easily make a partition on linux, mount it and write files to it. Clear the MBR and I can no longer mount it, but the datas all still there, just need to use fdisk to recreate the MBR with the same start and end points and I'll be able to remount without any problems.
As these are all imaged computers, if it was that easy, I'm sure they'd have done that instead of spending a lot of time rebuilding and PXE installing, each PC would have the same sector sizes for each partition.

Dot Matrix said,
*Nelson_Muntz_HAHA.jpg*

In all seriousness, sounds like someone hit a wrong switch somewhere.

I'm not sure how they got that far without realizing what they were doing. Typical of IT departments in the education system. Lots of credentials little know how.

daorbed9 said,

I'm not sure how they got that far without realizing what they were doing. Typical of IT departments in the education system. Lots of credentials little know how.

Even if you do it on PXE installs you still need to confirm the re-format on every single PC... Something is fishy here.

Not necessarily. You can create completely unattended reinstall jobs in SCCM. It wouldn't even take much to make this mistake if you're a novice. This is why test environments should be used.