Editorial

Google Binged us in the Yahoo so hard, we had to Ask Jeeves!

As you know, Neowin upgraded its backend Content Management System (CMS) on Friday January 15.

We also upgraded the forums to Invision Power Board 3. Unfortunately we had to roll back the changes a few days later due to extremely high loads on our server, specifically on our forum install. Our CMS (Ignition) remained online throughout the load issues and performed extremely well.

The same can't be said about our forum install though; each time we tried to place it back online it would spike with load forcing us to pull it offline again. One would have assumed this was the fault of Invision Power Board but it wasn't. We determined that the problem was spiders from Google, Bing and Yahoo were hitting us at around 10 times every second. When we upgraded to IPB3, due to our completely new URL structure, the spiders freaked out at all the new links. The spiders went into overdrive trying to re-index not only the 9 million posts we have, but also the news. Fortunately we do not have 9 million news posts so the news pages suffered less.

The idea of spiders causing the problem had been suggested before by a member of the Invision Power Services support team, but we never fully investigated it. After all, our spider traffic has always been really high. However, yesterday, in an attempt to do a load test, we put IPB3 onto our dev server and included it in an iframe on every page of the live forums. In non dev speak, every time someone loaded a page on the live site for a brief period yesterday, the development forums would get hit. Theoretically, that would recreate the load we saw over the weekend. However... our load stuck at around 0.3. This result puzzled us for a little while until we thought about it a little bit: spiders do not index iframes. While we had the same number of members on the live forums as the dev forums, the spiders weren't hitting it at all.

We knew we needed a solution, and we found one: tell the spiders to index less frequently. We amended what's called a "robots.txt" file to tell all search spiders to only index us a maximum of once a second. Our hits lowered, and we decided this might be the solution we were looking for: load was falling on IPB2, we couldn't imagine the effect it may have on IPB3.

This led to yesterday's successful load test. For 20 minutes, we put up a live version of IPB3, using the database we generated over the weekend. After Simon fixed a misconfiguration we had (which was causing the forums to stall and the load to spike initially), our loads fell. The forum was speedy. Essentially, we determined that it was fixed.

We have now successfully moved to Invision Power Board 3 and would like to thank you all for being extremely patient during last weekend's downtime. Neowin would also like to extend its thanks to Brandon from Invision Power Services who helped us determine the exact issue.

So this time, welcome to Ignition and Atlas :)

Neowin Developer Simon Andrews contributed to this report

Report a problem with article
Previous Story

Twitter users vulnerable to security flaw thanks to Adobe Flash

Next Story

What happens when you ask Steve Ballmer to sign a MacBook?

144 Comments

Commenting is disabled on this article.

andy2004 said,
aaaand its all down again here internal server error ;-(

Seconded. And they just posted about spy being back!
Though 500 errors should be caused by that (load and stuff). Hrrm.

Edited by cybertimber2008, Jan 24 2010, 3:23am :

cybertimber2008 said,

Seconded. And they just posted about spy being back!
Though 500 errors should be caused by that (load and stuff). Hrrm.

Still up here (spy too)

Edited by Chester0, Jan 24 2010, 3:52am :

Nice 1 Neowin....

So the search engines needed to re-cache every link of every topic of every..............them bloody spiders, they`ll have yea, darn little critters need squishing.

Great job guys. Glad it all worked out and the issue has been resolved. Can't wait to see the spy features up and running

Amazing news. Finally Neowin has movied to IPB3! Great work on everyones part! Good going and here is to an awesome 2010!

When I saw this new article title this morning on my phone, i couldnt believe it. Opened it and haha loved the article and the topic title is awesome!

But, to the devs, I never knew spiders could do that. While I thought neowin had a default "robots.txt", I couldnt believe a) spiders would hit the server so hard and mess it up because of the new url scheme and b) you guys didnt use a robots.txt by default.

But, great job in figuring it out and great job on rolling it out! I love how the news and forums match and everything loads nice and smooth.

Intelman said,
Should the forums be fully functional now? Spy does not seem to work and the themes do not appear available?

Themes are available at the bottom right corner, via drop down menu.

Spy and Minispy are temporarily disabled until tweaks and upgrades are finished.

Neowin Team Keep Rocking Good to See You Guys hunting down the real problem Cheers !!! that makes this a great article too.

That's a prize-winning title! I'm really glad that everything worked out well! Congrats to the Devs for fixing the issue. It is greatly appreciated. :)

I'm sure this story is interesting to some, but I have to ask a simple question...

How could your tech staff have not planned for this and limited the bots as part of the migration plan? This really makes a lot of us question your staff's tech ability if something so simple was not planned out. (Let alone that you couldn't flip the site/forums on the fly and needed a full day 'downtime').

PS. I like the new site except for the graphical look, it reminds me too much of an outdated XWindow manager. 1999 this would have been cute, today, just kind of 'clueless straight man boring'.

It's only simple after the fact. Like people who spend hrs trying to fix their PC's power problem only for someone to come in and say, "oh did you check if it's plugged in?" Sometimes the simpliest things are the ones that slip our minds.

SHoTTa35 said,
It's only simple after the fact. Like people who spend hrs trying to fix their PC's power problem only for someone to come in and say, "oh did you check if it's plugged in?" Sometimes the simpliest things are the ones that slip our minds.

Maybe for non-tech people or anyone that hasn't moved or migrated a medium/large site. A site even of this size you would imagine the people would know what they are doing and not drawing out plans in the dirt out in the school yard...

SHoTTa35 said,
It's only simple after the fact. Like people who spend hrs trying to fix their PC's power problem only for someone to come in and say, "oh did you check if it's plugged in?" Sometimes the simpliest things are the ones that slip our minds.
I am with you on this SHoTTa35. I have been trouble-shooting many PC problems and yes, sometimes it was the simplest things that seem to always escape my logic because it was ignored.

thenetavenger said,

Maybe for non-tech people or anyone that hasn't moved or migrated a medium/large site. A site even of this size you would imagine the people would know what they are doing and not drawing out plans in the dirt out in the school yard...

We get the point, no need to continue to harp and get pissy about it.

If you have legit issues to discuss or want to give developers feedback, then I suggest you PM them directly.

Thank you.

thenetavenger said,
...just kind of 'clueless straight man boring'.
thenetavenger said,
A site even of this size you would imagine the people would know what they are doing and not drawing out plans in the dirt out in the school yard...
Wow, you're not a very nice person, are you.

There's a way to give criticism, and a way not to. That's how not to.

Edited by Kirkburn, Jan 23 2010, 12:16am :

thenetavenger said,

Maybe for non-tech people or anyone that hasn't moved or migrated a medium/large site. A site even of this size you would imagine the people would know what they are doing and not drawing out plans in the dirt out in the school yard...

Yes because you are perfect. Sorry the poeple who run this site are human beings and make mistakes, it happens plain and simple. Kudos to them on the upgrade though.

Kirkburn said,
Wow, you're not a very nice person, are you.

There's a way to give criticism, and a way not to. That's how not to.

Agreed, my thoughts on this were a bit over the top. I was just surprised that in 2010 people are still pulling web sites offline to upgrade/migrate them when there are a lot of easy ways even on one server to not leave a site down for more than a minute or two.

I was reacting like these people were on one of my teams and I would have not been happy with their performance; however, they aren't and it really isn't any of my business.

I do like the new site, so I didn't mean for my comments of how it looks visually to sound as harsh as it did. I just wish the styles were a bit more advanced, as they do have people that get graphics, just look at the leadin images which are top notch.

The whole 'Motif' looking square button look of the site just reminds me of 1989 too much maybe, which also shows my age.

I apologize for sounding too dicky, I didn't mean to ruffle feathers.

I never would have thought it would be spiders causing the problem, either. Thank you for the update Tom, and it's great to see IPB3 up and working fully. Kudos to the devs and admins.

Disallowing the robots might affect rankings though? I think the best solution is the one they've gone for, changing the crawl rate.

need to create htaccess rewrites to rewrite the new urls in the same manner the old URL's were written, giving you the ability to push the new updates out while at the same time maintaining old search engine placements, rankings, etc....

Soldier101 said,
need to create htaccess rewrites to rewrite the new urls in the same manner the old URL's were written, giving you the ability to push the new updates out while at the same time maintaining old search engine placements, rankings, etc....

That's done by IP.B already They will all be fixed.

And like Neowin could drop in ranking lol.

That has to be one of the most awesome news titles ever! XD

But anyway, a big kudos to the devs are all of their hard work on this.

Excellent work to everyone who helped, it does make sense about the robots reindexing the site again.

It's a new Neowin, and it's better than ever.

wahoospa said,
It took 22 seconds to get into the forums.

Yes there could be a few "slow downs" as the post conversion is running in the background.

AWESOME!!!! Great work everyone. Neowin team and users is what makes this site so great! Cant wait till the next refresh lol ;)

giggsey said,
Awesome, well done guys :)

On a side note, the forums are still loading slowly, is this IPB 3's fault?


They will have a slight load as the post conversion is going on in the background and also Google/Bing etc are indexing all 9 million posts. I'd expect over the weekend it will get better, fingers crossed.

Tom W said,

They will have a slight load as the post conversion is going on in the background and also Google/Bing etc are indexing all 9 million posts. I'd expect over the weekend it will get better, fingers crossed.

Of course, forgot the post conversion would still be going on.

webeagle12 said,
lol at title
I think we can all safely say that the title of this article deserve a golden trophy/award of some kind.

Just want to everyone involved, great work! The downtime was worth it in the end, and I'm glad we managed to get IPB3 up and running.

Also, I vote that this news article receives "epic title of the year".

Kristan K said,
Just want to everyone involved, great work! The downtime was worth it in the end, and I'm glad we managed to get IPB3 up and running.

Also, I vote that this news article receives "epic title of the year".


Yes, I agree... Nice explanation and I felt like it was like a movie or something.

Corris said,
You guys have no idea how long they took to decide on the name.
Tom was so excited when we decided on it, too.

Kristan K said,
"epic title of the year"

This Title of the year award sponsored by KalBox Corp is handed over to Tom and the whole Neowin team. Have fun with the empty beer cans :).

Kristan K said,

Also, I vote that this news article receives "epic title of the year".

+1
I read this oen allowed and everyone in my dorm room almost fell over laughing

Kristan K said,
Also, I vote that this news article receives "epic title of the year".

Absolutely! Its brilliant and hilarious - but thanks for putting all this work into this excellent site!

Kristan K said,
Also, I vote that this news article receives "epic title of the year".

I think you enjoyed getting Binged in the Yahoo! while Jeeves watched! :P

Kristan K said,
Just want to everyone involved, great work! The downtime was worth it in the end, and I'm glad we managed to get IPB3 up and running.

Also, I vote that this news article receives "epic title of the year".


Definitely! I agree with all of that.

Nice to know what the problem was and great work by the Invision support staff.

I hope they actually get some financial compensation for all the effort they spent troubleshooting and fixing a problem that was nothing to do with their software.

Edited by dvb2000, Jan 22 2010, 11:35pm :

jimbo2150 said,

I think you enjoyed getting Binged in the Yahoo! while Jeeves watched! :P
Wahahaha hahaha... you guys are hilarious and yes I vote this article as an epic in the making... Tom Warren, I can give you a golden trophy for the news title!

Kristan K said,
Just want to everyone involved, great work! The downtime was worth it in the end, and I'm glad we managed to get IPB3 up and running.

Also, I vote that this news article receives "epic title of the year".

+1 Neowin Team Keep Rocking Good to See You Guys hunting down the real problem Cheers !!! that makes this a great article too.

Kristan K said,
Just want to everyone involved, great work! The downtime was worth it in the end, and I'm glad we managed to get IPB3 up and running.

Also, I vote that this news article receives "epic title of the year".

+ million... damn this is a good story... can't even imagine the trouble you guys went thru for this..

Kristan K said,
Just want to everyone involved, great work! The downtime was worth it in the end, and I'm glad we managed to get IPB3 up and running.

Also, I vote that this news article receives "epic title of the year".

Tom wins this for concocting such an epic title: http://www.babble.com/CS/blogs/strollerderby/2009/06/chocolate_chip_cookie.jpg

haha it wasn't totally my idea. Steve came up with the title and then Dave changed it and then we tweaked it again. Combination!