When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works.

Google Turns Away Robots From Its Front Door

Citing the need to protect its server resources, Google has prevented competing search engines from indexing much of its own Web site, the company confirmed Wednesday.

While Google has amassed an index of over 2 billion Web pages by automatically "spidering" or "crawling" sites all over the Web, the popular search portal has effectively walled off numerous sections of its own site from other search "bots."

By placing a special "robots.txt" file on its server, Google has prohibited other crawlers from indexing 19 areas of its site, including one that offers searches of the company's archive of Usenet newsgroup discussions, as well as an area for exploring its index of graphic images on the Web. Also blocked are a special index of mail-order catalogs, and a section that allows searches of news articles at other sites.

As a result, a search of the phrase "LL Bean" performed through a link to Google set up at AltaVista.com produces no results. Doing the same search going directly though Google.com produces 20 pages that include the LL Bean name.

News source: TechNews.com

View: The Full Story

View: The Robot Exclusion Standard

View: Meyer's posting on RISKS

Report a problem with article
Next Article

miTweak-XP v2.3.3

Previous Article

MS scarier than the CIA, Gateway exec says