Search engines 'struggling' to index the Internet

A WHITE PAPER on Internet search engines claims there's a revenue limit to the market to deliver answers via the Web unless "pay for placement" is introduced into the equation. The article, Search Engines: Evolution and Diffusion, says that the market is far from limitless and that firms relying on seaches are unlikely to generate revenues above $200 million unless they introduce pay schemes, such as Overture's.

That's important because the cost of indexing is sky-rocketing, the authors claim. The paper also warns that natural language processing is still a way away, especially where huge content is involved.

The article, written by Harry Collier and Stephen E. Arnold, reckons that content is growing much quicker than any machine or human hybrid can cope with. It's hard to make software behave like humans and rule based systems fall over when there's a mixture of media, including PDFs, Powerpoint files, text, radio and video. One single index entry can cost between $3 and $50, the authors claim.