Can search engines track down terrorists?
Posted by vincent on 20 April 2002 - 11:02 · 1 comment & 215 views
About the Author
Full name: Unknown
Forum name: vincent
Contact: via forum PM
Submissions: Normal · Headline
Full name: Unknown
Forum name: vincent
Contact: via forum PM
Submissions: Normal · Headline
Slashdot It!
Submit to reddit
Submit to blinklist
Bookmark on del.icio.us
Add to furl
Share on Facebook
Add to Windows Live
Add to Google
Some US government officials engaged in the so-called war on terror would like to see privacy laws relaxed so they can get better access to e-mail and other sensitive material exchanged over the Internet.
Other bureaucrats would be happy just to have a better filing system.
Tracking down terrorists, after all, does not necessarily require intercepting top-secret conversations. Sometimes it is a more mundane task of making sure one government agency can share its records with another.
Speculation has been rife, for instance, that September 11 hijacker Mohammed Atta might not have been allowed back into the United States last year if officials at the Department of Immigration had known about an outstanding warrant for his arrest resulting from a Florida traffic violation.
There is also the curious case of Ziad Samir Jarrah, the hijacker believed to have seized the controls in the United Airlines flight that crashed in Pennsylvania. Jarrah remained on the Federal Aviation mailing list long after September 11, and just this month the FAA sent a newsletter to his old address in Florida.
Communication between different agencies is not the only problem. Some government agencies cannot even access critical documents produced by their own employees, if those records have never been transferred out of e-mail.
Yet another problem is that foreign-language documents can fall through the cracks because many computers are not advanced enough to read them.
New challenge for search engines
Several Internet search engines and software makers are trying to address those challenges. Companies like CMGI's AltaVista and Inktomi, which started out organising the billions of documents on the Web, today are selling similar technologies to help government agencies organise all the material they have collected over the years.
These companies say many government offices that are bogged down in paper and old computer records constitute a promising market for their services, especially at a time that so many private-sector businesses are cutting back on tech spending.
"I would say about 25 percent of our business is selling to the government," said Phil Rugani, executive vice president of AltaVista's software division, which counts the US Army, Navy and Air Force among its customers.
"The need to find information within the vast repositories of the government world has, shall we say, been heightened since September 11," he said. "It is a very good business for us."
AltaVista stepped up efforts to sell software to both the private sector and the government after its consumer search engine was hit by the collapse in the online advertising business.
Today, while Google holds a commanding lead in the consumer Internet search business, several second- and third-tier search players are looking elsewhere for business.
"A lot of government agencies are realising that a lot of the valuable data they store is unstructured," explained Troy Toman, general manager of enterprise search at Inktomi.
"Since the generation of desktop publishing, it has become much easier for people to create and publish documents. The problem is that the set of tools to find all this material has been a little slow to keep pace."
Walk before you run
If searching the finite number of documents that exist within a single government organisation seems like a minor task relative to searching everything that exists on the Internet, it actually presents a number of unique challenges.
Unlike the Web itself, where all the text exists in the same HTML format, most individual organizations have no such consistency. Files typically exist in hundreds of different formats from Microsoft Word to Lotus Notes, as well as old computer languages that are not used anymore.
The challenge becomes not just locating files, but knowing where to look for them, and that in turn presents a whole new set of privacy concerns over who should be able to see what.
"The reality is that after 30 or 40 years of technology, information sits everywhere, absolutely everywhere," said Rugani of AltaVista Software. He said his company's software, which searches in 36 languages and 225 file formats, is like a door to a refrigerator--opening it provides a window to all the things that are stored on different shelves.
Verity, another Silicon Valley software maker, says that its work selling software to the government has forced it to add a new level of security so that unauthorised personnel cannot even get as far as seeing a list of search results when querying subjects outside of their jurisdiction.
Verity recently helped the US Air Force integrate information from hundreds of different Air Force Web sites, the first step in enabling it to communicate with other branches of the military and other government groups.
"They have all this information scattered all around the world, and it is not all organised," explained a Verity representative. "It is an example of having to be able to walk before you can run."