News Web sites seek more search control

NEWS WEB SITES SEEK MORE SEARCH CONTROL
[SOURCE: Associated Press, AUTHOR: Anick Jesdanun]
Leading news organizations and other publishers have proposed changing the rules that tell search engines what they can and can't collect when scouring the Web, saying the revisions would give site owners greater control over their content. Top search companies now voluntarily respect a Web site's wishes as stated in a document known as "robots.txt," which a search engine's indexing software, called a crawler, knows to look for on a site. Under the existing 13-year-old technology, a site can block indexing of individual Web pages, specific directories or the entire site. Some search engines have added their own commands to the rules, but they're not universally observed. The Automated Content Access Protocol proposal, unveiled Thursday by a consortium of publishers at the global headquarters of The Associated Press, seeks to have those extra commands - and more - apply across the board. With the ACAP commands, sites could try to limit how long search engines retain copies in their indexes, for instance, or tell the crawler not to follow any of the links that appear within a Web page.
http://www.newsobserver.com/1595/story/799961.html