# # $Id: robots.txt,v 1.3 1997/06/04 03:52:13 vic Exp $ # # (why am I awake right now writing letters to robots?) # # Well, aren't you clever? You found the REP file. # # A NOTE TO THOSE WHO'D BOTHER TO READ THIS FILE: # # The purpose of the "/robots.txt" file is to keep listed directories # from being indexed by Internet search companies' robots|crawlers|agents # so that the average network user doesn't stumble across them while # performing searches, and those that require some form of authentication. # Many search engine companies parse the login errors generated by such # authentication-protected pages incorrectly and then inappropriately # index and display those errors to end users which makes search results # look ugly or deceptively interesting (without disclosing proprietary # information). # User-agent: * Disallow: /dl Disallow: /gfx Disallow: /include