# I'm robots.txt # http://www.robotstxt.org/wc/robots.html : more information # http://www.sxw.org.uk/computing/robots/check.html : syntax checking User-agent: * Crawl-delay: 10 Disallow: /core/ Disallow: /~readme.txt