# robots.txt for http://www.unboutdeparadis.com/ # # This file is used to allow crawlers to index our site. # Any empty [Disallow] value, indicates that all URLs can be retrieved. # At least one Disallow field needs to be present in a record. # # List of all web robots: http://www.robotstxt.org/wc/active/html/index.html # # Check robots.txt at: # http://www.searchengineworld.com/cgi-bin/robotcheck.cgi # # Details about Googlebot available at: http://www.google.com/bot.html # The Google search engine can see everything User-agent: Googlebot Disallow: /cgi-bin/ User-agent: Mediapartners-Google Disallow: /cgi-bin/ User-agent: Slurp Crawl-delay: 100 Disallow: /cgi-bin/ User-agent: Yahoo-NewsCrawler Disallow: /cgi-bin/ User-Agent: msnbot Crawl-delay: 100 Disallow: /cgi-bin/ User-agent: * Disallow: /cgi-bin/