# www.robotstxt.org/ # Allow crawling of all content User-agent: * Disallow: /query/ Disallow: /annotation/ Disallow: /autocomplete/ Disallow: /download/