User-agent: Applebot User-agent: Amazonbot Crawl-delay: 60 Disallow: / Allow: /psse-help-forum/question/ Allow: /psse-help-forum/questions/$ User-agent: * Crawl-delay: 30 Disallow: /psse-help-forum/account/ Disallow: /psse-help-forum/users/ # The feedback form. An anonymous GET writes a session row and a # Set-Cookie before it even redirects (FEEDBACK_MODE is auth-only), # so a crawl of it is one row per hit. psspy_facet_gate 403s it for # anonymous visitors regardless -- this is the polite half of the # same message, for the bots that do read robots.txt. Disallow: /psse-help-forum/feedback Disallow: /psse-help-forum/questions/scope: Disallow: /psse-help-forum/questions/*/edit/ Disallow: /psse-help-forum/questions/*/close/ Disallow: /psse-help-forum/questions/*/retag/ Disallow: /psse-help-forum/questions/*/delete/ # Revision history. The view htmldiff()s every consecutive pair of # revisions on every request (~3.2s median) and one crawler walking # it produced 2678 504s in an hour. psspy_facet_gate 403s these for # anonymous visitors regardless -- this is the polite half of the # same message, for the bots that do read robots.txt. Disallow: /psse-help-forum/questions/*/revisions/ Disallow: /psse-help-forum/answers/ Disallow: /*?sort= Disallow: /*?query=