# https://www.robotstxt.org/robotstxt.html User-agent: * Disallow: # Translated website pages (//) stay crawlable on purpose: they # answer with `X-Robots-Tag: noindex, nofollow` and a matching meta robots tag, # and a crawler has to fetch a page to see either one. Disallowing them here # would hide that directive and leave copies already in search results ranking # against the sites they were translated from.