User-agent: * Allow: / # HTML pages carry noindex. Crawling is permitted so that instruction can be read.