Robots.txt A file placed in the root directory of a website to instruct search engine bots on which pages to crawl Read More »