A file placed in the root directory of a website to instruct search engine bots on which pages to crawl or exclude from crawling.
Meta tags
HTML tags that provide information about a webpage to search engines, such as its title, description, and keywords.
