Question 1

What is robots.txt?

Accepted Answer

robots.txt is a plain-text file placed at the root of your website (e.g., yoursite.com/robots.txt) that tells search engine crawlers and AI bots which pages they are allowed or disallowed from accessing. It follows the Robots Exclusion Protocol, a de facto standard since 1994.

Question 2

Why do I need a robots.txt file?

Accepted Answer

Without a robots.txt file, crawlers assume they can access every page on your site. A robots.txt file lets you block crawlers from private areas, reduce server load from excessive crawling, prevent duplicate content from being indexed, and control which AI bots can train on your content.

Question 3

How is robots.txt different from a sitemap?

Accepted Answer

robots.txt tells crawlers what NOT to access (or explicitly allows access), while a sitemap.xml tells crawlers what pages exist and should be indexed. They are complementary — robots.txt controls access and sitemap.xml aids discovery. You typically reference your sitemap URL inside robots.txt.

Question 4

What are AI bot user-agents I should know about?

Accepted Answer

Common AI bot user-agents include GPTBot and ChatGPT-User (OpenAI), ClaudeBot and Claude-Web (Anthropic), Google-Extended (Google AI training), PerplexityBot, CCBot (Common Crawl), Amazonbot, Bytespider (ByteDance), and Meta-ExternalAgent. You can allow or block these individually in your robots.txt.

Question 5

Does blocking a crawler in robots.txt guarantee it won't access my content?

Accepted Answer

No. robots.txt is a voluntary protocol — well-behaved crawlers respect it, but malicious bots may ignore it. For truly sensitive content, use server-side authentication or access controls rather than relying solely on robots.txt.

Question 6

What does the Crawl-delay directive do?

Accepted Answer

Crawl-delay tells a crawler to wait a specified number of seconds between requests. This helps protect servers with limited resources from being overwhelmed. Note that Googlebot does not honor Crawl-delay — use Google Search Console to control Google's crawl rate instead.

Question 7

Where do I place the robots.txt file on my website?

Accepted Answer

The file must be accessible at the root of your domain: https://yoursite.com/robots.txt. For static sites (Netlify, Vercel, GitHub Pages), place it in your public/ or root output folder. For server-based setups, place it in the web root alongside index.html.

Question 8

Can robots.txt affect my SEO rankings?

Accepted Answer

Indirectly, yes. Blocking important pages via robots.txt prevents search engines from crawling and indexing them, which means they won't appear in search results. Be careful not to accidentally block pages you want ranked. Use robots.txt for genuinely private or low-value pages only.

Free Robots.txt Generator

User-Agent Blocks

Sitemap Directives

AI Bot Control

Instant Preview

100% Free & Private

Frequently asked questions