Web Tools · Free · No signup
One bad line in robots.txt can hide your whole site from Google — this generator produces a clean, valid file from simple choices: a preset (allow everything, block everything, WordPress, Shopify), your own disallow paths, an optional crawl-delay and the sitemap URL. The output follows the Robots Exclusion Protocol as crawlers actually parse it, with the sitemap declaration that many sites forget. Generated entirely in your browser — no account, no submission, no server round-trip.
🔒 100% private: everything runs inside your browser — nothing is uploaded, stored or logged.
Open, closed, and platform-aware WordPress/Shopify templates that protect admin paths without blocking assets.
Type the paths, get the syntax right: one per line, trailing wildcards supported, comments included.
Google ignores crawl-delay; Bing and others honor it — the generator writes it the way parsers expect.
The Sitemap: declaration helps crawlers find your sitemap without Search Console.
The file is built in your browser. Your staging paths and private folders are nobody's business.
Input & output. Input: preset choice + paths. Output: a plain-text robots.txt file, downloadable and copy-paste ready, to be placed in the site root (not in a subfolder).
robots.txt is a request, not a wall: well-behaved crawlers obey it, malicious bots ignore it, and it does not hide pages from search results that are linked elsewhere (use noindex meta tags for that). Google ignores crawl-delay. Disallow does not remove pages already indexed — that takes removal tools or noindex. Always verify the file after upload at https://yourdomain/robots.txt.
No. It prevents crawling, not indexing — a blocked page can still appear in results if other pages link to it. To keep a page out of results, use a noindex meta tag and leave it crawlable.
Most live sites want Allow all plus a few disallowed paths (admin, cart, internal search). Block all is for staging sites and maintenance windows — remember to switch back.
In the site root, so it answers at https://yourdomain.com/robots.txt — not in a subfolder. One file per hostname (and per protocol/port).
No — Googlebot ignores it. Bing, Yandex and many smaller crawlers do honor it. If you need to slow Google specifically, use the crawl-rate setting in Search Console.
Asterisk matches any sequence and $ anchors the end: Disallow: /*.pdf$ blocks every PDF. The generator includes your paths as-is, so type the pattern you need.
Keep going
Generate SEO and Open Graph meta tags with a search preview.
Use Tool →
Build tracked campaign URLs with clean, lowercase UTM parameters.
Use Tool →
Check whether your images are properly lazy-loaded.
Use Tool →
Generate complete favicon packs (ICO + PNG) for all platforms.
Use Tool →
Turn titles into clean, SEO-friendly URL slugs — batch mode included.
Use Tool →
On mobile?
This tool runs in any mobile browser — no app needed. And for the image & PDF side of the toolbox, the CompressoPanda Android app ships 36 tools that run with no connection at all: compression to target size, editing and PDF essentials included.