searchata

Robots.txt generator

Create a robots.txt file with crawler rules and sitemap URLs. Copy or download a draft from the paths you supply.

Use * for all crawlers, or a product token such as Googlebot.

One path per line. Paths start with /.

One path per line. Review rules before publishing.

One complete HTTP(S) URL per line.

Free. No account required. This form processes the values in your browser.

WebMCP available in compatible browsers

How this tool works

Enter a crawler product token, allowed paths, disallowed paths and optional sitemap URLs. The generator validates the input and formats a robots.txt draft in your browser.

  • This generator does not read your existing file, fetch your website or publish changes.
  • Robots.txt is not access control. A blocked URL can still appear in search results.

Read the result, then take action.

Illustrative example — not a live result

Keep a private section out of crawler requests

Add /private/ as a Disallow path for the crawler you select.

Next step: Review the draft against your existing rules before publishing it at /robots.txt.

Common questions

Where do I put robots.txt?

Publish the file at the root of the relevant host, such as https://example.com/robots.txt. Subdomains can have separate files.

Will Disallow remove a page from Google?

Not necessarily. Robots.txt controls crawling. Search engines can discover a URL through links. Use the appropriate indexing controls when you need to exclude a page.

Can I keep existing crawler rules?

Yes. Review the full existing file and merge rules carefully. This tool creates one crawler group; it does not merge or test an existing file.

Use this tool with an agent

Open this page in a WebMCP-compatible browser. Ask your agent to use your supplied values. The generated result appears here and is returned as JSON. Your agent can also use the public POST endpoint.

generate_robots_txt({
  "userAgent": "*",
  "allow": [
    "/"
  ],
  "disallow": [
    "/private/",
    "/admin/"
  ],
  "sitemaps": [
    "https://example.com/sitemap.xml"
  ]
})

POST /api/free-tools/robots-txt-generator
Content-Type: application/json

{
  "userAgent": "*",
  "allow": [
    "/"
  ],
  "disallow": [
    "/private/",
    "/admin/"
  ],
  "sitemaps": [
    "https://example.com/sitemap.xml"
  ]
}

Results include the generated output, supplied-data results, and usage limits. View the tool catalog.