Key Takeaways
- A robots.txt generator builds crawler access rules for your site without you writing the syntax by hand or touching server files.
- WordPress gives you three native ways to generate one: the Rank Math editor, the Yoast file editor, and the RankReady AI crawler rules builder. No FTP required for any of them.
- Most online robots.txt generators still produce 2019-era files. They skip the AI crawler user-agents (GPTBot, OAI-SearchBot, PerplexityBot) that decide whether ChatGPT and Perplexity can cite you.
- Always include a Sitemap directive in the output. Without it, crawlers have to guess where your index lives.
- Verify every change with Google Search Console’s robots.txt tester and a quick curl check before you trust it.
The first thing I did on a new client site last month was open yoursite.com/robots.txt in a browser tab. Three lines. It blocked /wp-admin/, allowed everything else, and had not been touched since 2019. So I did what most people do: I searched “robots.txt generator,” clicked the top result, and pasted in my domain. The tool spat out a clean little file that was just as useless as the one already live. No sitemap reference. No mention of a single AI crawler. It was a generator built for a web that stopped existing around the time ChatGPT launched.
Here is the part nobody tells you: you almost certainly do not need an external robots.txt generator at all. WordPress already ships the tools to build one, and they understand your site structure in a way a generic online box never will. This guide walks through every way to generate a robots.txt file for a WordPress site, which method fits which situation, and the exact rules your 2026 file actually needs so search engines and AI engines both read you correctly.
What a WordPress robots.txt file does (and what it does not)
A robots.txt file is a plain text file at the root of your domain that tells crawlers which parts of your site they may request. It is the first file a well-behaved bot reads before it touches anything else. Google, Bing, and the major AI crawlers all check it on arrival.
What it is not: a security tool. A robots.txt file is a polite request, not a lock. It cannot stop a bad actor from reading a page, and any URL you list in it is publicly visible to anyone who opens the file. If you want to keep something private, use authentication or a noindex directive, not a Disallow line. That single misunderstanding is behind half the broken robots.txt files I review.
Why this matters more now than it did three years ago: according to Cloudflare’s public bot analytics, AI crawlers reach 39% of the top one million websites every month, yet only 3% of those sites make a deliberate decision about that traffic. Your robots.txt file is where that decision gets made, so the tool you use to generate it now has to speak the AI crawler dialect, not just the Googlebot one.
What a robots.txt generator actually produces
Every robots.txt generator, online or built into WordPress, does the same three things. It takes your inputs (which bots, which paths), turns them into the correct User-agent and Allow or Disallow syntax, and outputs a file you can save to your site root. The difference between a good generator and a bad one is entirely in what inputs it knows to ask about.
A generic online generator asks about Googlebot, maybe Bingbot, and a crawl-delay setting almost nobody needs. It does not ask whether you want OpenAI’s search crawler to cite you, or whether you are opting out of AI training. It does not know that a WordPress site has a /wp-json/ REST endpoint or a /feed/ path worth handling. So the file looks valid, passes a syntax check, and quietly leaves out everything that matters in 2026.
Also Read: for the strategy behind which AI bots to allow or block and why, read WordPress robots.txt for AI Crawlers: A Gutenberg Builder Guide before you generate anything. This guide covers the how, that one covers the which and the why.
5 ways to generate a robots.txt for your WordPress site
Ranked roughly from “already installed on your site” to “only if you have no other option.” Pick the first one that applies to you.
Method 1: The Rank Math robots.txt editor (free, most likely already installed)
If you run Rank Math, you have a robots.txt generator and editor built in. Go to WordPress admin, Rank Math, General Settings, Edit robots.txt. You get a text editor that writes a virtual robots.txt file served at your site root. Paste your rules, save, and Rank Math handles the file for you. No FTP, no file permissions, no server access. This is the method I reach for first on nine out of ten sites because the plugin is already there.
One catch: if a physical robots.txt file exists in your site root, the in-plugin editor turns read-only, because a real file always wins over a virtual one. Delete the physical file over FTP first, then the editor unlocks.
Method 2: The Yoast SEO file editor
Yoast offers the same capability under Yoast SEO, Tools, File editor. If no robots.txt exists yet, Yoast shows a “Create robots.txt file” button. Click it, and you get an editable file with WordPress defaults already filled in. Edit, save, done. The same physical-file caveat applies: a real file in the root overrides the editor.
Functionally Rank Math and Yoast are interchangeable here. Use whichever SEO plugin you already run. There is no reason to install one purely for its robots.txt editor.
Method 3: Online robots.txt generator tools
Tools like SEOptimer and the TechnicalSEO.com robots.txt generator are fine for one job: producing a starting template you then paste into your Rank Math or Yoast editor. They are quick and they get the basic syntax right.
Their limit is what I described earlier. They do not know about WordPress-specific paths, and most of them have no AI crawler options at all. Treat their output as a skeleton, never as a finished file. You will be hand-adding the AI crawler block and the WordPress paths regardless, so the time saved is smaller than it looks.
Method 4: Create the file manually over FTP or cPanel
If you have no SEO plugin and cannot install one, create a plain text file named exactly robots.txt, paste your rules, and upload it to your site root (the same folder as wp-config.php). Use the File Manager in cPanel or any FTP client. This is the most manual route and the easiest to get wrong, since a single typo in a filename or path means crawlers ignore the file entirely. Reserve it for when a plugin genuinely is not an option.
Method 5: The RankReady AI crawler rules builder
This is the only WordPress-native generator I know of that treats AI crawlers as first-class inputs rather than an afterthought. RankReady is a free AI SEO plugin we built at POSIMYTH Innovations, and its crawler rules module generates the AI bot section of your robots.txt for you: it splits training crawlers from search crawlers, defaults the search bots to allowed so you stay citable, and writes the Sitemap directive automatically. It also flags when a host or CDN is blocking AI bots before your robots.txt is ever read, which a text editor cannot tell you.
If your goal is specifically to control how ChatGPT, Claude, and Perplexity see your site, this is the method that was designed for the job. Install it free from store.posimyth.com/plugins/rankready/ and the rules builder lives in the WordPress admin sidebar.
What your WordPress robots.txt should include in 2026
Whichever method you pick, the output should contain these sections. This is the template we run across our own WordPress properties. Generate the structure, then adjust the training-crawler lines to your preference.
# Standard WordPress paths
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /wp-json/
Disallow: */feed/
Allow: /wp-admin/admin-ajax.php
# AI search crawlers, always allow these
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# AI training crawlers, your call to allow or block
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
# Blocked: legacy scrapers
User-agent: CCBot
Disallow: /
User-agent: Bytespider
Disallow: /
Sitemap: https://yoursite.com/sitemap.xml
Three things every generated file needs and most online tools skip. First, the standard WordPress paths so crawlers do not waste budget on admin and JSON endpoints. Second, the AI crawler split: search bots like OAI-SearchBot and PerplexityBot should almost always stay allowed, because blocking them makes you invisible inside ChatGPT Search and Perplexity, while training bots like GPTBot and ClaudeBot are your judgment call. Third, the Sitemap line, which points every crawler at your index.
Also Read: not sure which user-agent belongs to which company? Our Web Crawlers List: 15 Most Common Bots and Spiders maps every search, AI, and SEO-tool bot so you know exactly what each line in your file controls.
How to test that your generated robots.txt actually works
Generating the file is half the job. A file that looks right but blocks the wrong bot is worse than no file at all. Run these three checks after every change.
Google Search Console. Open the robots.txt report in Search Console, which shows you the live file Google fetched and flags any syntax errors. This is the authoritative view of what Googlebot sees, not what you think you saved.
The curl test. From a terminal, pretend to be an AI crawler and check the response code:
curl -A "OAI-SearchBot/1.0" -I https://yoursite.com/An HTTP 200 means that bot can reach you. A 403 or 429 means something upstream, usually your host or CDN, is blocking the crawler before your robots.txt is ever read. No generator can fix that, because the request never reaches WordPress. Swap the user-agent string to test GPTBot, PerplexityBot, or any bot you care about.
Automated, inside WordPress. The RankReady AI Readiness Diagnostics module runs that same curl test on a schedule from inside your admin and flags any 403 or 429 with a link to the host dashboard where you fix it. It is the difference between checking once and knowing the moment a host update silently starts blocking ChatGPT.
Which method should you actually use?
If you already run Rank Math or Yoast, use their built-in editor and paste in the 2026 template above. That covers the vast majority of WordPress sites and costs nothing. If your priority is controlling AI crawler access specifically, or you want the file kept current automatically as the bot landscape shifts, the RankReady rules builder was made for exactly that. Reach for an online generator only as a quick template source, and for manual FTP only when no plugin is possible.
The honest takeaway: the best robots.txt generator for WordPress is usually the one already sitting in your plugin list. The skill that matters is not finding a tool, it is knowing the rules to feed it, and now you have them.
Suggested reading
- WordPress robots.txt for AI Crawlers: A Gutenberg Builder Guide
- Web Crawlers List: 15 Most Common Bots and Spiders
- Cloudflare Turnstile vs Google reCAPTCHA
- RankReady: the free WordPress AI SEO plugin
