Robots.txt for WordPress — Best Practices & Yoast / Rank Math Setup

A safe, modern robots.txt for WordPress. What to block (wp-admin, search, login), what NOT to block (uploads, CSS, JS), and how to edit the virtual robots.txt in Yoast or Rank Math.

WordPress serves a virtual robots.txt from /robots.txt automatically — but the default is bare. Use one of the templates below and ship it via Yoast, Rank Math, an SEO plugin, or a real file at the site root.

OPEN GENERATOR →
ADVERTISEMENT

Recommended Baseline robots.txt for WordPress

# Recommended baseline for WordPress
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-login.php
Disallow: /xmlrpc.php
Disallow: /?s=
Disallow: /search/

Sitemap: https://yourdomain.com/sitemap_index.xml

This blocks the admin area, the WordPress login and XML-RPC endpoints, and on-site search results pages — those generate near-duplicate content that wastes crawl budget and can dilute rankings. The Allow: /wp-admin/admin-ajax.php line is critical: many themes and plugins call AJAX from the front end, and blocking it would break them in Google's renderer.

WordPress + WooCommerce

# WordPress + WooCommerce
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-login.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /?add-to-cart=
Disallow: /?orderby=
Disallow: /?filter_*

Sitemap: https://yourdomain.com/sitemap_index.xml

WooCommerce adds cart, checkout, account and dynamic filter URLs. None of these belong in the index. The ?add-to-cart= and ?orderby= patterns are common sources of duplicate URLs.

Don't Block These

  • /wp-content/uploads/ — Google needs to crawl images for Image Search and rich snippets.
  • /wp-includes/ JS / CSS — needed for page rendering during indexing.
  • /wp-content/themes/*.css and .js — same reason.
  • /feed/ — feeds link to your posts and help discovery; only block if you have noindex problems.

How to Edit Robots.txt in WordPress

With Yoast SEO

  1. Go to SEO → Tools → File editor.
  2. If robots.txt doesn't exist yet, click Create robots.txt file.
  3. Paste your rules and save. Yoast writes a real file to the site root.

With Rank Math

  1. Go to Rank Math → General Settings → Edit robots.txt.
  2. Paste your rules and save. Rank Math edits the virtual robots.txt unless a physical file exists.

Without a plugin (FTP or hosting file manager)

  1. Create a plain text file named robots.txt with your rules.
  2. Upload it to the site root (the same folder as wp-config.php and index.php).
  3. A real file overrides any virtual robots.txt that WordPress or a plugin would generate.

What WordPress Generates by Default

Without a real file or plugin override, WordPress emits a minimal virtual robots.txt that disallows /wp-admin/ and allows /wp-admin/admin-ajax.php. The defaults do not include a Sitemap: line — adding one is one of the highest-leverage SEO changes you can make.

Block AI Crawlers

# Add to the bottom of your robots.txt to opt out of AI training
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

This opts your site out of training for ChatGPT, Claude, Gemini, and the Common Crawl dataset. It does not affect Google Search rankings, since Googlebot is a separate user-agent from Google-Extended.

Verify After Publishing

  1. Visit https://yourdomain.com/robots.txt in a private window — you should see your rules.
  2. Open Google Search Console → Settings → robots.txt and confirm Google reads the latest version.
  3. If a CDN sits in front (Cloudflare, etc.), purge the /robots.txt URL after every change.

WordPress Robots.txt FAQ

Do I need a robots.txt if WordPress already generates one?

Technically no — WordPress will emit a minimal virtual file. But the default has no Sitemap: line and no rules beyond blocking /wp-admin/. Adding a customized robots.txt with your sitemap and the patterns above gives crawlers a much better starting point.

Should I block /wp-content/uploads/?

No. Google needs your images for Image Search, rich results, and rendering. Blocking the uploads folder is one of the most common SEO mistakes on WordPress.

Should I block /wp-includes/?

No. /wp-includes/ contains JavaScript and CSS that Google's renderer fetches when indexing your pages. Blocking it can break rendering. Older guides suggested disallowing it; current Google guidance is to leave it open.

How do I block on-site search results from being indexed?

Add Disallow: /?s= and Disallow: /search/ to robots.txt. As an extra layer, configure your SEO plugin to set noindex on search result templates so any URLs already in the index get removed.

Where does Yoast write the robots.txt file?

When you save robots.txt in SEO → Tools → File editor, Yoast writes a real text file to the WordPress installation root (the same directory as wp-config.php). The real file overrides WordPress's virtual robots.txt.

My robots.txt changes aren't showing up. Why?

Three usual suspects: (1) your CDN is caching /robots.txt — purge that URL specifically; (2) a security plugin or hosting layer is serving a hardcoded version; (3) two plugins are both generating robots.txt and one is winning. Hit https://yourdomain.com/robots.txt in a private browser window with cache disabled to see what crawlers actually receive.

RELATED TOOLS

SITEMAP XML OG TAGS SCHEMA MARKUP