Robots.txt for Next.js — App Router & Pages Router Examples

Option 1: Static public/robots.txt (simplest)

Drop a file at public/robots.txt and Next.js serves it at /robots.txt. This works for both the App Router and the Pages Router. Use this when your rules don't depend on environment.

# public/robots.txt
User-agent: *
Allow: /
Disallow: /api/

Sitemap: https://yourdomain.com/sitemap.xml

Option 2: Dynamic app/robots.ts (App Router)

Since Next.js 13.3, the App Router supports a robots.ts (or .js) file that exports a default function returning a Robots object. Next.js generates the final text file at build time.

// app/robots.ts
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: '/api/' },
      { userAgent: 'GPTBot', disallow: '/' },
      { userAgent: 'ClaudeBot', disallow: '/' },
      { userAgent: 'Google-Extended', disallow: '/' },
    ],
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

Reference: Next.js Metadata API — robots.

Option 3: Pages Router

The Pages Router has no robots.ts convention. Use the static file approach (public/robots.txt) or write an API route at pages/api/robots.ts and rewrite /robots.txt to it via next.config.js:

// next.config.js
module.exports = {
  async rewrites() {
    return [
      { source: '/robots.txt', destination: '/api/robots' },
    ]
  },
}

Don't Block /_next/

Common mistake: blocking /_next/ in robots.txt. This stops Googlebot from fetching your JavaScript and CSS, and Google needs both to render the page for indexing. Leave /_next/ open. Only consider blocking /api/ if those endpoints have no SEO value.

Conditional Rules per Environment

Block everything on previews so deploy URLs don't get indexed, then loosen on production:

// app/robots.ts
import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  const isProd = process.env.VERCEL_ENV === 'production'

  if (!isProd) {
    return { rules: [{ userAgent: '*', disallow: '/' }] }
  }

  return {
    rules: [{ userAgent: '*', allow: '/', disallow: '/api/' }],
    sitemap: 'https://yourdomain.com/sitemap.xml',
  }
}

Internationalized Routes (i18n)

Next.js i18n routes (/en/..., /ja/...) don't need special robots rules — leave them all crawlable and use hreflang tags to tell Google how the locales relate.

Block AI Crawlers

List each AI user-agent on its own block:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Note: Google-Extended only opts you out of Gemini and Google's AI training. It does not affect Google Search rankings — those use the regular Googlebot user-agent.

Test Your Robots.txt

1. Build and start your app: next build && next start
2. Open http://localhost:3000/robots.txt and confirm the file is served.
3. After deploy, verify https://yourdomain.com/robots.txt returns 200 with Content-Type: text/plain.
4. Test individual rules with the Google Search Console robots.txt tester.