Robots.txt for Astro — Static Site & Dynamic Endpoint Examples

Option 1: Static File in public/

Drop the file into public/robots.txt and Astro copies it as-is to the build output. This is the recommended approach unless you need conditional logic.

# public/robots.txt
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap-index.xml

If you use the official @astrojs/sitemap integration, the sitemap is emitted at /sitemap-index.xml (with one or more sitemap-N.xml files referenced from it).

Option 2: Dynamic Endpoint at src/pages/robots.txt.ts

Astro pages can be plain .ts files that export a GET handler. The file becomes a route, and the response body is whatever you return. This lets you read env vars or compute rules at build/render time.

// src/pages/robots.txt.ts
import type { APIRoute } from 'astro';

const SITE = 'https://yourdomain.com';

const body = `User-agent: *
Allow: /
Disallow: /api/

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

Sitemap: ${SITE}/sitemap-index.xml
`;

export const GET: APIRoute = () => new Response(body, {
  headers: { 'Content-Type': 'text/plain; charset=utf-8' },
});

The endpoint runs at build time for static output and per request for SSR — both produce the correct /robots.txt URL.

Option 3: Per-Environment Robots.txt

Block crawlers entirely on staging or preview deploys to keep them out of the index:

// src/pages/robots.txt.ts
import type { APIRoute } from 'astro';

export const GET: APIRoute = () => {
  const isProd = import.meta.env.PROD && import.meta.env.PUBLIC_SITE_ENV === 'production';
  const body = isProd
    ? `User-agent: *\nAllow: /\n\nSitemap: https://yourdomain.com/sitemap-index.xml\n`
    : `User-agent: *\nDisallow: /\n`;
  return new Response(body, {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
};

Set PUBLIC_SITE_ENV=production in your production deploy environment so non-production builds always disallow everything.

Don't Block /_astro/

The /_astro/ directory contains the JavaScript and CSS chunks that hydrate your islands. Google fetches these to render your page during indexing. Blocking the directory in robots.txt will degrade rendering — leave it open.

i18n Routing

If you use Astro's i18n config (locale subpaths like /ja/, /ar/), you don't need separate robots rules per locale. Use hreflang tags in your layout (Astro generates them via the link rel="alternate" pattern) and let crawlers see all locales freely.

Block AI Crawlers

Whether you use the static file or the endpoint approach, the AI crawler block is the same standard robots.txt syntax — five user-agent groups (GPTBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot), each with Disallow: /.

Verify After Build

1. Run astro build && astro preview and open http://localhost:4321/robots.txt.
2. Confirm the file is served with Content-Type: text/plain.
3. After deploy, fetch the production URL and verify the same content.
4. Submit your sitemap in Google Search Console if you haven't already.