# AI Anyone robots.txt # # All crawlers (search engines + AI bots) are welcome on every public surface. # The Disallow lines block administrative and internal API paths so they don't # burn crawl budget or surface as "Crawled - currently not indexed" in GSC. # # IMPORTANT: robots.txt rules apply per User-agent GROUP. When a bot has its # own User-agent block, it ignores rules from OTHER blocks (including the # wildcard `*`). Earlier revisions of this file put `Disallow: /api/` after # per-bot `Allow: /` declarations, which scoped the disallows to only the # last group (Meta-ExternalFetcher) and let Googlebot crawl ~50 API image # endpoints per cycle. Fixed 2026-05-27 by consolidating into a single # wildcard group, Allow: / is the default anyway, so the per-bot blocks # were redundant. AI/search bots are NOT singled out as separate groups # anywhere on this file; if any specific bot needs to be banned later, # add a dedicated User-agent block that mirrors these Disallows. # # CRITICAL (2026-07-14): `Allow: /api/personalities/` MUST stay above the # /api/ Disallow. Google renders pages with JS; the /chat/ client # fetches /api/personalities/ during render, and when robots blocks it # the rendered page becomes the "Personality not found" error card, which # Google classifies as a Soft 404 and DE-INDEXES the character page. The # 2026-05-27 blanket /api/ Disallow caused a mass Soft-404 de-indexing of # /chat/* (confirmed via GSC live tests, fixed here). Longest-match wins, # so everything else under /api/ stays blocked. # # CRITICAL (2026-08-02): `Allow: /api/blog/posts/` is the SAME bug as the # /chat/* one above, on the blog. /blog/ fetches /api/blog/posts/ # during render (and ...//related), so while robots blocked it, Googlebot # rendered the "Article Not Found" card instead of the article and filed every # post as a Soft 404. Confirmed the same day: GSC live tests on # /blog/shakespeare-leadership-lessons and /blog/leonardo-da-vinci-creativity # both returned "Page cannot be indexed: Soft 404" with a rendered screenshot # showing the not-found card, and Request Indexing was REJECTED, even though # the server sends the full article HTML to bots and the API returns 200 to # everyone else. The /best/* pages passed the identical test because their # comparison data is bundled in the JS instead of fetched, which is what # isolated the cause. This gated all ~474 blog URLs, not just new ones. # # The rule this keeps proving: any path the client fetches DURING RENDER must # be crawlable, or the rendered DOM becomes an error state and Google # de-indexes the page. Blocking it does not save crawl budget, it destroys the # page. Before adding a Disallow under /api/, check what the SPA fetches on # first paint. # # The two blog Allow lines are deliberately written WITHOUT a trailing slash. # robots.txt matches on raw prefix, so `/api/blog/posts/` would NOT cover the # /blog index, which fetches `/api/blog/posts?category=...` (no slash before # the query string). Without the slash both the index and the per-post # `/api/blog/posts/` and `...//related` calls are covered. # # NOTE: /api/blog//image (og:image) is still blocked by the /api/ # Disallow, so post thumbnails cannot be fetched for rich results or Discover. # That was the deliberate 2026-05-27 crawl-budget call, left as-is rather than # reversed silently. Worth revisiting: 474 posts is a real thumbnail surface. # # 2026-08-06: the personalities line had the SAME trailing-slash bug the two # blog lines above were written to avoid, three lines under the comment # explaining it. `/api/personalities/` does not prefix-match # `/api/personalities?sort=new-this-week&limit=12&fields=card`, which is what # /explore, /directory and the homepage rails actually fetch # (client/src/hooks/use-explore-data.ts:96,120,158), because the character after # "personalities" is "?" and not "/". It fell through to `Disallow: /api/`. # Measured before the fix: Googlebot's HTML for /, /explore and /directory each # contained ZERO /chat/ links, so the only crawl path to 8,607 character pages # was the sitemap plus sibling links. Dropping the slash still covers the # per-character `/api/personalities/` calls, since it is a prefix match. # AI training and corpus crawlers (2026-09-30). robots.txt is their only # control: they honor no noindex, so the private /admin pages are Disallowed # for them, while search engines may fetch them and read the noindex. A named # group inherits NOTHING from `*`, so this one repeats every rule the `*` group # below carries, and adds only /admin. Keep the two in step. User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: CCBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: anthropic-ai User-agent: ClaudeBot User-agent: Claude-Web User-agent: Claude-User User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Meta-ExternalAgent User-agent: Meta-ExternalFetcher User-agent: Amazonbot User-agent: Bytespider User-agent: cohere-ai User-agent: Diffbot Allow: / Allow: /api/personalities Allow: /api/blog/posts Allow: /api/blog/categories Allow: /blog-media/ Disallow: /api Disallow: /node_modules/ Disallow: /admin Crawl-delay: 1 User-agent: * Allow: / Allow: /api/personalities Allow: /api/blog/posts Allow: /api/blog/categories # 2026-09-28: a blog post's brand images (og card, hero, card art) are served # from /blog-media//, deliberately OUTSIDE /api so # the og:image and the Article ImageObject are fetchable by every crawler and # unfurler. `Allow: /` already covers it; the explicit line is there so that # any User-agent group added later, which inherits NOTHING from this one, is # caught by server/__tests__/robots-blog-media.test.ts if it omits it. Allow: /blog-media/ # 2026-08-07: both of these lost their trailing slash. A prefix match against # "/admin/" does not cover the bare "/admin", so /admin and /api each returned # 200 with "index, follow" and were crawlable. Same class as the # /api/personalities/ bug fixed the day before, in this same file. # # The longer Allow rules above still win under longest-match, so # /api/personalities and /api/blog/posts stay reachable. # # /admin is NOT Disallowed here since 2026-09-30: it serves noindex (meta and # X-Robots-Tag, server/middleware/transactional-noindex.ts), and a path that is # both Disallowed and noindexed cancels the noindex: the crawler never fetches, # never reads it, and can still index the bare URL from an external link. The # house shape for a private page is crawlable by search engines, noindex, and # Disallowed for AI crawlers only (the group above). Pinned by # server/__tests__/robots-never-disallows-a-noindexed-page.guard.test.ts. Disallow: /api Disallow: /node_modules/ # Sitemap location (index with sub-sitemaps) Sitemap: https://aianyone.com/sitemap.xml # LLMs.txt for AI crawlers # See https://aianyone.com/llms.txt # Full index (top characters per category, blog index, pricing): # See https://aianyone.com/llms-full.txt # Crawl delay for polite bots Crawl-delay: 1