Ranking and citation are not the same job
Googlebot fetches, indexes and ranks. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and Applebot-Extended carry citable facts into generated answers. Both families read robots.txt; they do not share a purpose. Optimising only for Google while blocking AI crawlers at robots or WAF is investing in one surface and walking off the other. GEO is not a new keyword density. It is making sure a citation system can fetch, understand and attribute the page.
robots.txt is advice; HTTP 403 is a refusal
A well-behaved bot that sees Disallow will leave. A 403 never reaches your origin robots.txt at all — the edge answers “Your request was blocked.” On our own audit, the three Hong Kong locale hosts returned 200 to those AI bots. www.igears.net, the hreflang x-default, returned Cloudflare 403 to GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User and PerplexityBot. The fallback locale of the whole cluster was invisible to the crawlers this work exists to reach.
Cloudflare has two switches, not one
If Cloudflare sits in front, the policy is usually in the dashboard. Super Bot Fight Mode / “Block AI Scrapers and Crawlers” produces the 403. Managed robots.txt injection prepends Disallow rules for GPTBot, ClaudeBot, Google-Extended and Applebot-Extended before your origin file. Duplicate user-agent groups are resolved differently by different parsers. Origin should state the citation-bot allow list clearly; the dashboard must not overwrite it. Scrapers with no citation behaviour can stay blocked at the edge.
Bytespider is not automatically junk if you serve Chinese
Western templates often Disallow Bytespider to save bandwidth. We also run a Simplified Chinese host. Bytespider feeds ByteDance; Doubao is one of the most used assistants in that market. Blocking it on cn.igears.com.hk would exit the surface that locale was built for. We allow Bytespider on the three Hong Kong hosts, keep it blocked on the Global/.net asset host, and would rate-limit rather than ban if volume became a problem. ImagesiftBot, which scrapes images and does not cite, stays Disallow.
301-to-category is a soft 404
Redirecting /news_details/some-article to /insights/ looks tidy and throws away the ranking signal. The right order is: 301 to the equivalent new article; 410 if there is no equivalent and no traffic; recreate the piece if it still earns impressions. This article replaces /news_details/seo-robots-txt, which still surfaced for queries about our robots.txt. A 410 would have discarded a live asset on the exact topic we need authority in.
What a maintainable robots.txt looks like
User-agent: * Allow: / already permits most bots. Repeating Allow: / under GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Applebot and Applebot-Extended is documentation: it stops a future engineer adding a blanket Disallow “to save bandwidth”. Date the policy. Use a per-host absolute Sitemap. Serve production robots.txt by an explicit nginx alias. Never leave a file named robots.txt in the docroot whose contents are Disallow: / — that file is a tripwire disarmed by one location block. Staging should alias a differently named file such as robots.staging.txt.
llms.txt is an authoritative summary, not a substitute
llms.txt is a short company and solutions list for assistants. A stale file is worse than none, because the assistant treats it as official. We generate it from the same company record and page registry, and serve text/plain; charset=utf-8 so unfamiliar MIME types are not downloaded unparsed. It supplements robots, sitemaps and JSON-LD. It replaces none of them.