# ---------------------------------------------------------------------------
# agecalculator.su — robots.txt
# Rewritten 2026-08-24 against Google's current robots.txt specification.
#
# HOW GOOGLE READS THIS FILE — the trap that already bit this site:
# Google selects the ONE most specific user-agent group matching a crawler and
# ignores every other group, including "*". A per-crawler group containing only
# "Allow: /" therefore switches OFF every Disallow in the "*" group for that
# crawler. This file used to carry "User-agent: Googlebot / Allow: /", which did
# exactly that: /core/ and /includes/ were crawled and returned 200 the whole
# time they were listed as disallowed. Never add a bare per-crawler "Allow: /"
# group unless you specifically mean to exempt that crawler from everything.
# (Groups sharing one user-agent are merged; on an equal-length conflict the
# least restrictive rule wins, so Allow beats Disallow.)
#
# WHAT THIS FILE IS NOT FOR:
# robots.txt controls CRAWLING, not INDEXING. Google may index a disallowed URL
# without a snippet, and disallowing it also stops Google ever seeing a noindex
# or a 404 there — so blocking a page you want removed keeps it in the index.
# Everything that must stay out is enforced at the server instead: /core/,
# /includes/, /content/**.json and every backup artefact return 404, and retired
# pages send "noindex". That is deliberate, and it is why there is almost
# nothing to disallow below.
# ---------------------------------------------------------------------------

User-agent: *
Allow: /

# ---------------------------------------------------------------------------
# AI assistants — allowed on purpose, so the calculators can be cited in
# answers. Cloudflare injects a managed block above this file that disallows
# some of these; because same-user-agent groups merge and the least restrictive
# rule wins on an equal-length conflict, the Allow lines below are what take
# effect. Delete a block to genuinely disallow that crawler.
# ---------------------------------------------------------------------------

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: MetaExternalAgent
Allow: /

User-agent: Applebot-Extended
Allow: /

# ---------------------------------------------------------------------------
# Bulk scrapers that return no citation or referral value.
# ---------------------------------------------------------------------------

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

Sitemap: https://agecalculator.su/sitemap.xml