# SigniBull — crawler rules
#
# WHY THIS FILE IS SHORT
# The application itself is not crawlable, by design. Everything behind
# sign-in is reached through hash routes (#/home, #/portfolio), and a search
# engine does not treat a fragment as a separate page — so there is nothing
# to disallow there.
#
# The public surface is small and all of it is meant to be found:
# /welcome, /faq, /accessibility, the blog (/blogs plus its articles, moved
# there from the site root on 2026-08-09 -- the old paths are 301s in
# vercel.json), the user guide (/guide plus its chapters, published
# 2026-08-09) and the Academy (/academy plus its lessons, published
# 2026-08-17). Some short guide chapters carry noindex: published and
# linked, deliberately not in the index, and so deliberately absent from the
# sitemap.
#
# NO COUNTS IN THIS FILE, and that is the THIRD time it has been corrected for
# the same reason sitemap.xml's own header gives. It said "four articles" and
# "the 21 indexable ones" while the sitemap held eleven and fifty-eight; the
# correction that removed those left "the public surface is 26 URLs" standing
# one paragraph above the rule against it, and by 2026-08-21 the sitemap held
# seventy. A number written in prose rots the first time somebody publishes a
# page and does not think to come back here. Count the <loc> elements in
# sitemap.xml if you need a number; it cannot disagree with itself.
#
# Requesting a crawl delay or blocking specific bots is deliberately absent.
# The site is static behind a CDN; crawler traffic costs nothing.

User-agent: *
Allow: /

# Endpoints, not pages. They require a session or a secret and answer JSON, so
# a crawler gets nothing but 401s — and a 401 in Search Console's coverage
# report is noise that hides real problems.
Disallow: /api/

# A fixture used to seed the browser for documentation screenshots. Not a page
# anybody should land on from a search result.
Disallow: /shot-seed

# Design mockups and review dashboards. They are deployed because it is useful
# to open one from a phone, but they are working documents: half-finished
# copy, notes to ourselves, and pages that deliberately duplicate a real one
# while it is being designed. Two reasons to keep crawlers out — nobody should
# reach a mockup from a search result, and /mockups/faq.html must never
# compete with the real /faq for the same query.
Disallow: /mockups/

# ---- AI assistants and their crawlers ----------------------------------
#
# These are welcome, and saying so explicitly is the point of this block: the
# `*` group above already allows them, but an AI-visibility strategy that
# depends on a wildcard is one restrictive edit away from silently ending.
#
# Anthropic runs THREE agents and they do different jobs:
#   ClaudeBot        - the general crawler
#   Claude-User      - fetches a page when someone asks Claude to read it
#   Claude-SearchBot - fetches to improve search results
# Blocking one and not the others is a common accident; all three are here.
#
# THE TRAP IN THIS FILE FORMAT: a named user-agent group REPLACES the `*`
# group for that agent -- it does not add to it. So every Disallow above has
# to be repeated here, or these crawlers would be the only ones allowed into
# /api/ and /mockups/. That is why the rules are duplicated rather than
# tidied up.
#
# Trade-off accepted deliberately: allowing these permits training use, not
# just citation. For a site whose whole problem is that nobody knows it
# exists, being read is worth more than being withheld.

User-agent: ClaudeBot
User-agent: Claude-User
User-agent: Claude-SearchBot
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: CCBot
User-agent: Applebot-Extended
Allow: /
Disallow: /api/
Disallow: /shot-seed
Disallow: /mockups/

Sitemap: https://www.signibull.com/sitemap.xml