AI CRAWLER INDEX · UPDATED WEEKLY
Which sites let AI crawlers in.
And which ones don't.
We read the crawler rules of the web's most-visited sites every week, and show you which ones deliberately named an AI crawler — not just the ones with blanket rules that catch everything. Check any site yourself, below.
CHECK ANY SITE
See what a site tells AI crawlers, right now.
Enter any domain — not just the ones in the index. We read its crawler rules live and show you exactly who it lets in. Instant, and nothing is saved.
Current status
Every site we track, against every crawler we watch, most-visited first. Filter to one crawler, search for a site, or click any dot to see the exact rule behind it.
* block
Recent changes
Crawler policies that moved between the last two runs — a site adding a block, or reversing one.
Methodology
Open data, open method, nothing papered over.
The rule that actually applies
When a site has both a general rule and one written for a specific crawler, the specific one wins — exactly as the crawler itself would read it.
Only one of them is a decision
Most sites have blanket rules that catch every crawler by accident. Naming an AI crawler is a choice someone made about it. We show you which is which — filled dots were named, hollow ones just got swept up.
A gap is shown as a gap
If a site doesn't respond, it's marked not yet checked — never quietly counted as welcoming every crawler.
A request, not a lock
These rules record what a site asked crawlers to do. They aren't enforcement, and not every crawler obeys them.
What the statuses mean. Blocked means the whole site is closed to that crawler. Partial means some paths are closed and the rest are open. Allowed means nothing is closed to it. Not yet checked means the site didn't respond when we last looked — a gap in our coverage, never a claim that the crawler is welcome. Which sites are in the index. The 500 most-visited sites on the web that serve real pages, ranked by an independent traffic ranking rather than picked by hand. Infrastructure, ad networks and adult sites are left out.