We sent an AI agent to 100 websites, from the biggest on earth to the niche ones builders live on. Then we ranked every site by how far that agent got: robots.txt rules for AI agents, bot walls, CAPTCHAs, cookie walls, and whether the site speaks Markdown or ships an llms.txt.
Average agent-access score per tier (25 sites each, ordered by popularity).
Click a row to see what the agent hit. Higher score means more open.
Context.dev is the web context API for agents. One call turns any URL into clean Markdown, HTML, screenshots or typed JSON, with JavaScript rendering and proxies handled for you. It crawls, maps and monitors sites too, and it has a hosted MCP server for Claude, Cursor and friends.
context.dev also does what this index asks of every site: llms.txt, Markdown on request, and a robots.txt that welcomes every agent we tested.
Try context.dev, 1,000 free credits/monthFour checks per site, 100 points in total. Every probe is automatic, and the raw results are in data.json.
Can 12 AI agents fetch the home page? Agents acting for a person (ChatGPT-User, Claude-User, Perplexity-User) weigh 4 points each, AI search bots 3, training crawlers 1.5.
A plain HTTP request that says it's an AI agent in its user agent. It has to get real content back, not a 403, a challenge or an empty JavaScript shell.
Unmodified headless Chromium, with no stealth plugins, proxies or CAPTCHA solving. 30 points if the page loads with real content, plus 10 with no cookie banner or 5 if a banner doesn't block the page.
5 points for an llms.txt and 5 for returning Markdown when asked with Accept: text/markdown.
Caveats: we visit each home page up to twice and keep the better result, from one datacenter IP in the EU. Results move with region, IP reputation and time of day, so treat a single row as a snapshot, not a verdict. A cookie banner counts against a site because an agent has to click through it. robots.txt is a request, not a wall. We score it because it says what the site wants. We fetched only robots.txt, llms.txt and each home page, once, and never solved or bypassed a challenge.