I Made Four Websites Agent-Readable. Some of the Checks Were Theatre.

I ran a scanner against my own site and it came back Level 2: Bot-Aware. Twenty-nine out of a hundred.

isitagentready.com is a Cloudflare tool that scores a domain on how readable it is to the AI agents now browsing, comparing, and shopping on behalf of people who used to click through pages themselves. Three tiers sit above the floor: Bot-Aware, Agent-Readable, Agent-Integrated. My own site, cybersapien.xyz, sat at the bottom despite a robots.txt file that already said the right things.

By the end of the day it read Level 3, Agent-Readable, the tier below the top. Sivella picked up the same real infrastructure in the same afternoon. Vowframes and Chatterbox got real infrastructure the same week, and neither one is agent-optimized yet, confirmed by rescanning both rather than assuming the files were enough. Four sites, one underlying question asked four times: is this actually infrastructure, or is it a box the checklist wants ticked?

Why any of this is worth the engineering time

Product discovery is moving off the results page and into a chat window. Someone asks an assistant to compare vendors, summarize what a company does, or find a business that solves their problem, and the assistant answers from whatever it can retrieve and read, not from ten links a person clicks through one at a time. That retrieval step runs on embeddings, not keyword matching. A page that's crawlable, cleanly structured, and says what it does without three layers of marketing scaffolding retrieves better than one that doesn't, before ranking or relevance ever enter the picture.

That's the actual stake behind a scanner score. Not passing an audit. Being the site an agent's retrieval step picks up and reads correctly when someone asks about a product like mine, instead of getting skipped for a competitor whose sitemap resolves and whose robots.txt doesn't fight the agent trying to read it. A lot of the field still isn't readable at all, which means the bar for winning that retrieval is lower than it looks, for now.

Sivella and Chatterbox raise the stakes past discovery-for-a-mention. Both already run a real MCP server behind the marketing site: mcp.sivella.ai for Sivella's household assistant, chatterbox.cbrspn.cloud/api/v1/mcp/ for Chatterbox's knowledge base, not prose describing a capability but a working endpoint another agent can connect to and use. Someone looking for a household assistant or a support chatbot to hand to their own agent has to find the product before they can ever reach the part where they'd connect it. For these two, agent-readability isn't just the step before a mention in a chat answer. It's the step before an agent decides there's a tool here worth adding at all.

It's also why the two buckets of fixes in this piece look nothing alike. A correct sitemap, an honest robots.txt, a real 404, is the same information architecture that's always mattered for search, extended to a client that reads markdown and JSON instead of rendering CSS. DNS-AID and an MCP server card describing a capability I don't have is a different thing: a stub built to be read by the scanner, not by the agent the scanner stands in for.

The canonical fixes were overdue, scanner or not

cybersapien.xyz's build had been emitting sitemap and canonical URLs on the www host while the rest of the site lived at the apex domain. Google's Search Console property was a domain-level property, so it papered over the mismatch on the search side. Agents reading the sitemap directly got no such grace: a www.cybersapien.xyz URL and a cybersapien.xyz URL describing the same page, with nothing telling anything which one was real.

That got fixed alongside a second, unrelated problem: every unmatched path on the site returned HTTP 200 with the homepage. No real 404 existed anywhere. A mistyped link, a stale bookmark, a scraper hitting a path that no longer existed, all of it got a silent, confident homepage instead of an error, an unbounded space of duplicate-looking URLs for anything crawling the site.

Sivella's pass added its own sitemap, robots.txt, and canonical tags in the same style, all anchored to one canonical-origin constant instead of scattered string literals.

Alongside those, on both sites: <link rel="canonical"> and og:url on every page, a real robots.txt, an llms.txt describing the product in plain prose, and on cybersapien.xyz a Markdown twin of every essay served when a client asks for text/markdown, with Link headers advertising the alternate representation per RFC 8288. Sivella got the same markdown-for-agents treatment, plus an MCP server card and an auth.md describing how an agent authenticates.

None of it is exotic. All of it should have existed already. The scanner didn't teach me anything about agents so much as it noticed infrastructure debt that had been sitting there under a passing Lighthouse score, on both sites.

The robots.txt I'd written wasn't the one being served

cybersapien.xyz's scan gave the site 2 out of 2 on bot access rules. That checked out against the file in the repository: 69 bytes, permissive, search=yes, ai-input=yes. I checked the live file anyway.

It was 1,905 bytes. Cloudflare's own "AI Crawl Control" product, a zone-level setting turned on somewhere in the account's history, was silently injecting Disallow: / for ClaudeBot, GPTBot, and a list of other agent user-agents, layered on top of whatever the repository's robots.txt said. The scanner scored the rule as present because the rule was, technically, present. It just blocked the exact category of visitor the rest of the day was trying to let in.

The fix wasn't a robots.txt edit, since that file was never the thing lying. It was a Content-Signal declaration (search=yes, ai-input=yes, ai-train=no) plus disabling the zone-level override so the repository's own file could be the one that shipped. The lesson isn't really about Cloudflare. It's that a passing check on a rule's existence tells you nothing about which layer of infrastructure actually gets the last word, and it's worth checking the live response even when the repo file already looks right.

Refusing to sign a check nobody reads, twice

The scanner also rewards DNS-AID, an IETF draft for discovering an agent's capabilities through a DNS TXT record. Publishing one would have added a few more points, on either site.

I looked at what would consume it. Nothing does. The draft itself gives away how early it is: it tells implementers to use numeric, unregistered key names "until they are registered," and its own example advertises an ALPN protocol ID that isn't registered either. As far as I could tell, the only thing currently reading a _mcp._agents TXT record on either domain is the scanner, which is a circular reason to add anything: the check exists to reward publishing the record, and the record's only reader is the check.

The same reasoning ruled out a stack of other point-scoring surfaces the tool offers credit for on the Discovery side: an API catalog, a WebMCP hook, an A2A agent card. Sivella already had a real MCP server card and a real auth.md, since Sivella's assistant genuinely has an API surface for an agent to call. cybersapien.xyz doesn't, so publishing the same files there would have been describing a capability the site doesn't have. Publishing stub files to score points would be lying to agents, on either site, and I made the same call twice in one afternoon for exactly that reason.

I checked whether Level 4 was reachable at all without one of those files. It isn't, not for this site. The scanner names exactly one thing standing between cybersapien.xyz and the top tier: an MCP server card. Everything else still failing on the Discovery tab, an API catalog, OAuth discovery, an A2A agent card, a WebMCP hook, describes a capability a portfolio-and-blog site has no reason to build. Level 4 is available for the price of one convincing lie. Level 3 is what's actually true, and it stays true until there's a real capability to put behind an MCP endpoint, not a card describing one.

Vowframes and Chatterbox aren't agent-optimized yet

Vowframes and Chatterbox's public marketing sites picked up real infrastructure in the surrounding week too: llms.txt and llms-full.txt on both, a real sitemap.xml, and a robots.txt that keeps the authenticated app out rather than leaving it open by omission. Neither one is where cybersapien.xyz and Sivella ended up, and rescanning both live confirmed that rather than assumed it.

Vowframes passes on robots.txt and sitemap.xml, both correct at the application level: the marketing site's crawl rules and sitemap resolve cleanly, and the private, guest-facing surfaces are excluded. Markdown negotiation is the clearest remaining gap: nothing on the site answers an Accept: text/markdown request with anything other than the same HTML a browser gets, even though /llms.txt exists as a static route. The rest of what the scanner checks for Vowframes is still settling, and I'm not calling a final tier on it here.

Chatterbox scores Level 1, Basic Web Presence, a tier below where the rest of this piece would suggest. Its robots.txt allows /llms.txt and /llms-full.txt by name and disallows the authenticated app, all correct, but it never picked up AI-bot-specific rules or a Content-Signal declaration; the file's only User-Agent line is a bare wildcard. That's the entire gap the scanner names for the next tier. The two files that make a site legible to an agent, llms.txt and a working sitemap, were not enough on their own to clear the first checkpoint for how that agent is allowed to use what it reads.

Vowframes needed one extra layer of judgment the other three sites didn't. One Next.js app serves both vowframes.com, the marketing site, and every {slug}.vowframes.com wedding site off the same route tree, so crawl rules have to decide per request host. Only the host matching the canonical origin gets a populated sitemap and permissive robots rules. Every tenant wedding subdomain gets Disallow: / and an empty sitemap, on purpose, because a wedding site is guest-only. Making the marketing shell agent-readable can't mean making a couple's guest list and photos agent-readable along with it. The checklist looks the same across a portfolio; what it's safe to expose does not.

Where the scanning stops being useful

cybersapien.xyz and Sivella both have the same real infrastructure in place now: correct canonicals, a real sitemap, a real 404, markdown for agents. Chatterbox hasn't cleared Level 2 yet, missing AI-bot rules and Content Signals despite carrying real infrastructure otherwise. Vowframes is close on markdown negotiation and still has other gaps to close. None of the four publish DNS-AID, an MCP server card that describes a capability that isn't real, or any of the other Discovery-tab stub files the tool would reward, and cybersapien.xyz's own ceiling, Level 3 instead of Level 4, is what refusing those files actually costs.

That's the actual decision this week produced, more than any of the scores. A scanner is a good forcing function for finding infrastructure that's been silently wrong, or silently missing, on one site or four. It's a bad judge of which speculative standard deserves engineering time next, because its incentive is to reward publishing the standard, not to check whether anything downstream cares yet, and a bad judge of what a multi-tenant product is allowed to expose in the first place. Both of those calls stayed mine across every site, not the scanner's. What stayed the scanner's job, usefully, was telling me plainly which of my four sites aren't done, when shipping the files would otherwise have looked like finishing the work.

Building with AI: Claude ran the scanner against cybersapien.xyz and Sivella, read the diffs, and proposed and shipped each fix, in the same style, in the same day. The Vowframes and Chatterbox infrastructure was built and reviewed the same way in the surrounding week. I made the calls on which checks were worth shipping and which were stub files dressed as infrastructure, including turning down a free tier upgrade on cybersapien.xyz itself. Rescanning Vowframes and Chatterbox live, rather than assuming the shipped files were sufficient, is what turned up that neither one is finished yet.