Why I Made Six AI Reviewers Disagree

I was ready to rebuild the Chatterbox website crawler.

Chatterbox turns a business website, its documents, and its supplied FAQs into grounded answers for visitors. Its first crawler worked one page at a time. It could not resume after a failure, did not retain the page behind each extracted fact, and produced thousands of near-duplicate facts on websites built from repeated location pages.

The replacement had a page ledger, sitemap-first discovery, parallel fetching, incremental recrawls, path exclusions, page-level attribution, and a merge stage for repeated facts. I had worked through the product decisions. The design document was detailed enough to turn into an implementation plan.

Before writing that plan, I gave the design to six AI reviewers.

They returned eight blockers. The first implementation task changed. Several data-model decisions changed. One input format disappeared. Two cost claims did not survive the review.

The six reviewers also disagreed with each other.

That was the useful part.

One review prompt tends to produce one blended opinion

I have often asked an AI coding agent to review a plan before implementation. A prompt such as “check the architecture, security, product fit, and edge cases” sounds broad. The answer usually settles into one voice. It notices several categories of concern, then smooths them into a tidy list.

That makes the review easy to read. It also hides trade-offs.

A security reviewer should distrust the architect who wants fewer moving parts. A product reviewer should question technical work that users cannot understand. A QA reviewer should try the ordinary mistake that neither the architect nor the security engineer considered important. A reviewer checking claims should be willing to stop the entire plan over one load-bearing number that nobody can support.

I built Pantheon around those conflicts. It is an open-source Claude Code skill and plugin that sends the same work to six isolated sub-agents:

• Athena, Chief Architect, looks for code we should not write, existing tools we should reuse, and structures that will fail at real scale.
• Ares, Security Engineer, attacks trust assumptions and every “good enough” safeguard.
• Hera, Chief Product Officer, ignores engineering elegance and asks whether the work solves a problem a user has.
• Hephaestus, Tech Lead, traces the built system and separates reachable failures from theoretical ones.
• Hermes, User and QA reviewer, makes ordinary mistakes and checks whether the user can recover.
• Apollo, Research Reviewer, checks whether claims, figures, sources, and reasoning deserve confidence.

Each agent receives only its own reviewer brief, the work under review, and the stakes tier. It cannot see what the other five found. Shared context would let the first persuasive argument anchor the rest of the review.

For the crawler, I marked the stakes as production. Real businesses would depend on it, and it would process untrusted websites inside a multi-tenant system.

Ares changed the first implementation task

The design proposed a small thread pool for fetching pages. Network fetching is mostly waiting, so a few concurrent requests appeared to be a contained performance improvement.

The existing crawler protected against server-side request forgery by validating a hostname and temporarily replacing Python's process-wide DNS lookup with a pinned result. The old sequential worker made that approach safe enough. Several fetch threads could replace the same global function at the same time. The proposed concurrency reopened the DNS-rebinding path that the safeguard existed to close.

Ares marked it as a blocker. The crawler could not add fetch concurrency until it replaced the global patch with per-request IP pinning.

The same review found another network route through sitemap parsing. A sitemap belongs to the website being crawled, so it is untrusted input. A permissive XML parser can resolve external entities and turn the sitemap into a way to read internal addresses or cloud metadata. The revised design required defusedxml, bounded sitemap indexes, and a test fixture containing a malicious external entity.

Raw regular expressions in the path-exclusion field created a third problem. A short pattern such as (a+)+$ can consume a worker through catastrophic backtracking. Limiting the pattern to a few hundred characters would not make it safe.

None of those findings asked for a general security programme. They named reachable failures in the proposed design and changed specific implementation tasks.

Hephaestus followed the code through the threads

Chatterbox scopes database queries to a tenant through a Python context variable. The design assumed the fetch and extraction stages could keep using the existing tenant-aware model manager inside the new thread pool.

Worker threads do not inherit that context automatically.

The result was worse than a failed query. An ambient model query inside a thread could run without the tenant scope the application expected. Hephaestus traced the actual manager and call path, then required every threaded query to use an explicit tenant filter or run inside a copied context.

He also found that the existing stale-job sweeper watched the source record's updated_at field. A long crawl did not update that field while it worked. A healthy 5,000-page crawl could therefore look dead after sixty minutes and be killed by its own cleanup job.

The fix was a dedicated, indexed heartbeat_at field on each crawl run. Every stage updates the same clock, and the sweeper reads it.

Another existing cleanup rule deleted facts that were not extracted during the current run. That worked when every run processed every page. Incremental crawling deliberately skips unchanged pages, so reusing the rule would delete valid facts from those pages. The new rule removes facts only when their page is known to be gone.

These were integration failures. Each component looked reasonable alone. The failure appeared after following state across the real system.

Hermes removed regex from the product

The security problem with regular expressions had an ordinary-user version.

A business owner could type team, intending to ignore a team page, and also exclude /steam-cleaning. A single . could match almost every path. The crawler might then finish successfully with nothing useful in the knowledge base.

Hermes required a concrete recovery path. The revised input accepts glob-style patterns such as /blog/*, /locations/*, and *.pdf. It warns when the rules remove at least 90 percent of discovered pages and fails clearly when they remove everything.

Hermes also tried changing the “include dated content” setting after the first crawl. The design stored skipped blog pages in a terminal state, so turning the setting on later did nothing. The interface offered a control whose result had already been decided by an earlier run.

Skipped pages now return for evaluation whenever the crawl configuration changes.

The security and QA reviews arrived at the same area from different directions. Ares saw a worker-level denial of service. Hermes saw a person entering a plausible value and receiving a silently empty result. Switching to globs addressed both without adding a regex sandbox to a control meant for non-technical users.

Apollo checked the confident numbers

The draft described the final merge pass as costing about one-tenth of the initial extraction. It also proposed a cosine-similarity threshold of 0.83 for grouping related facts.

The arithmetic did not support the first claim. On a badly duplicated site, the merge pass could cost as much as one or two full extraction passes. The threshold had no calibration dataset behind it.

Apollo marked both as unsupported. The revised plan says what we know:

• the merge pass can be expensive, so it runs only when the ratio of extracted facts to pages shows a real blow-up;
• structural URL families group obvious repetitions before an LLM receives anything;
• the similarity threshold remains a hypothesis until we calibrate it on real sites.

The review also corrected the description of a prevention digest used during extraction. The underlying mechanism could still help, but the prose claimed it retrieved “nearest questions” when it compared a different representation. The plan kept the mechanism, described it accurately, skipped it for small cold-start knowledge bases, and added a counter to measure whether it prevented repeated facts.

A clean number can make a design feel finished. Apollo's job is to make that feeling earn its place.

Hera asked what the business owner would see

The crawler could skip dated pages, merge repeated facts, and archive old material. Most of those decisions happened outside the business owner's view.

Hera did not object to reducing noise. She objected to asking a user to trust an unexplained result. If a crawl found 500 pages and used 120, the dashboard needed to show what happened to the rest. If an automated merge removed a fact from retrieval, the system needed a recovery path.

The revised API exposed counts for ignored, dated, unchanged, capped, failed, and merged pages. Merged facts became archived records pointing to the surviving fact instead of deleted rows. A full merge-review interface stayed out of this release, but an administrator could recover a bad merge and the wire format left room for a future user-facing history.

Hera preserved a dissent in the final verdict: recovery through the admin was weaker than a review surface for the business owner. I accepted that gap for this release and kept it in the record.

The reviewers found three real clashes

Pantheon does not vote. One demonstrated blocker outweighs five mild approvals.

After the six isolated reviews, a separate reconciler merges duplicate findings and identifies incompatible recommendations. It adds no opinion of its own. The crawler review produced three clashes.

The first concerned the merge pass. Athena wanted to avoid an expensive model stage on normal sites. Hera wanted the crawler to solve the repetition problem that prompted the rebuild. The resolver combined their positions: measure the fact blow-up first, group by URL structure for free, run the model only when needed, and archive its removals.

The second concerned scale. Hera and Apollo questioned whether a 5,000-page target represented the small businesses Chatterbox serves. Athena argued that the system already allowed operators to raise the page cap, making the quadratic vector loop a present failure rather than speculative scale work. The resolver adopted Athena's position and moved nearest-neighbour work into PostgreSQL with pgvector, while using one NumPy matrix operation for clustering.

The third concerned the prevention digest. Apollo challenged the claims made for it. Hephaestus found the mechanism reachable and useful enough to test. The resolution kept it with accurate language, disabled it when the knowledge base was too small to benefit, and added a counter that could justify removing it later.

One isolated resolver handled each clash against the production stakes. The final report kept the losing positions in a dissent log. That matters because “resolved” does not mean the rejected concern became foolish. It records the trade we chose.

The review changed the design before code made it expensive

The final verdict was “approve with required changes.” Eight blockers had to move into the specification before implementation:

1. replace process-wide DNS pinning before adding fetch threads;
2. make tenant scoping explicit inside worker threads;
3. parse sitemaps with XXE-safe tools and bounds;
4. give crawl runs a real heartbeat;
5. replace full-run fact cleanup with page-status cleanup;
6. replace raw regex exclusions with globs and coverage guardrails;
7. re-evaluate skipped pages when configuration changes;
8. replace quadratic Python vector loops with the database and NumPy operations already available.

The resulting crawler later merged with 909 backend tests passing. More important for this story, the review moved those decisions while they were still paragraphs in a design document. The first implementation task became “make DNS pinning thread-safe” instead of “create the crawl ledger.”

Six agents can still be wrong

Pantheon does not prove that a design is safe or correct.

The reviewers can share a blind spot. A persona can overstate its case. The reconciler can group two different failures together. A resolver can make the wrong trade. The agents can inspect only the code, documents, tools, and project context they receive.

I still choose the stakes tier, decide which findings to accept, review the changed design, and require tests against the real failure paths. Pantheon gives me a structured argument and an audit trail. It does not take ownership of the decision.

The structure helps me notice when a review has gone soft. Six instant approvals on a substantial production change are suspicious. A security reviewer demanding enterprise controls for a throwaway script has also failed its brief. The stakes tier constrains both complacency and ceremony.

Building with AI: I set the crawler's product boundaries and made the final calls. Six isolated Claude agents reviewed the design through the Pantheon roles; separate agents reconciled and resolved their disagreements. The implementation still went through tests, code review, and a coordinated frontend handoff.

Pantheon is open source

I extracted this review method into the-cybersapien/pantheon. It is MIT licensed and follows the SKILL.md format. Claude Code users can install it as a plugin:

/plugin marketplace add the-cybersapien/pantheon
/plugin install pantheon@pantheon

Then point it at a plan, a diff, a pull request, or a research document:

Have the Pantheon review this crawler design before I build it.
The stakes are production.

Pantheon runs the six independent reviews, detects clashes, resolves them against the stated stakes, and returns one verdict with the dissent intact. Individual Athena, Ares, Hermes, and Apollo commands are available when one narrow review is enough.

I built it because I wanted an AI review to expose the argument inside an engineering decision. The crawler gave the six reviewers enough to argue about, and the design was better for having to answer each of them.