Last week we published everything we measured about our AI detector, including the parts that didn't go our way. This week a human reviewer took that same detector, fed it a real university assignment from 2023, and found the worst false positive of the entire project.
Passage #24 of that assignment was a verbatim quotation of South Africa's Children's Act — quotation marks around it, citation attached: (South African Government, 2005). Text drafted by a legislature two decades before ChatGPT existed. Our detector's verdict: STRONG AI-generation signals · 75% · confidence HIGH.
That is the worst category of false positive a detector can produce. Quoting law correctly requires copying it exactly. A student who did everything right — quoted, cited, attributed — got accused for it.
This post is the story of why that happened, the second false positive we found while fixing it, and why the real fix turned out to be bigger than either bug: our detector's entire vocabulary could accuse, but it couldn't affirm.
Anatomy of a bad verdict
Our detector scores every passage with two layers. A measured layer computes stylometrics in plain code — sentence-rhythm uniformity, formulaic transitions, stock AI phrases — and emits signals you can read. An LLM judge reads the passage and estimates the probability it was machine-generated, with its reasons attached. By design, the judge's impressions are labeled "corroborating only, not proof" in the interface.
On passage #24, the report's own internals confessed the failure:
- Measured signals: "No measured signals — the rhythm and phrasing read naturally." The quantitative layer found nothing.
- Model impressions: "Structured legal obligations with exhaustive list… suggests AI generation."
The only evidence in the entire report was the layer explicitly labeled non-evidence — and the verdict still read STRONG at HIGH confidence. Three separate design flaws had to stack for that to happen:
1. The detector didn't know what a quotation is. Asking "did AI write this?" about a quotation is a category error — the quote is supposed to be a verbatim copy of someone else's words. Our engine scored it like any other paragraph.
2. A confident judge could convict alone. Our combiner capped any uncorroborated judge read at the review tier — except above a threshold of 75, where we'd allowed the judge to convict solo to protect catch rate. Legislative prose pushed the judge to exactly 75. The sibling passages in the same document, judged 60–70, were correctly held at the cap; #24 walked through the one door we'd left open.
3. Formality bias at maximum volume. That exhaustive enumeration — "dentist, homeopath, immigration official, labour inspector, midwife…" — is how statutes are written. Legislative drafting is deliberately exhaustive so nobody escapes the definition. The judge read the most human-institutional prose that exists as machine fingerprints.
The fixes, with receipts
Quoted material is no longer scored. Passages that are substantially quotation — one long verbatim block, or a major quoted share with a citation attached — are excluded from authorship analysis entirely and labeled QUOTED. Fiction dialogue still gets scored; the detection is structural and deliberately conservative.
No more solo convictions. The 75+ carve-out is gone: with zero measured signals, the judge caps at the review tier at any confidence. We didn't take this on faith — we ran our full fixture suite with the carve-out disabled first, and it cost zero known-AI catches. Real AI text carries measured signals; the only verdicts that dropped were exactly the uncorroborated ones that produced this bug.
Confidence now means agreement. "HIGH" requires the measured layer and the judge to point the same way. A judge with no corroboration reads moderate, at best. The label "corroborating only" is now enforced in math, not just printed in the UI.
Then we found the citation bug
While verifying the fix, we fed the detector a paragraph of ordinary academic prose — paraphrased claims with inline citations, the kind every literature review contains. It scored 56, ELEVATED, and dragged the whole document verdict to "heavily AI edited."
The judge's reasons, verbatim from our logs: "references to studies and legal documents, such as '(Mathews & Benvenuti, 2014)'… characteristic of machine-generated text." It named the citations themselves as evidence. And the one measured signal that corroborated it — uniform sentence lengths — is endemic to academic register. Scholarly writing cannot avoid that rhythm.
So citations were being read as AI tells, and academic cadence was vouching for the accusation. The fix has two parts: the judge is now explicitly instructed that citations, attributed claims, and references to studies or legislation are scholarly convention, never machine evidence — and on citation-dense passages, rhythm-based signals no longer count as corroboration at all. Without a content-level tell, cited prose caps below the line where passages feed the document verdict. Measured result: 56/ELEVATED → 50, document verdict back to "mixed." Bibliographies, it turned out, were already safe — reference entries are too short to score, by design.
The bigger problem: a vocabulary that could only accuse
Fixing the bugs still left the thing our reviewer put most precisely: our verdict scale — LOW / MODERATE / ELEVATED / STRONG — was a one-directional accusation ladder. Every rung measured an amount of AI. Even the bottom rung read as "AI, but only a little." There was no rung that said human, and no rung that honestly said we don't know. And the middle band's yellow chip was a mild accusation regardless of the word printed on it — yellow is a warning color.
So we replaced the whole vocabulary with a two-sided one:
| Score | Verdict | Color | What the user reads |
|---|---|---|---|
| 0–25 | Reads human | green | "Human-typical variation is present; nothing AI-typical was measured." |
| 25–40 | Nothing notable | soft green | "No meaningful signals in either direction." |
| 40–55 | Inconclusive | gray | "Signals are weak or conflicting — this is no verdict, not a low AI score." |
| 55–75 | AI-typical patterns | amber | "Some measured patterns are common in AI drafting. Worth a human look — patterns, not proof." |
| 75+ | Strong AI-typical patterns | red | "Multiple measured signals typical of AI drafting. Still signals, not proof of authorship." |
Three deliberate choices in that table. The middle band is gray, not yellow — gray reads as "no data," which is what that band actually means; our instrument abstained. The accusing bands describe patterns, never authorship — "AI-typical patterns present" is a measurable property of text; "likely AI-written" is a claim about a person. And the low end affirms: imperfect-grammar, non-native-English writing — the most documented victims of commercial detectors — now reads "Reads human," not "LOW."
Every wince becomes a test
The part of this process we'd recommend to anyone building detection tools: each of these failures is now a permanent regression fixture that runs against the live model before any change ships.
- The Children's Act passage must score not scored — QUOTED, forever. If any future change gives quoted statute an authorship verdict, the build fails.
- The citation-dense paragraph must never cross the flag line, forever.
- The known-AI catches and the real human documents — including the non-native-English set — must keep passing on the same run.
A false positive you fixed is an anecdote. A false positive that fails your build is a guarantee.
What hasn't changed
Everything from our measurement post still stands, including the uncomfortable parts. Deliberately casual or humanized AI text still evades detection — ours and, as far as our testing shows, everyone's underlying methods. A clean scan is still absence of evidence, not proof of human authorship, and the interface still says so on every result. We didn't make the detector more confident this week. We made it more honest about the difference between what it measured and what it merely suspects — and gave it, for the first time, a way to say the thing most detectors never say: this reads human.
Detectors are instruments. An instrument that can only swing toward guilty isn't measuring — it's prosecuting. If you're going to point one at real people's writing, build it a dial that can also point the other way.