AI detection is a category with a credibility problem. Tools advertise "99% accuracy," teachers act on the verdicts, and every few months another story breaks about a student falsely accused by a percentage nobody can explain.
We build an AI detector inside Krex AI's Analyzer suite. Over the past week we did something the category mostly avoids: we built a labeled evaluation set, measured our own detector against it until we understood exactly where it breaks, tested the most promising upgrade in the research literature at three different scales — and then set our product claim to match the data instead of the other way around.
This post is the full result, including the parts that didn't go our way.
How we tested
We generated a fresh evaluation set instead of relying on the public benchmarks most detectors quote, because those benchmarks are built from 2022-era chatbot text — and detecting 2022 AI in 2026 is a solved, irrelevant problem. Our set:
- 96 modern AI documents — current-generation models writing across six genres and four voices, from "default assistant" to "write casually, with personality"
- 24 adversarially humanized documents — the same AI text run through a "make it sound human" rewrite, which is what anyone using a humanizer tool produces
- 105 real human documents — public pre-AI-era answers plus never-published private writing, including imperfect-grammar, non-native-English text
Every document got scored, and we fixed our false-positive budget first: no more than 1% of human documents may ever be flagged. That constraint is the product. Catch rate comes second, always.
What our detector actually does well
It catches unedited AI drafts. On default-voice AI text — the report someone pasted straight out of a chatbot — our primary signal separates AI from human at 0.87 AUC. That's the honest use case: "did this arrive untouched from a model?"
It almost never accuses a human. At our threshold, 1% of human documents flag, by design. And in a test we didn't expect to become a headline: we ran five real, never-published documents by a non-native English writer — imperfect grammar, real-world business writing — through the full detector. Zero flagged. Non-native writers are the most documented victims of commercial AI detectors; ours is deliberately tuned so that the cost of caution falls on the catch rate, not on the writer.
It shows its work. Every flag comes with the measured signals behind it, passage by passage, alongside the counter-signals pointing the other way. No unexplained percentage. If our detector suspects a paragraph, you can read exactly why — and disagree.
What walks right past it — and past everyone
Here's the part most detection tools won't print: casually-styled AI text defeats our detector, and our testing says it defeats the underlying methods everyone uses. Tell a modern model to "write this casually" — which is how most real people prompt — and detection collapses. Run it through a humanizer and the text often scores more human than actual humans.
We didn't accept that quietly. The research literature's strongest answer is likelihood-based detection (Binoculars, Hans et al. 2024) — a clever method that published 0.95+ accuracy against the generators of its era. We implemented it faithfully and tested it at three scales, ending with the paper's own reference configuration on rented GPU hardware:
| Model pair | Modern AI (AUC) | Humanized AI (AUC) | Score spread |
|---|---|---|---|
| Small (0.5B) | 0.36 | 0.11 | 0.058 |
| Medium (1.5B) | 0.54 | 0.20 | 0.059 |
| Reference (7B — the paper's config) | 0.78 | 0.26 | 0.058 |
Accuracy climbed with scale, exactly as theory predicts. But two things never moved: the score spread stayed compressed at every size — meaning the signal lives in a sliver where noise competes with truth — and humanized text stayed below a coin flip throughout. The method that benchmarked brilliantly against 2022 generators erodes against 2026 ones. As far as we know, nobody had published that measurement. Total cost of finding out: about a dollar of GPU time.
What we did about it
We changed the product to say only what the data supports:
- Flags mean something: "this reads like an unedited AI draft," with the evidence attached — signals, never an accusation.
- A clean scan means exactly this: no AI-draft signals found. It is not proof of human authorship, and our interface says so on every result.
- Formal, templated writing — business plans, spec sheets, structured lists — gets extra protection, because "reads like a template" is not evidence of authorship in either direction. We learned this one from a false positive on a document our own founder wrote.
If a detection tool tells you it can reliably catch casually-written or humanized AI text, ask to see the curve. We looked very hard for that capability, at reference scale, with money on the table. It isn't there — not in our stack, and if our measurements generalize, not in theirs.
Why publish this
Because the alternative is the status quo: tools that sell certainty they don't have, and people who get hurt by verdicts that were never as solid as the percentage made them look.
A detector that knows its limits is genuinely useful — as a first-pass screen, as a way to understand why a document reads the way it does, as one signal among several that a human being weighs. That's what ours is, inside an Analyzer suite that also checks citations against real registries, scans for plagiarism on the open web, analyzes authorship consistency, and helps you polish your own writing.
Measured, honest, and it shows its work. We think that's the only kind of AI detector worth shipping.
The Analyzer suite — AI detection, grammar, plagiarism, citations, authorship, and rewriting — is part of Krex AI. Every claim in this post comes from our evaluation runs of July 2026; we re-run them monthly against current-generation models, because detector truth decays as generators improve.