What AUROC means for AI detectors, explained with real results
AUROC is the chance that a detector gives a random AI text a higher score than a random human text. 0.5 is a coin toss and 1 is perfect. In our test of seven free AI detectors, AUROC ranged from 0.579 for the OpenAI RoBERTa detector to 0.877 for Desklib. It shows how well the scores rank texts, regardless of where the detector draws its line.
Tested
The idea in one example
Pick one AI-written text and one human-written text at random and run both through a detector. AUROC is the chance that the AI text gets the higher score. A detector that guesses wins half the time, so its AUROC is 0.5. A perfect detector always ranks the AI text higher and scores 1.
Desklib’s AUROC of 0.877 means its scores put the AI text above the human text in most such pairs, but far from all. Fast-DetectGPT scored 0.815. The OpenAI RoBERTa detector scored 0.579, close to chance on modern AI text.
Why AUROC and accuracy can disagree
A detector turns its score into a verdict with a threshold. AUROC looks at the scores across every possible threshold, while accuracy, catch rate and false positive rate depend on the one threshold the detector actually uses.
That explains some odd results in our test. TMR had a higher AUROC than E5-small LoRA, 0.605 against 0.582, yet TMR is the least useful detector we tested: its default threshold flags 83.3% of human texts. Its scores carry a little information, but its threshold throws most of it away.
How to use it
Use AUROC to compare how much signal a detector’s scores contain, and use the false positive and catch rates to judge how it behaves as shipped. The leaderboard shows both for every detector, and our methodology explains how each is measured.
Questions people ask
Is a high AUROC enough to trust a detector?
No. AUROC ignores the threshold the detector actually uses. Desklib had the best AUROC, 0.877, but at its default threshold it still caught only 58.3% of AI texts.
What is a good AUROC for an AI detector?
For judging individual people, very close to 1, because even small error rates add up to many false accusations. None of the seven detectors we tested came close.