# How we test AI detectors

Source: https://toolbenchmarks.com/methodology/  
Tested: 30 October 2026. Publisher: Tool Benchmarks. Data licence: CC BY 4.0.

> We score 900 texts of known origin with seven open-source AI detectors at their published thresholds. The 150 human texts were all published before 30 November 2022; the 600 AI texts were written for the test by four Claude models, plus 150 humanized rewrites. We report results on 756 texts that were never used to tune anything, with 95% confidence intervals.

## Human texts

We collected 150 human-written texts, 30 from each of five kinds of writing, all published before 30 November 2022, the day ChatGPT launched.

| Kind | Source | Licence |
|---|---|---|
| Encyclopedia articles | English Wikipedia, the last revision before the cutoff | CC BY-SA 4.0 |
| News reports | English Wikinews, the last revision before the cutoff | CC BY 2.5 |
| Q&A answers | Stack Exchange answers posted and last edited before the cutoff | CC BY-SA 3.0 or 4.0 |
| Academic abstracts | arXiv abstracts first submitted and last updated before the cutoff | CC0 |
| Fiction | Passages from public-domain novels on Project Gutenberg, one per author | Public domain |

A script picked the texts at random with a fixed seed and recorded the source address, date and licence of each one. It kept only whole paragraphs of plain prose between 150 and 400 words and removed markup, references, captions, lists and code. It skipped Wikipedia pages created by bots, Stack Exchange answers with quotes or lists, and arXiv abstracts with formulas.

## AI texts

For each human text we wrote one prompt of the kind a person would type into a chatbot, naming the topic, the kind of writing and the length, for example: "Write a short encyclopedia-style article about Celtica Radio. About 290 words, in plain paragraphs, with no headings or lists." The prompt never quotes the human text.

Four Claude models answered all 150 prompts: Claude Opus 5.5, Claude Sonnet 5.5, Claude Haiku 5.5 and Claude Fable 5.1, which makes 600 AI texts. They ran as Claude Code subagents, 30 prompts at a time, and each was told to answer as it would in an ordinary chat, without trying to sound human or avoid detection. The only change made afterwards would have been removing a title line, and no model wrote one.

We also rewrote the 150 Opus texts with the open-source [humanizer](https://github.com/blader/humanizer) skill, run by Claude Opus 5.5, to measure how editing changes detection.

## Detectors and thresholds

Each detector ran on an Apple M4 Pro with the settings and input cleaning its authors publish. Classifiers flag a text as AI when their score passes the threshold from their model card or code, usually a probability of 0.5, because that is the result a user of the model gets. Fast-DetectGPT ships no threshold for the model we used, so we chose the one that best separated AI from human text on the calibration split.

## Calibration and test split

One in five human texts, chosen by a hash of its id, goes into the calibration split together with every AI text written from its prompt. That split holds 144 texts and is used only to set Fast-DetectGPT's threshold. Every number on this site comes from the other 756 texts: 126 human, 504 plain AI and 126 humanized.

## Measures

The catch rate is the share of AI texts a detector flags. The false positive rate is the share of human texts it flags. Balanced accuracy is the average of the catch rate and the share of human texts passed, so a detector cannot score well by flagging everything. AUROC measures how well the raw scores separate AI from human text across all thresholds, where 0.5 is chance and 1 is perfect. Every rate comes with a 95% Wilson interval, shown in the tables under each chart.

## What the test cannot show

The AI texts all come from Claude models. Text from other chatbots may be easier or harder to detect.

The human texts come from five public sources, and some detectors may have seen similar text in training. Student essays, emails and social media posts are not in the set.

Every text is in English. The human texts are 150 to 400 words long, and the AI texts were asked to match them, though a few ran longer.

The models wrote through Claude Code, not a consumer chat app, so their style may differ slightly from what a chat user sees.

The humanized texts went through one tool, run once, and other editing will give other results.

## Questions people ask

### Can I check your numbers?

Yes. The data page has every text, its source and licence, every detector score and every published result. The method that produced them is described on this page.

### Why only open-source detectors?

Because we only publish numbers we measured, and we can run open-source detectors ourselves without an account or a paid API. Commercial detectors will be added when we can test them the same way.