TMR AI detector accuracy test, and its paragraph problem
TMR flagged 83.3% of human texts as AI in our test, including all 101 human texts that had more than one paragraph. When we removed the line breaks from the same texts, it flagged 25.7% of them. Its high catch rate on AI text (83.3%) comes from the same quirk, so its verdicts say more about formatting than authorship.
Tested
Verdict: Unusable as published. It reacts to paragraph breaks, flagging every multi-paragraph text we gave it, human or AI.
TMR (Target Mining RoBERTa) test results
| Rank | 7 of 7 |
|---|---|
| Balanced accuracy | 50.0% |
| Catches AI text | 83.3% (420 of 504; 95% interval 79.8 to 86.3%) |
| Flags human text | 83.3% (105 of 126; 95% interval 75.9 to 88.8%) |
| Catches humanized AI text | 81.0% (102 of 126) |
| AUROC | 0.605 |
| Speed on an Apple M4 Pro | 0.09 seconds per 1,000 words |
| Type | Trained classifier |
| Threshold | 0.5 (model card (threshold=0.5)) |
| Licence | MIT |
| Model | Oxidane/tmr-ai-text-detector |
Numbers behind this chart
| Item | Rate | Texts | 95% interval |
|---|---|---|---|
| Claude Fable 5.1 | 88.1% | 111 of 126 | 81.3 to 92.7% |
| Claude Haiku 5.5 | 81.0% | 102 of 126 | 73.2 to 86.9% |
| Claude Opus 5.5 | 83.3% | 105 of 126 | 75.9 to 88.8% |
| Claude Sonnet 5.5 | 81.0% | 102 of 126 | 73.2 to 86.9% |
| Humanized Claude Opus text | 81.0% | 102 of 126 | 73.2 to 86.9% |
Numbers behind this chart
| Item | Rate | Texts | 95% interval |
|---|---|---|---|
| Academic abstracts | 16.0% | 4 of 25 | 6.4 to 34.7% |
| Q&A answers | 100.0% | 28 of 28 | 87.9 to 100.0% |
| Encyclopedia articles | 100.0% | 25 of 25 | 86.7 to 100.0% |
| Fiction | 100.0% | 24 of 24 | 86.2 to 100.0% |
| News reports | 100.0% | 24 of 24 | 86.2 to 100.0% |
Numbers behind this chart
| Item | Rate | Texts | 95% interval |
|---|---|---|---|
| under 200 words | 40.2% | 39 of 97 | 31.0 to 50.2% |
| 200 to 299 words | 89.6% | 223 of 249 | 85.1 to 92.8% |
| 300 words or more | 100.0% | 158 of 158 | 97.6 to 100.0% |
What TMR is
TMR, short for Target Mining RoBERTa, is a RoBERTa-base model trained on the RAID benchmark with a technique that retrains on human texts the model previously got wrong. Its model card describes it as built to keep false positives low. We used the threshold of 0.5 from the card.
Results as published
TMR caught 83.3% of AI texts, more than any other detector, and flagged 83.3% of human texts. It flagged every human news report, encyclopedia article, forum answer and fiction passage in the test split. The only human writing it mostly passed was academic abstracts, which it flagged at 16.0%.
Length looked like a factor at first. It flagged 100.0% of human texts of 300 words or more and 63.2% of those under 200 words. Longer texts in our set usually have more paragraphs, though, and that turned out to be the cause.
The paragraph test
Abstracts are mostly a single paragraph, so we scored every test text a second time with its line breaks replaced by spaces. Nothing else changed. TMR’s false positive rate on human texts fell from 83.3% to 23.8%, and on the 101 multi-paragraph human texts it fell from 100.0% to 25.7%. Its catch rate on AI text fell from 83.3% to 38.7%.
So most of TMR’s “AI” verdicts were responses to line breaks. The paragraph break guide compares all seven detectors on the same test.
Human writing TMR (Target Mining RoBERTa) flagged as AI
It is extra payments that you apply to the principal. Here is an example. Let's say that you have a mortgage with a monthly payment of $1000 per month (principal and interest), which you pay on the 1st of the month. Now, let's say that, in addition to your $1000 regular payment, you have an extra $1000 that you would like to pay to reduce your debt. So after you...
Last Sunday evening at the Beverly Hilton Hotel in Beverly Hills, California were the 67th Annual Golden Globe Awards hosted by English comedian Ricky Gervais. The choice of Gervais as host was a move to increase viewership which declined to 15 million viewers last year from 27 million in 2004. Canadian director James Cameron—who previously won in 1997 for his film Titanic—and his high-grossing 3D epic Avatar seemed to dominate...
Labeling data is an important step in the supervised machine learning lifecycle. It is a laborious human activity comprised of repeated decision making: the human labeler decides which of several potential labels to apply to each example. Prior work has shown that providing AI assistance can improve the accuracy of binary decision tasks. However, the role of AI assistance in more complex data-labeling scenarios with a larger set of labels...
Questions people ask
Is the TMR AI detector accurate?
No. It caught 83.3% of AI texts and flagged 83.3% of human texts, a balanced accuracy of 50.0%, which is what flagging everything would score.
Why does TMR flag human writing?
In our test it flagged all 101 human texts that contained paragraph breaks. With the breaks replaced by spaces, it flagged 26 of those 101. Its training data may have had little text with line breaks, but we cannot see that from outside.
Would TMR work on single-paragraph text?
Better, though not well. With line breaks removed it flagged 23.8% of all human texts and caught 38.7% of AI texts.