MAGE AI detector accuracy test
MAGE caught 46.4% of AI-written texts in our test, fewer than half, and flagged 19.0% of human texts as AI. It did especially badly on encyclopedia writing, flagging 40.0% of human Wikipedia excerpts, and it caught only 20.6% of AI texts under 200 words.
Tested
Verdict: A research detector that misses most AI text from current models and flags a fifth of human writing, so it is not useful for checking real documents.
MAGE test results
| Rank | 4 of 7 |
|---|---|
| Balanced accuracy | 63.7% |
| Catches AI text | 46.4% (234 of 504; 95% interval 42.1 to 50.8%) |
| Flags human text | 19.0% (24 of 126; 95% interval 13.1 to 26.8%) |
| Catches humanized AI text | 23.8% (30 of 126) |
| AUROC | 0.666 |
| Speed on an Apple M4 Pro | 0.39 seconds per 1,000 words |
| Type | Trained classifier |
| Threshold | 3.0858 (authors' deployment code (detect, th=-3.0858 on the negated logit)) |
| Licence | Apache-2.0 |
| Model | yaful/MAGE |
Numbers behind this chart
| Item | Rate | Texts | 95% interval |
|---|---|---|---|
| Claude Fable 5.1 | 28.6% | 36 of 126 | 21.4 to 37.0% |
| Claude Haiku 5.5 | 66.7% | 84 of 126 | 58.1 to 74.3% |
| Claude Opus 5.5 | 45.2% | 57 of 126 | 36.8 to 53.9% |
| Claude Sonnet 5.5 | 45.2% | 57 of 126 | 36.8 to 53.9% |
| Humanized Claude Opus text | 23.8% | 30 of 126 | 17.2 to 32.0% |
Numbers behind this chart
| Item | Rate | Texts | 95% interval |
|---|---|---|---|
| Academic abstracts | 12.0% | 3 of 25 | 4.2 to 30.0% |
| Q&A answers | 7.1% | 2 of 28 | 2.0 to 22.6% |
| Encyclopedia articles | 40.0% | 10 of 25 | 23.4 to 59.3% |
| Fiction | 4.2% | 1 of 24 | 0.7 to 20.2% |
| News reports | 33.3% | 8 of 24 | 18.0 to 53.3% |
Numbers behind this chart
| Item | Rate | Texts | 95% interval |
|---|---|---|---|
| under 200 words | 20.6% | 20 of 97 | 13.8 to 29.7% |
| 200 to 299 words | 49.4% | 123 of 249 | 43.2 to 55.6% |
| 300 words or more | 57.6% | 91 of 158 | 49.8 to 65.0% |
What MAGE is
MAGE comes from the paper “MAGE: Machine-generated Text Detection in the Wild” by Li and colleagues, published at ACL 2024. The detector is a Longformer model trained on text from many earlier language models. We used the exact decision rule from the authors’ own deployment code, which flags a text when the model’s machine score passes 3.0858, and cleaned each text the way their code recommends.
Results
MAGE caught 46.4% of AI texts. It did best on news, catching 91.7% of AI news articles, and worst on academic abstracts, at 12.0%. It caught 66.7% of Claude Haiku 5.5 texts and 28.6% of Claude Fable 5.1 texts.
On human writing it flagged 19.0% overall, with encyclopedia articles at 40.0% and news at 33.3%. Fiction and Q&A answers were rarely flagged.
Length and editing
Short texts were MAGE’s biggest weakness. Its catch rate rose from 20.6% under 200 words to 57.6% at 300 words or more. After a humanizer rewrite it caught 23.8% of Claude Opus 5.5 texts, down from 45.2%.
It ran at 0.39 seconds per 1,000 words on an Apple M4 Pro.
Human writing MAGE flagged as AI
It is extra payments that you apply to the principal. Here is an example. Let's say that you have a mortgage with a monthly payment of $1000 per month (principal and interest), which you pay on the 1st of the month. Now, let's say that, in addition to your $1000 regular payment, you have an extra $1000 that you would like to pay to reduce your debt. So after you...
Last Sunday evening at the Beverly Hilton Hotel in Beverly Hills, California were the 67th Annual Golden Globe Awards hosted by English comedian Ricky Gervais. The choice of Gervais as host was a move to increase viewership which declined to 15 million viewers last year from 27 million in 2004. Canadian director James Cameron—who previously won in 1997 for his film Titanic—and his high-grossing 3D epic Avatar seemed to dominate...
Despite the prevalence of community detection algorithms, relatively less work has been done on understanding whether a network is indeed modular and how resilient the community structure is under perturbations. To address this issue, we propose a new vertex-based metric called "permanence", that can quantitatively give an estimate of the community-like structure of the network. The central idea of permanence is based on the observation that the strength of membership...
Questions people ask
Is the MAGE AI detector accurate?
Not on modern AI text. It caught 46.4% of AI texts and flagged 19.0% of human texts in our test, a balanced accuracy of 63.7%.
Does MAGE work on short texts?
Poorly. It caught 20.6% of AI texts under 200 words, against 57.6% of those with 300 words or more.