Perplexity and burstiness in AI detection, explained
Perplexity measures how predictable a text is to a language model, and burstiness how much that predictability varies from sentence to sentence. AI text tends to be uniformly predictable, so detectors built on these ideas flag low, steady perplexity. Human writing that is formal or famous is predictable too, which is why such detectors flagged 40.0% of human academic abstracts in our test.
Tested
Perplexity
A language model reads text one word at a time and predicts what comes next. Perplexity summarises how surprised the model was by the words that actually appeared. Low perplexity means the text kept doing what the model expected.
Chatbots write by picking likely words, so their text usually has low perplexity. Many people write in less predictable ways, with odd word choices, asides and abrupt turns.
Burstiness
Burstiness describes how much that surprise varies. Human writing tends to mix plain stretches with unexpected sentences, while AI text often stays at the same level throughout. A text that is predictable from start to finish looks machine-made to detectors that use this idea.
How it shows up in our results
Fast-DetectGPT, the zero-shot detector in our test, refines the perplexity idea: it compares each word with the words the model itself would have chosen. It was one of the two best detectors overall, with an AUROC of 0.815, and it shows the weakness of the approach. It flagged 40.0% of human academic abstracts, which follow strict conventions, and no human fiction at all.
Famous texts expose the same weakness. A language model that has read a passage many times finds it very predictable. Fast-DetectGPT flagged 17 of the 30 famous texts we tested, including the Hamlet soliloquy, which every trained classifier passed. Our guide on how AI detectors work explains how those classifiers differ.
Questions people ask
Do all AI detectors use perplexity?
No. Six of the seven detectors we tested are trained classifiers that learn patterns from examples. Fast-DetectGPT is the one built on predictability, using a refinement of the perplexity idea.
Why does low perplexity not prove AI writing?
Because people write predictable text too, especially in formal genres, and language models find famous texts predictable because they have read them. Fast-DetectGPT flagged 17 of the 30 famous public-domain texts we tested.