Can AI read and understand a long document?

The answer

Stage 3: independently tested.

The rule that placed itSomeone other than the maker has tested it: an independent benchmark, a peer-reviewed paper, or a hands-on test by a trusted outlet.

The evidenceAsked to verify true and false claims about recent novels read in full, no open-weight model beat random chance and GPT-4o scored 55.8%, while human readers found the task easy. This 2024 test predates current models. Karpinska et al., One Thousand and One Pairs (NoCha), arXiv 2406.16264, Oct 22, 2024.

A stage says how far the evidence has got. It is not a rating of the AI and not advice.

1
ClaimedNo evidence yet
2
LaunchedNo evidence yet
3
Independently testedPlaced here
4
Tested with real peopleNo evidence yet
5
Regulator-clearedNo approval process exists
6
Widely used with a verified recordNo evidence yet

The evidence

Everything we found.

In ladder order, newest first within each kind. Makers' own claims are labeled as theirs.

Tested

What independent tests found

NoCha benchmark (Karpinska, Thai, Lo, Goyal and Iyyer)
Asked to verify true and false claims about recent novels read in full, no open-weight model beat random chance and GPT-4o scored 55.8%, while human readers found the task easy. This 2024 test predates current models.
“while GPT-4o achieves the highest accuracy at 55.8%.”
Karpinska et al., One Thousand and One Pairs (NoCha), arXiv 2406.16264 · Oct 22, 2024

Not found in our sources yet: what the makers claim; what has launched; tested with real people; how many people use it. No regulator approves AI for this task, so the approval step doesn't apply.

Beside the stage