1. What the research says
Published benchmarks from 2025 and 2026 are consistent. On one benchmark for finding errors in generated video, the best AI model scored about 0.25 where people scored about 0.51. Models lean strongly toward calling a clip normal: one scored 97.6% on ordinary clips and 10.4% on physically impossible ones. They answer from what they expect, not from the pixels, and they make little use of motion over time.
In practice that matches this project. A reviewer looking at two or three frames per shot found the floating loaf and the creature in a sneaker, but only because it was asked about each flagged moment against the script.
2. Ask narrow questions, built from the script
“Is there slop in this shot?” is the wrong question. Better:
- Is the loaf under the baker’s arm in the first, middle and last frame?
- Is every scripted prop present at the same size at the start and the end?
- Is anyone standing inside or on something impossible?
- Does each hand belong to the right character, at the right size?
- Who is wearing what? Does it match the wardrobe lock?
Asking one question per error type raised a small free model’s score on that benchmark from 0.148 to 0.254, level with the best large model asked the usual way.
3. Describe first, then compare
When shown a script, a judge can “confirm” it without looking. So in the first pass, give it the frames without the script and ask it to list every character, object and action, plus all visible writing. Then compare that description with the script in code or a second pass. Send more frames than you think (one every half second) and zoomed crops of faces, hands, writing and props.
4. Two of three signals, and an escape log
A finding is confirmed when two of three agree: the code check, the first judge and a second judge (a different model, ideally one that accepts real video). Disagreements go to a short human queue. A rules table in code, not the judge, assigns MUST, SHOULD or CHECK, so fix lists are consistent from run to run.
Keep an escape log: every fault a person spots that the pipeline missed becomes a new test case, and if it happens twice, a new question. Re-run the whole test set whenever a model changes.