AI VIDEO QUALITY

A judge that asks narrow questions

Vision AI judges miss far more than they invent, and they lean toward “looks fine”. Ask them small, specific questions built from the script, and make two signals agree.

Figure 1: Instead of one question, is there slop, the judge answers many narrow yes-or-no questions per shot, and two of three signals must agree.

1. What the research says

Published benchmarks from 2025 and 2026 are consistent. On one benchmark for finding errors in generated video, the best AI model scored about 0.25 where people scored about 0.51. Models lean strongly toward calling a clip normal: one scored 97.6% on ordinary clips and 10.4% on physically impossible ones. They answer from what they expect, not from the pixels, and they make little use of motion over time.

In practice that matches this project. A reviewer looking at two or three frames per shot found the floating loaf and the creature in a sneaker, but only because it was asked about each flagged moment against the script.

2. Ask narrow questions, built from the script

“Is there slop in this shot?” is the wrong question. Better:

  • Is the loaf under the baker’s arm in the first, middle and last frame?
  • Is every scripted prop present at the same size at the start and the end?
  • Is anyone standing inside or on something impossible?
  • Does each hand belong to the right character, at the right size?
  • Who is wearing what? Does it match the wardrobe lock?

Asking one question per error type raised a small free model’s score on that benchmark from 0.148 to 0.254, level with the best large model asked the usual way.

3. Describe first, then compare

When shown a script, a judge can “confirm” it without looking. So in the first pass, give it the frames without the script and ask it to list every character, object and action, plus all visible writing. Then compare that description with the script in code or a second pass. Send more frames than you think (one every half second) and zoomed crops of faces, hands, writing and props.

4. Two of three signals, and an escape log

A finding is confirmed when two of three agree: the code check, the first judge and a second judge (a different model, ideally one that accepts real video). Disagreements go to a short human queue. A rules table in code, not the judge, assigns MUST, SHOULD or CHECK, so fix lists are consistent from run to run.

Keep an escape log: every fault a person spots that the pipeline missed becomes a new test case, and if it happens twice, a new question. Re-run the whole test set whenever a model changes.

Layman rule: Do not ask a judge whether something is good. Ask it whether one specific thing is true.