AI VIDEO QUALITY

A scan that reads every frame

A small Python script reads every frame of an episode in about a minute and marks the moments most likely to be slop. It finds where to look. It does not decide.

Figure 1: Every frame goes through a scan that flags pops, dissolves, edges, colour shifts and blank backgrounds. Flags go to a judge.

1. The tools

Everything in lessons 6 to 8 runs on one Apple-silicon Mac with free software: ffmpeg to read video and audio, Python with numpy and Pillow for the maths and contact sheets, Apple’s built-in Vision text recognition (called from a twenty-line Swift program) for reading writing, and Whisper speech-to-text running on the Mac’s GPU through Apple’s MLX library. One command runs every check and writes a single summary with a verdict. A 13-minute episode takes about four minutes.

2. What the frame scan flags

The scan shrinks every frame to a thumbnail and compares each one with the frame before. Then it looks for patterns:

  • Cut detection. A big change that stands above its neighbours is a cut. Cuts in these files often spread over two or three frames, so neighbouring large frames are grouped.
  • Dissolve. A frame that looks like the average of the frames four before and four after, when those two differ a lot, is a blend: the ghost from a crossfade. Fades at the very start and end are ignored.
  • Pop. A one- to three-frame jump inside a calm shot: something appears, vanishes, flickers or morphs.
  • Burst. A longer stretch of fast change in an otherwise calm shot. Often real action, sometimes warping.
  • Edge. Change at the left or right edge much bigger than in the centre: someone walking in.
  • Colour. The centre of the shot shifts colour mid-shot.
  • Blank. Large flat areas: a missing background.

Each flag gets a strip of three frames, before, at and after, so a reviewer can judge it in a second.

3. Test it on faults you already know

Before trusting a check, run it on an old version whose faults are written down. On the first pilot, the scan found every crossfade the human review had found. That is the only honest way to know a check works on your art style.

4. Its blind spot, and why it matters

Comparing a frame with the one before only catches sudden change. On the second version of the pilots the scan raised 59 and 91 flags; a judge reduced them to 3 and 6 real problems. And some real problems raised no flag at all:

  • an error that is wrong for the whole shot (a face painted red from the first frame);
  • an error that creeps in slowly (a rubber duck growing over two seconds);
  • an error between shots (the jumper that became a coat);
  • an error of meaning (a loaf floating, a creature standing in a sneaker).

Research on generated-video benchmarks shows the same limit: “temporal consistency” scores measure similarity between neighbouring frames, so they cannot see these either. The fix is to compare against something fixed (the reference pack, the state ledger, the script), which is what lessons 7 to 9 do.

Layman rule: A detector only sees what it compares against. Choose the comparison before you choose the tool.