Measuring surprise: a classical residual as a conditioning signal for visual attention

The construction is old-fashioned on purpose. In the equestrian validation domain, pose estimation extracts a dual skeleton from each frame (39 keypoints on the horse, 17 on the rider), and a bank of kinematic channels is computed from it: joint angles, symmetry measures, depth-consistency terms. A classical statistical baseline, fitted per gait class and pooled across clips, predicts each frame's channels from the recent past. The signal is the residual: observed minus predicted, weighted by channel reliability, collapsed to a per-frame magnitude.

Three constraints drove the choice of a classical model over a learned one. The first is interpretability. A residual from a simple statistical fit has a precise meaning: the component of the observed signal not explained by recent dynamics. A neural forecaster's residual means only that the network was wrong, by some amount, for reasons that are themselves a research question.

The second is redundancy. The VLM is already a large neural system watching the same pixels. A second neural network risks learning overlapping features, and its failures would correlate with the failures it is supposed to flag. A classical model contributes genuinely different information.

The third is gradient hygiene. The predictor runs entirely outside the model's training loop, and its outputs enter as constants, like external sensor readings. Nothing about the trigger can be bent by the optimizer into something easier to satisfy.

The trace below is unedited pipeline output for a single fixed-camera arena clip: 126 frames at 30 fps, labeled and scored as a walk. The camera does not move, so nothing in the trace can be blamed on a shot change.

For the first two seconds the horse walks and the residual hugs zero; observed kinematics match the walk-baseline prediction, with magnitude mostly under 7. Around frame 64 the horse breaks into a trot, surges briefly into a canter near frame 85, and settles back through trot to walk by frame 94. The residual flags the entire transient and peaks at frame 90, the moment the canter breaks back toward trot, at 61.2 — fifteen standard deviations above the quiet baseline. Every checkpoint was verified frame-by-frame against the source video before this note claimed it.

That is the behavior the signal exists to produce: quiet while dynamics follow prediction, loud when the motion regime changes. A downstream model reasoning about this clip should be looking hardest at the window the residual marks.

The right-hand side of the plot needs reading too. The spikes at frames 112 and 120 reach nearly the height of the true peak, and they are not motion: inspection of the source frames shows leg-keypoint tracking degrading as the horse approaches the frame boundary. A residual over pose features cannot, by itself, distinguish surprising motion from a surprised tracker.

One reading treats this as a failure of the signal. A better one treats it as a specification: channels carry unequal reliability at different moments, so the residual is reliability-weighted before it conditions anything. A third, more speculative reading says degraded observability is itself a moment when a reasoning model should distrust its summary of the scene, so firing there is not purely wrong. That argument is untested; until it is, these spikes count as noise.

At training time, ε enters REVEAL as the exogenous conditioning term alongside the model's reasoning uncertainty; the mechanisms it drives are described at block level on the REVEAL page and in the April 2026 provisional filing. The attention-level training runs are being prepared on local hardware. Whether a signal this cheap can move a model's visual grounding is the central empirical question of the project. Results, whichever way they land, will be reported in a future note.