A model rotates a photograph of an object and produces a plausible new view. The result can reveal useful geometric structure. It gives much less evidence that the model could predict what happens when the object is pushed, opened, or heated.

This is a problem I keep returning to from two directions: working on rotational novel-view synthesis, and thinking about models that could operate physical systems. The tempting move is to put these on one smooth capability curve. Better pictures become better world models, and better world models become competent controllers. Each arrow needs its own experiment.

The object that barely changes

In our COIL-100 study, objects differed substantially in how much their appearance changed with rotation. Copying the input image was a surprisingly strong baseline in the low-change regime. The more demanding regime produced a different ordering of methods. A single average concealed that crossover.

A featureless object makes the point without any machine learning. If rotating it produces almost the same image, a predictor can score well while ignoring the requested rotation. The benchmark has supplied a shortcut. This does not establish that the predictor has no geometric representation; it establishes that this test cannot require one.

There is a second difficulty. Two objects can share their visible surface while having different mass distributions, internal mechanisms, or friction. A predictor conditioned on a single image should be uncertain about those differences. A confident answer can be less physically sensible than several incompatible hypotheses.

Replace the picture with a choice

Imagine two sealed containers that look the same. One is empty; the other contains a loose mass. Ask a system to move them without tipping. Before the move, it can choose one inexpensive probe: a small tilt, a gentle acceleration, or another view.

The probe is useful because its possible outcomes imply different actions. Another photograph may resolve geometry while revealing almost nothing about the moving internal mass. A small acceleration may be more informative for this task even if it produces a less visually impressive observation.

A reasonable evaluation would hide the relevant physical variation at training time, offer the same probes to every method, charge for the probes, and score the eventual action. A system should also be allowed to say that the available observations do not identify a safe action. Otherwise the benchmark rewards unsupported confidence.

Three different claims

I would separate visual consistency, intervention prediction, and closed-loop competence. Visual consistency asks whether rendered views agree with observations. Intervention prediction asks whether changing a physical input produces the predicted response. Closed-loop competence asks whether the system can choose actions and measurements while the state evolves.

These claims overlap, but none is a substitute for the next. A compact simulator can have excellent intervention predictions in a narrow domain and terrible visual generation. A video model can produce compelling trajectories while failing to distinguish a change in camera motion from a change in force. A controller can succeed through feedback without identifying every latent parameter.

The goal should determine the representation. A maintenance decision might need to distinguish insulation loss from door traffic. A manipulation skill might need contact transitions rather than a detailed mesh. There is no reason to demand a complete reconstruction of the world when several physically different worlds imply the same correct decision.

A test worth failing

My proposed test is a family of tasks with matched appearances and deliberately varied hidden dynamics. Evaluate passive prediction first, then allow budgeted probes, then introduce a new actuator or embodiment. Report where each method loses its advantage. Keep specialist identification and control methods as serious baselines.

This is an experiment I want to see, rather than a result I can report. Its appeal is that a failure would be legible. The model might lack an identifying probe, fail to update after the probe, or know the dynamics but choose a poor action. “It does not understand physics” is too coarse to tell us which system to build next.

Reading and related work