A question arrives after an image is gone. It might ask whether there was food in the scene, which color dominated, or which of several similar images was shown. A small memory cannot preserve everything. It has to choose before learning which detail will matter.

That timing was the part of my visual-memory project I cared about. If the query is already visible, the selector can keep the answer. Before the query, it must spend a fixed budget on information with uncertain future value.

A constrained version of remembering

The project uses a menu of frozen visual features: language-aligned gist, self-supervised gist, object features, and a low-level color and texture summary. A policy selects exactly K feature units. Frozen probes then ask attribute and recognition questions using only the retained units.

Freezing the representations and probes matters. It makes the experiment primarily about selection within this menu, rather than allowing a better feature extractor to masquerade as a better forgetting policy. It also limits the claim. The policy is choosing among precomputed representations; it is not learning a general theory of biological memory.

At K = 2, the learned selector reached a combined reward of 0.908, compared with 0.831 for random selection and 0.864 for the strongest discrete heuristic in the reported test. The budget sweep is more revealing than that single operating point: the advantage was largest at K = 1 and faded by K = 4, where the task began to saturate.

Those units are feature slots, not an equal number of physical bytes across every method. The continuous-compressor recognition comparison also has a shared query–gallery encoding confound. The cleaner conclusion concerns the discrete menu and the attribute comparison. It would be easy to overstate the result by turning every axis into one leaderboard.

A future question is a distributional assumption

A memory policy is implicitly betting on the questions that will arrive. Suppose it mostly sees semantic questions during training. Preserving semantic gist is sensible. A new stream of questions about small spatial details can make the same policy look forgetful.

The natural extension is to vary the future query distribution after memory has already been committed. How quickly does the policy lose value? Can it preserve a small amount of information that protects against plausible shifts? How much reward does that insurance cost on the original distribution?

This is also where a reconstruction objective can be misleading. A trace that supports a visually faithful reconstruction may still omit the one feature needed for a later decision. Conversely, a compressed trace can answer the task without reconstructing the image. Evaluation should ask what the retained representation permits someone to do.

The next memory may need a different kind of correctness

My separate work on composed Hebbian memories makes a related failure concrete. A first memory can decode a fact correctly while emitting a continuous vector that is a poor input to the next memory. Single-hop success does not automatically preserve a multi-hop answer.

Nearest-neighbor re-encoding can repair that interface in the constructed setting, but its dictionary has a memory cost. Ignoring that cost would let the “small” memory borrow an uncounted external store. The downstream use determines what must be preserved, and the whole interface belongs in the budget.

This is the connection I want to investigate further: selecting information for an unknown query, then preserving the right structure as it moves through several representations. The first problem concerns what to keep. The second concerns what a later computation can still recover from it.

A useful next test would combine them. Choose a bounded visual trace, pass it through a learned or constructed intermediate memory, and ask delayed questions under query shift. Compare end-to-end performance with the sum of individually good components. I would expect that comparison to reveal failures that neither component’s original benchmark sees.

Reading and related work