I show a tool how to load a results CSV, convert milliseconds to seconds, regenerate a plot, and update a benchmark table. A week later it receives another CSV. Should it repeat the procedure?

The demonstration does not answer several questions. The new file might already use seconds. A missing column might mean a different experiment rather than a formatting mistake. The table might contain a manual correction that should survive. The original trace records what happened once, while the automation needs a rule for when it is appropriate to happen again.

Counterfactual Apprenticeship is my proposed local tool for learning that boundary. It starts with a recurring research workflow and asks a small number of questions about the cases around the demonstration. It remains a project proposal, including its planned user study and performance targets.

More than one program fits the example

A candidate hypothesis contains both a program and applicability conditions. Several hypotheses may reproduce the demonstrated output. One always divides a column by one thousand. Another checks a units field. A third applies only to one experiment family.

The first trace does not distinguish them. A counterfactual can: present an otherwise similar file that declares seconds. Ask whether the conversion should still run. The answer changes which programs remain plausible and which future inputs they should accept.

A finite candidate set makes this manageable. A complexity preference discourages elaborate exceptions, but simplicity cannot decide every case. A simple wrong rule is still wrong. Executable checks and additional demonstrations provide evidence; they do not eliminate ambiguity by themselves.

Ask only if the answer changes future work

An automation that asks about every possible difference is unusable. The proposed question policy values a clarification by the expected reduction in future error cost, minus the user’s attention and the cost of delaying the update.

This creates a distinction between uncertainty and consequential uncertainty. The tool may be uncertain about a filename convention that never changes the output. It should care much more about whether to overwrite a manually edited cell. A beautifully calibrated probability over irrelevant hypotheses is not a useful product.

The frequency of the task matters too. Ten minutes of setup can be reasonable for a daily workflow and wasteful for a task that will never repeat. The evaluation must count the repetitions needed to recover that initial attention.

Corrections need scope

Suppose I say, “Keep this manually edited value.” The tool could preserve that one cell, every cell in that row, or every value anyone has ever touched. Each generalization can match the immediate correction and cause a different later failure.

The update should therefore affect the applicability conditions as well as the program. A useful test includes an unrelated input on which the learned procedure should refuse to run. Successful abstention is part of the task, especially when the tool can write to files.

The initial system would use designated files, versions, and an isolated workspace. Those restrictions make the effects inspectable. They do not prove that the learned rule is correct, and rollback cannot recover attention already spent checking a confusing result.

The comparison that could defeat it

Existing demonstration-to-program and workflow systems already cover substantial ground. My proposal names OpenAdapt, TraceCompiler, and Motif as related systems. Generating a reusable script is not the claimed contribution.

The planned comparison gives a strong agent persistent instructions, examples, and a warm reused sandbox. Another baseline receives a generated script with validation and retries. An ablation disables active questions. All methods should retain setup and corrections across repetitions.

The primary measure is human attention per correctly completed update, including teaching, questions, checking, and repairs. I proposed repeated use with about six technical friends, reporting by participant and workflow rather than treating every run as an independent person. If active questioning costs more attention than it saves, the central idea fails even if the scripts execute correctly.

Source and related notes