What we found
01 / Confidence cannot decide which checks to skip
The verifier separated real successes from failures well: AUROC was 0.89 on contact-rich assembly. But approvals issued at 0.99 confidence were wrong 9.7% of the time, compared with 8.6% of approvals overall. This pattern held across the thinking levels and resolutions tested. Strong discrimination did not make the model’s confidence a reliable self-audit.
02 / The log detects faults in the model’s inputs
Wrist-camera clips were playing about 7% fast. The model’s labels drifted while still looking plausible. Aligning its step boundaries with gripper events estimated playback speed at 1.080, against a true speed of 1.073. Correct clips were estimated at 0.98–1.00.
In a blind test, three speeds were drawn and sealed before the model labeled fresh clips. The audit estimated 0.932, 0.936 and 0.972, against hidden speeds of 0.936, 0.934 and 0.968, flagging all three faults.
Pre-registration result: the registered comparison used the reciprocal of the speed, an algebra error. That criterion fails as written; both the registered and corrected scorings are reported. The audit also missed partial drift when the model mixed two cameras running on different clocks.
03 / The log learns physical completion checks
A reward based only on the model can pay for stopping just before a step finishes. From the model’s own step labels and the instrument log, a learner contrasted each step’s end with the instant before it and wrote readable goal-state rules: an insert or placement must release the part; a pick or removal must hold it and lift it.
The checks were learned on separate recordings without human labels. In the primary held-out configuration, near-complete states rewarded as “done” fell from 36.2% to 10.0%; real completions rewarded fell from 89.4% to 88.8%. A hand-written engineering check reduced the stop-short rate to 3.1%, while rewarding 86.2% of real completions.
On a pre-registered confirmation set of 310 fresh frames, the learned checks reduced stop-short rewards from 25.9% to 7.4%, with no additional real completions rejected. The hand-written check reached 3.7%, with 3.4% of real completions rejected.
Target missed: the learned checks blocked 71% of the model’s false “done” calls, below the pre-registered 75% target. The zero additional completion rejection applies to the fresh-frame confirmation, not every configuration.
04 / Disagreement directs human review
In the reported agreement subset, the verifier was wrong once in 82 steps. The 24 disagreement steps contained 9 of its 10 errors. Reviewing about 23% of steps therefore caught 90% of errors, compared with about two errors expected from a random review of the same size.
This review-routing result is exploratory, with only ten model errors. It is a promising routing signal, not a calibrated guarantee.
Evidence
Read the values in Figure 2
Percentages below are the rounded chart labels. Within each cell: model alone / learned checks / hand-written check. Lower is better for stop-short rewards; higher is better for real completions rewarded.
| Configuration | Stop-short rewarded | Real completions rewarded |
|---|---|---|
| Model A, medium | 36 / 10 / 3 | 89 / 89 / 86 |
| Model A, high | 38 / 9 / 2 | 92 / 91 / 88 |
| Model A, wrist camera | 41 / 8 / 2 | 86 / 85 / 81 |
| Model A, external cameras | 34 / 8 / 6 | 78 / 77 / 75 |
| Model A, 8 frames | 29 / 8 / 4 | 78 / 78 / 74 |
| Model B (smaller) | 86 / 16 / 11 | 86 / 86 / 82 |
| Model B, high | 91 / 16 / 10 | 84 / 84 / 81 |
The goal states learned from the log
| Step | Learned goal state | Hand-written equivalent |
|---|---|---|
| Insert | Gripper opened past 74 mm within 0.5 seconds. | Released. |
| Place | Gripper fully open, having opened more than 30 mm since the step began. | Released. |
| Pick | Gripper closed on the part and the arm moving up. | Holding the part, lifted at least 2 mm. |
| Remove | Gripper closed and the arm at least 5 mm above its lowest point. | Holding the part, lifted at least 2 mm. |
Using the audit
One audit call per log
The research interface audit(model_labels) returns an input-clock estimate, readable completion rules, a physical verdict per step and a review queue. It uses no human labels, task-specific code or additional model calls. The reported runtime was 1.5 seconds per hour of robot work on a laptop. A machine connects through two functions: its log and an event stream from an actuator signal.
Pair the model with the instruments
Positions, forces and actuator states offer evidence that fails differently from a model verdict. If a check cannot be resolved, the system returns unknown rather than approving. The reward becomes a conjunction: the model says the task succeeded and the instruments say the step finished.
A separate engineering-software case
In a separate study, two separately generated simulators agreed on a circuit design that violated its 8 V limit, reaching about 9.94 V. Checking the decision against the continuous-time requirement rejected that design and selected one that passed. This result covers four retrospective cases in one ideal circuit family, using a hand-built reference; it is not yet automatic for new programs.
How the work was done
Data and held-out evaluation
The study used REASSEMBLE (CC BY 4.0): a parallel-gripper robot arm assembling and disassembling parts on a task board. The data includes human action, outcome and skill labels; external and wrist cameras; and gripper, arm and force logs. Every method was designed on 12 recordings and tested on 12 other recordings.
Pre-registered tests
Test rules, thresholds and pass/fail criteria were frozen in hashed records with the code commit before held-out runs. Results are reported whether tests passed or failed, and settings tried during design are disclosed in each record. The clock-comparison algebra error is reported both as registered and as corrected. Tampered proofs, injected faults and near-complete states were used to red-team the checks; gaps found and closed are disclosed in the study logs. Review routing remains exploratory.
What “zero-label” means here
The clock audit uses the model’s labels and the gripper log. Learned completion checks use the model’s step labels and the gripper, arm and force logs. Human labels are used only to score the results, not to build the checks. The model verifier itself still produces the labels being audited.
Limitations
- One robot cell and one dataset. A second machine type remains the next test. Clock auditing needs step boundaries that coincide with instrument events, and missed partial drift across mixed camera clocks.
- Learned checks retain a trade-off. On fresh frames, about 7% of stop-short states were rewarded with learned checks, against about 4% with the hand-written check. The learned checks missed the 75% blocking target. The number of hard negatives was chosen on development data; using three instead of two made one step type substantially worse there.
- Physical completion is not semantic correctness. The checks do not determine whether the right part went to the right place. That judgment still belongs to the model.
- The circuit result has narrow scope. It covers four retrospective cases in one ideal circuit family and relies on a hand-built reference.
- No reward-trained policy was evaluated. These are results on recorded data and saved verdicts. The review-routing counts are small, with ten model errors.