The Robot Already Knows · CosmicBrain Research

What the robot records

NVIDIA’s robot-learning flywheel grades data, rollouts and checkpoints with automated judges, notably Cosmos Reason used as a critic for real and synthetic trajectories. A simpler judge is already on the robot. When a gripper closes on an object, its fingers stop short of the commanded position. This gap, together with how the hand moves while holding, records whether a grasp, lift, carry and release happened.

On RoboArena, the real-robot evaluation NVIDIA has cited for GR00T, this label-free signal ranks ten policies at Spearman ρ = 0.95 against 10,783 human judgments. Applying the same check of what really happened to public code exposed a GR00T loss whose update changes sign with batch layout, normalization statistics that go stale when a dataset grows, and DexMimicGen subtask labels that fire 50 frames early. A separate software-only coordination layer addresses the two-arm GB300 connector handover described in NVIDIA’s October factory report.

Two real RoboArena rollouts: a held object creates a gap between commanded and measured gripper closure; an empty grasp closes fully even as the arm moves as if carrying. — **Figure 1. The signal.** In the first rollout, the shaded gap means an object is in the hand, and the hand then rises more than 3 cm. In the second, the fingers close all the way: nothing was grasped. Cosmos Reason must infer this difference from pixels; the robot records it directly. Select the figure to inspect it at full size.

Why a second channel matters

GR00T policies train on mixtures of teleoperated demonstrations and synthetic trajectories generated with Cosmos and DexMimicGen, then are evaluated on real robots and in Isaac Lab. At every step, data-pipeline labels, Cosmos Reason, world models or human raters decide what counts as success. Optimistic or quietly wrong graders compound the error: bad demonstrations are kept, failed rollouts are rewarded and checkpoints are ranked on noise.

Cosmos Reason judges from a few frames and must infer contact, holding and slip. The robot measures these quantities at every control step. This report asks how far that signal goes and where it could fit into NVIDIA’s stack.

Contributions

A label-free log score that reaches 73% human A/B agreement against 56% for Cosmos Reason 2 (8B), ranks ten policies at ρ = 0.95 and reduces the model’s error by 41% when fused with it.
Evaluation and data uses: anytime-valid A/B decisions with zero wrong calls in the reported test, a veto for weak critics on real assembly failures and an audit of public GR00T humanoid teleoperation data.
Three silent failure modes in public GR00T and DexMimicGen code, each with a minimal witness and an opt-in fix.
A fail-closed coordination layer for the GB300 two-arm connector handover.
A full pre-registered ledger with six negative results, including distillation into Cosmos Reason 2.

Table 1. Where each finding plugs into NVIDIA’s stack
Finding	NVIDIA component	What it changes
Robot log beats and boosts Cosmos Reason 2 on real rollouts (73% vs 56%; 41% lower error when fused)	Cosmos Reason as a critic for real rollouts and synthetic trajectories	Give the critic the robot log as a second channel; veto “success” when the log shows no hold.
Label-free policy ranking (ρ = 0.95) and anytime-valid A/B tests (0 wrong calls)	GR00T checkpoint evaluation on real robots and RoboArena-style evaluations	Rank checkpoints without a rater per rollout; stop A/B tests as soon as one is reliably better.
Flagged teleoperation demonstrations are 62–76% failed or flawed in blind review	GR00T humanoid teleoperation datasets	Filter or down-weight flawed demonstrations before training.
Loss update flips sign with batch layout	GR00T training code, mixed embodiments	An opt-in explicit objective that is invariant to layout.
Stale normalization statistics after dataset growth	GR00T data statistics cache	A source-bound cache that regenerates when data changes.
Contact check can never succeed; labels fire 50 frames early	DexMimicGen subtask annotations	A two-line patch.
Measured-holding release gate	GB300 cable-connector workcell	Release only on measured contact, load and no slip; diagnose failures from the log.

All results use public NVIDIA code, models and datasets. These proposed applications do not establish results on NVIDIA’s internal systems.

Related work

Vision-language critics and reward models, including Cosmos Reason, score rollouts from images or video to filter synthetic data and rank policies; critics trained mostly on successes tend to call attempts successful. World-model evaluation runs policies inside action-conditioned video models. Self-supervised grasp labels from gripper closure date to early large-scale grasping work; this report uses that physical cue as a verifier for evaluation and data. Data-curation methods rank demonstrations by influence or mutual information. Anytime-valid testing gives error guarantees under continuous monitoring.

Method and protocol

Three signals, five physical events

The method reads commanded gripper closure, measured gripper closure and end-effector position from standard robot logs in LeRobot and GR00T parquet layouts. A hold is a settled measured closure that stops short of the command by more than 0.05, with the fingers not fully closed. An empty grasp closes fully, with a gap below 0.03. A lift is a rise of at least 3 cm while holding; a carry is a horizontal move of at least 5 cm; and a release is an opening after a hold.

The fixed, label-free score

Z = held + lifted + carried + released − 0.25 · min(empty grasps, 4) − 1[idle > 0.9]

The formula and thresholds were fixed before any human score was viewed. It needs no training, camera or labels and runs at about 0.3 seconds of CPU per robot-hour.

Learned score and fusion

A gradient-boosted model uses about 30 physics features to predict human progress. It is trained leave-one-policy-out, so it never sees the policy it scores. Fusion adds Cosmos Reason 2’s output as one more feature.

Decisions and pre-registration

To compare two checkpoints, an anytime-valid sequential test uses empirical-Bernstein confidence sequences on per-session score differences. Every study was registered before results were viewed, with its data split, metric, pass bar and a SHA-256 hash: 23 registrations in total. Human feedback used for diagnosis was coded blind.

Robot state and action logs become physical events, then a fixed label-free score, a learned score or a fusion with Cosmos Reason 2, supporting ranking, A/B tests, data audits, diagnosis and vetoes. — **Figure 2. Method.** The same physical events support all three scores and the downstream uses. Only the fixed label-free score needs no video, model or labels. Select the figure to inspect it at full size.

Evidence from real robots

Setup

The main testbed is RoboArena: 10,783 real rollouts from ten generalist policies on the single-arm, parallel-gripper DROID platform, run by many labs, with human progress scores and A/B preferences. Critic comparisons sample 1,000 A/B sessions, or 2,000 rollouts. Cosmos Reason 2 (8B, NVIDIA Open Model License) and a frontier proprietary critic see the same three frames and rubric.

Judging real rollouts

The label-free log score picks the human-preferred rollout in 73% of sessions, against 56% for Cosmos Reason 2 and 70% for the frontier critic. Fusing the log with Cosmos Reason 2 lowers progress-scoring error from 1.25 to 0.74, a 41% reduction. Under the model’s recommended reasoning mode with seven frames, Cosmos reaches 52% A/B agreement against 72% for the log on the same sessions; fusion still lowers error by 37%.

A trivial baseline provides context: one bit, “did the gripper close at all,” already reaches 64%. Much of the gap is what three frames cannot show; the full formula adds nine percentage points over that bit. The log improves Cosmos Reason by supplying evidence the images may miss.

Table 2. Judges on the same 2,000 rollouts
Judge	Input	Progress error	A/B agreement
Cosmos Reason 2 (8B)	3 frames	1.25	56%
Log, one bit (closed at all)	Log	n/a	64%
Frontier VLM critic	3 frames	0.87	70%
Log, label-free Z	Log	n/a	73%
Log, learned	Log, task	0.75	76%
Log + Cosmos Reason 2	Both	0.74	75%

Error is mean absolute error against human progress on a 1–5 scale. Always predicting the median gives 1.13. The seven-frame reasoning-mode comparison uses a pre-registered subset of 100 sessions.

Three-panel comparison: human A/B agreement is 56% for Cosmos Reason 2, 70% for a frontier VLM and 73% for label-free log scoring; fusion reduces progress error to 0.74; seven-frame reasoning mode reaches 52% against the log’s 72%. — **Figure 3. Judging real RoboArena rollouts.** (a) Human A/B agreement on 1,000 sessions, with baselines in gray. (b) Progress-scoring error, with the median-prediction error marked by the dotted line. (c) Recommended reasoning mode with seven frames, on a pre-registered 100-session subset. Select the figure to inspect it at full size.

Ranking GR00T-class checkpoints without raters

Averaged per policy, the label-free score ranks the ten policies at ρ = 0.95 against human progress. Scaling every threshold from 0.5× to 2× keeps ρ between 0.92 and 0.98; removing the lowest policy gives 0.93. No single feature matches the formula: lift height alone gives 0.84 and “closed at all” gives 0.29 in the report’s prose.

Ten policies across 10,783 rollouts show Spearman 0.95 between mean log score and human progress; single-feature correlations range from 0.28 for gripper closed at all to 0.84 for lift height. — **Figure 4. Ranking without raters.** (a) Mean label-free score per policy against mean human progress on 10,783 rollouts. (b) Single log features compared with the full formula. The original figure labels “gripper closed at all” as 0.28; the report’s prose gives 0.29. Both are retained here. Select the figure to inspect it at full size.

Faster A/B decisions

Across all 28 policy pairs, the anytime-valid test on the label-free score made no wrong calls against human verdicts. It decided half of the human-decided pairs at a median of 44 sessions and abstained on close pairs. For checkpoint comparisons on real robots, this supports stopping once sufficient evidence is available.

Policy-pair scatter plot showing no wrong calls among 28 pairs, with large differences often decided in about 40 sessions and close pairs taking longer or remaining undecided. — **Figure 5. Anytime-valid A/B tests.** Each point is a policy pair. Large differences are decided in about 40 sessions; close pairs take longer or remain undecided. No decision disagreed with the humans. Undecided pairs are shown at zero on the vertical axis. Select the figure to inspect it at full size.

Contact-rich assembly and a critic veto

On 758 labelled actions from public REASSEMBLE contact-rich assembly recordings, a log-only verifier reaches held-out AUROC 0.86 overall and 0.95 on insertions. An exploratory analysis lets the log’s completion check veto any “success” call from 23 saved vision critics on 240 real actions, including 80 failures. Critics starting below 0.70 balanced accuracy rise from as low as 0.52 to about 0.79; false-pass rates fall from 16–85% to 7.5–30%. This offers a low-cost check against accepting failures when a Cosmos Reason-style critic filters trajectories.

Before-and-after lines for 23 vision critics show how a log veto changes balanced accuracy and the share of 80 real failures passed as successes. — **Figure 6. The log as a veto, exploratory.** Each line is one critic on 240 real assembly actions. (a) Balanced accuracy before and after the veto; blue lines started below 0.70. (b) The share of real failures each critic passed as successes. Select the figure to inspect it at full size.

Auditing GR00T teleoperation data

On NVIDIA’s public GR00T humanoid teleoperation dataset, the physics check flags about 11% of demonstrations. In blind review, 62–76% of flagged demonstrations were failed or flawed, versus 5–20% of unflagged or random ones. The finding includes a task the check was not designed on. On the robomimic multi-operator benchmark, a log quality score separates the best operators’ demonstrations from the worst at AUROC 0.94, and 0.92 on a second task, without operator labels.

Blind GR00T demo audits find 76% versus 20%, 62% versus 7%, and 71% versus 5% failed or flawed for flagged versus comparison demos; robomimic quality scores separate hidden operator ratings at AUROC 0.94 and 0.92. — **Figure 7. Auditing data.** (a) Failed or flawed GR00T humanoid demonstrations in three blind audits, comparing flagged with unflagged or random demonstrations. (b) Log quality scores of robomimic demonstrations grouped by the hidden operator rating; black bars mark medians. Select the figure to inspect it at full size.

Diagnosis and progress

On fresh sessions, a refined missed-grasp rule agrees with blind-coded human feedback at κ = 0.60, up from 0.56, and catches 55% of the missed grasps humans described, up from 32%. Partial-log scores track the final human score better as more of the rollout is seen: ρ rises from 0.35 at 25% to 0.55 at 100%. On DROID, success prediction reaches AUROC 0.81 on labs held out from training. A replication without task instructions was inconclusive: 0.64 against 0.58 for Cosmos Reason 2.

Refined failure rules increase human-feedback agreement from 0.56 to 0.60 and missed-grasp capture from 0.32 to 0.55; partial-log progress correlation rises from 0.35 to 0.47, 0.53 and 0.55. — **Figure 8. Diagnosis and progress.** (a) The first and refined failure rules against blind-coded human feedback on fresh sessions. (b) Correlation with the final human score at 25%, 50%, 75% and 100% of the rollout. Select the figure to inspect it at full size.

Auditing public robot-learning code

The same habit of checking physical and numerical facts was applied to the public Isaac GR00T repository and DexMimicGen. Each finding reproduces on unchanged public code, with a minimal witness and an opt-in fix that changes no default. These findings make no claim about internal training runs or released checkpoint quality.

GR00T: the loss depends on batch layout

Mixed-embodiment GR00T batches hold different numbers of valid action values in one padded tensor: 112 for LIBERO versus 680 for DROID in a 40 × 132 extent, from the public processors. The action head divides each microbatch’s masked error by its own valid count. The Trainer then averages microbatches and distributed data parallel (DDP) averages ranks. An average of ratios differs from a ratio of totals, so batch composition becomes part of the objective.

With four fixed examples and one optimizer step through the unchanged Trainer, a shared parameter moves to −0.0576 as one batch and +0.0500 as two batches. An opt-in adapter makes the objective explicit, per value or per example, and invariant to layout across 36 Trainer cases and 24 two-process DDP cases. Update error is at most 1.3 × 10⁻⁸ in FP32. The witness is constructed to expose the effect, rather than measure it in released models.

The same four examples move a shared parameter to minus 0.0576 in one stock GR00T batch or plus 0.0500 in two batches; either explicit objective stays consistent across layouts. — **Figure 9. Same examples, opposite update.** Red shows the stock objective. Blue shows that each explicit objective produces the same update across layouts, while the two objectives retain their distinct meanings. Select the figure to inspect it at full size.

GR00T: stale normalization statistics

After a real DROID episode is appended and metadata updated without a schema change, GR00T reuses its statistics cache. A clean recompute changes 24 statistic arrays. A cache bound to source data regenerates correctly and matches all 4,044 clean values. Cold computation is 47.6 times faster than the stock code, with all 36,396 values identical. A fully vectorized control is just as fast: the contribution is correctness, with the speedup explained by ordinary vectorization.

Cold statistics on 100 DROID episodes take 60.26 seconds in stock GR00T, 1.32 seconds for a vectorized control and 1.26 seconds for a source-bound cache; all 36,396 values are identical. — **Figure 10. Cold GR00T statistics.** The comparison uses 100 real DROID episodes. A vectorized control explains the speedup; binding the cache to its source data addresses stale results. Select the figure to inspect it at full size.

DexMimicGen: labels that fire 50 frames early

In DexMimicGen’s two-arm three-piece assembly environment, the check for whether the robot still holds the piece loops over the gripper dictionary’s keys, or names, instead of the grippers. It therefore never detects contact: all 70,225 single-contact geometry pairs evaluate false. The stored stage-completion label turns on at frame 104, while the witnessed release completes at frame 154.

A two-line patch using .values() matches an independent formula in all 229 recorded states. Final task success is unchanged. One demonstration was audited.

A DexMimicGen timeline shows a stored label starting at frame 104, 50 frames before witnessed release completes at frame 154; the patched contact check aligns with release. — **Figure 11. DexMimicGen subtask label timeline.** One demonstration, before and after the contact-check fix. Labels, like critics, must check contact rather than geometry alone. Select the figure to inspect it at full size.

The GB300 connector handover

NVIDIA’s October 7 factory report on GB300 cable-connector assembly reports 90–95% end-to-end success and about 40 seconds per cable, against a requirement of 99.5% and 72 seconds for all four. Remaining failures center on coordinating two arms during the connector grasp; stop-and-adjust repositioning dominates cycle time. The hypothesis here is that much of the gap is a verification problem: the controller needs to know in real time whether the connector is actually held.

Reported GB300 cell success is 90–95% against 99.5% required; approximately 160 seconds for four cables is compared with a 72-second requirement. — **Figure 12. Reported GB300 performance against requirements.** Four-cable time assumes about 40 seconds per cable, giving approximately 160 seconds. Select the figure to inspect it at full size.

A software-only, fail-closed coordination layer learns online how the dangling connector moves with the holding arm and resets when that behavior changes. A joint planner proposes bounded moves for both arms. It closes the gripper only if the whole uncertainty envelope stays inside the grasp region for the entire closing interval.

Every proposal must pass an external validator bound to the exact observation, model version and proposal hash. A release gate requires measured holding from current telemetry: contact on both sides, load transferred, no slip and at least 0.3 seconds of retention. A commanded closure never counts as holding.

Software-only evidence: the layer passes 60 software tests and has not run on the GB300 cell or in a physics simulator. The fastest first step is a log-based breakdown of where real exposure-to-grasp cycles fail and where time is spent.

A fail-closed sequence from observation to response model, joint planner, external validator and release gate; any missing, stale or mismatched check admits nothing. — **Figure 13. Fail-closed handover layer.** Each stage must pass before the next. Any missing, stale or mismatched check admits nothing. The release gate consumes the same measured-holding evidence extracted from logs in the real-robot experiments. Select the figure to inspect it at full size.

Six negative results

All six studies below were pre-registered and did not meet their pass bars.

01 / Distilling the log into Cosmos Reason 2

A LoRA fine-tune of Cosmos Reason 2 (8B) on 1,940 log-labelled RoboArena rollouts, with no human labels, answered the most common label for every test rollout. Its probabilities rank rollouts at 56%, equal to stock Cosmos Reason 2 and below the log’s 74% on the same test set. Episode-level labels from three frames were too thin a signal.

02 / Physics checks on world-model rollouts

Eighty real grasps, 40 missed and 40 lifted, were replayed in an open DROID action-conditioned world model using the true, a fixed 0.75 and the raw gripper signal. The judge read imagined outcomes at only 0.61 balanced accuracy, against 0.75 on real video, failing the sanity gate. The gripper signal changed how the fingers were drawn without changing the rest of the scene.

03 / Curation

Training on the 150 physically cleanest of 300 robomimic demonstrations gave 11% success against 25% for 150 random demonstrations, over three seeds × 50 rollouts. The ranking favored short demonstrations and removed variety; both arms were undertrained.

04 / GR00T N1.7 padding schedule

On the LIBERO checkpoint, the padding schedule changed action-head outputs by RMS ≥ 0.01 in one of three seeds. The pre-registered bar required two of three.

05 / Contact-aware demonstration replay

Placement error fell by 1.21 mm against filtered replay, with a 95% interval of −6.72 to 6.54 mm. Completions were 16/40 against 15/40. The primary comparison did not pass.

06 / Sharing one cable simulation

Sharing a cable simulation across rollouts with different arm commands was rejected: cable states diverge within about 67 milliseconds.

Six panels preserve negative results: collapsed distillation, failed world-model sanity gate, curation with lower success, padding changing one of three seeds, inconclusive replay improvement and cable states diverging in about 67 milliseconds. — **Figure 14. Negative results.** (a) Distilled Cosmos Reason 2 outputs versus teacher labels. (b) Replayed grasps judged held and lifted under each gripper signal. (c) Success per seed after curation. (d) GR00T N1.7 output change per seed against the bar. (e) Median placement error by replay method, descriptive; the primary comparison did not pass. (f) Time until cable state diverges from the idle case. Select the figure to inspect it at full size.

Real and imagined wrist-camera frames of a missed grasp while stacking cubes onto a red cylinder, comparing real video with world-model replays under true and fixed 0.75 gripper signals. — **Figure 15. Real and imagined wrist views of a missed grasp.** The task is “stack all the cubes onto the red cylinder.” Top: real camera. Middle and bottom: world-model replays of the same arm motion with the true signal and a fixed 0.75 signal. Imagined frames follow the arm but blur the outcome, preventing the judge from reading it reliably. Select the figure to inspect it at full size.

The lesson

The log’s value is contact at each moment: when the fingers met something, whether it stayed and where it slipped. Each failure compressed this into one number per episode or asked another model to reproduce it. The next proposed step labels every frame from the log and trains a video contact detector, so the same physics can check Cosmos-generated rollouts, synthetic GR00T training data and human video where no log exists. The first kill test is per-frame AUROC of 0.90 on held-out labs.

Limitations

One headline evaluation platform. RoboArena uses a parallel gripper, ten policies and mostly grasping tasks. Cosmos Reason 2 ran locally on three still frames, rather than NVIDIA’s video pipeline; the separately reported reasoning-mode check used seven frames.
Per-session agreement remains below human consistency. Agreement is 73–76%. The method replaces raters but still requires robot time. It is most useful for unattended evaluation and large log archives.
Timestamped readers and matched-trial pairing remain missing. The log tool’s first version lacks both, which causal two-arm use requires.
Public-code witnesses have narrow scope. They use constructed or single-demonstration cases. No new GR00T policy improvement has been shown; further validation requires additional real-robot evaluation logs.

Next steps and sources

Robots already record whether they hold something. Used as a verifier in these experiments, that record outperforms and improves Cosmos Reason 2 at judging real rollouts, ranks policies without raters, catches flawed public GR00T training data and exposes silent errors in public training and data-generation code. The next tests use additional GR00T evaluation and connector-cell logs, alongside a per-frame contact detector that carries the physical signal to generated footage.

Further validation

Four next studies can test the findings on additional real-robot logs, using state and action only and fixing each pass bar in advance:

Cosmos: add the log as a second channel to Cosmos Reason on real or synthetic rollouts with known outcomes.
GR00T / evaluation: rank GR00T checkpoints from their real-robot logs and compare with independent verdicts.
Isaac / training code: review the opt-in loss objective, the source-bound statistics cache and the DexMimicGen patch.
Manufacturing: break down failure stages and time losses in GB300 connector-cell logs.

Data, models and evidence

Code, all 23 pre-registrations and evidence packages are available on request from CosmicBrain AI. Data and models used: RoboArena (MIT), DROID (CC BY 4.0), robomimic (MIT), REASSEMBLE, NVIDIA GR00T teleoperation data, Cosmos Reason 2 under the NVIDIA Open Model License, and the public Isaac GR00T and DexMimicGen repositories under their respective licenses.