Abstract
Deciding whether a robot completed a task underlies training-data curation, policy evaluation, reinforcement-learning reward and the decision to hand control to a remote operator. Automated scoring services now offer such verdicts for teleoperated demonstrations, but their reliability is rarely measured. We audit one commercial service on 323 teleoperated bimanual episodes (9.4 hours) from a public humanoid teleoperation sample.
Headline scores are high (mean 8.88 of 10; 84.2% success) yet fail several reliability checks: scores are derived entirely from a single camera view with zero weight on robot telemetry; human corrections only raise scores; 70% of episodes are flagged for reviewer disagreement; about 1% of episodes receive a clean quality grade despite 90% scoring 8 or higher; and score is uncorrelated with episode duration (Pearson r = 0.07). Operator-level scoring could not run because the data carried no operator identifiers. We conclude that video-only scoring is suitable for triage but not for filtering training data.
We then propose a telemetry-first verifier ensemble with two-way correction, disagreement routing to calibrated humans and an auditable evidence ledger that serves data quality control, policy evaluation and runtime operations from a single stack, together with a validation protocol that accepts a scoring method only if it predicts downstream policy performance.
1 / Introduction
Physical-AI systems are increasingly trained on teleoperated demonstrations and deployed with a remote human in the loop. Every stage of this pipeline relies on a verifier: a procedure that decides whether an attempted task succeeded and how well it was executed. Verifiers select training data, compare policies, supply reward for reinforcement learning, and decide at run time when a robot is stuck and an operator must intervene. A recent survey of roughly 150 verifiers for robot policies argues that verification faces an inherent trade-off: credibility falls as availability rises [1]. Human experts are credible but slow and costly; automated checks are cheap and constant but less trustworthy.
Commercial services have begun to sell automated demonstration scoring to robotics teams [8], [10]. If such scores are used to filter training data or to certify datasets for buyers, their failure modes propagate directly into trained policies and into commercial claims. Yet independent audits of these scores are scarce. This paper makes three contributions:
- An audit of a commercial demonstration-scoring service on 323 public teleoperation episodes, quantifying score distribution, label consistency, reviewer agreement and sensitivity to camera choice (Sections 3 and 4).
- A design for a telemetry-first verifier stack and evidence ledger that serves data curation, policy evaluation and runtime operations from a single set of verdicts (Section 5).
- A validation protocol that accepts a scoring method only if it predicts downstream policy performance, agrees with expert humans without directional bias, and is invariant to camera choice (Section 6).
3 / Audit setup
3.1 / Data
We use a publicly released, access-gated teleoperation sample of about ten hours [9], recorded on a wheeled bimanual humanoid and stored in the LeRobot v3.0 format [4] at 30 fps. Each frame carries a 52-dimensional action, a 290-dimensional state vector, a flag indicating the action source, and five camera streams. Tasks include RAM-module insertion, refilling and extinguishing alcohol lamps, packing medicine into first-aid kits, packing and taping boxes, wiping lockers and handling documents.
3.2 / Scoring system under audit
The episodes were indexed by a commercial demonstration-scoring service, hereafter Service A [8], in a workspace we control. The service assigns each episode a quality score out of 10 (a weighted blend of facet scores such as arm visibility, task engagement, object interaction and task completion), an analysis outcome label, a review quality grade (clean, mixed, rough or needs review), and free-text evidence with reviewer dissent.
3.3 / Protocol
On 2 October 2026 we read all 323 rows of the episode table (score, outcome label, issue tag, category, duration and task title), the three report views, and the full evidence pages of two episodes selected for contradictory score and label. Task titles were mapped to seven task families by keyword rules (first match wins; families can overlap semantically). Table 1 summarizes the corpus.
| Property | Value |
|---|---|
| Episodes / duration | 323 episodes / 9.4 h; mean 105 s (range 20 to 291 s) |
| Categories (service) | Bimanual manipulation 188, insertion 75, pick and place 43, other 11, pouring 5, folding 1 |
| Distinct task titles | 189 titles describing roughly 10 underlying tasks |
| Operator identifiers | None |
| Review coverage | 100% of episodes carry a quality grade |
4 / Results
4.1 / Score and label distributions
Scores are concentrated near the top of the scale (Figure 1, Table 2): 90% of episodes score 8 or higher and 51% score 9 or higher. The outcome distribution reported by the service is 272 success, 39 partial and 12 needs review, an 84.2% success rate (Figure 2). The episode table, however, labels the middle group inconclusive (38 episodes) while the report labels it partial and lists zero inconclusive episodes, indicating vocabulary drift between views.
| Score band | Episodes | Share |
|---|---|---|
| 9.5 to 10.0 | 70 | 22% |
| 9.0 to 9.5 | 96 | 30% |
| 8.0 to 9.0 | 126 | 39% |
| 7.0 to 8.0 | 20 | 6% |
| 6.0 to 7.0 | 6 | 2% |
| Below 6.0 | 5 | 2% |
4.2 / Variation across tasks
Contact-rich and fine-placement tasks score lowest: locker and cloth tasks average 7.41 and medicine-to-first-aid-kit tasks 8.34, while short RAM insertions average 9.30 (Figure 3, Table 3). Insertion episodes are short (31 s on average) and end in a visually distinctive state, which plausibly favors a video-based judge; contact-rich tasks are where a single camera is least informative.
| Task family | n | Mean | Median | Min | Max | Mean length |
|---|---|---|---|---|---|---|
| RAM insertion | 86 | 9.30 | 9.4 | 7.4 | 10.0 | 31 s |
| Alcohol lamp | 26 | 9.12 | 9.4 | 3.7 | 9.9 | 232 s |
| Documents / drawer | 37 | 9.06 | 9.4 | 7.0 | 9.8 | 121 s |
| Other | 17 | 9.05 | 9.1 | 7.0 | 9.9 | 126 s |
| Box packing | 74 | 8.95 | 8.95 | 8.5 | 10.0 | 158 s |
| First aid / medicine | 69 | 8.34 | 8.6 | 5.3 | 9.2 | 80 s |
| Locker / cloth | 14 | 7.41 | 7.6 | 5.2 | 8.8 | 106 s |
4.3 / Reliability of verdicts
The service reports reviewer agreement per category (Figure 4). Outcome agreement ranges from 67% (insertion) to 86% (bimanual), and quality-grade agreement from 59% to 70%. In total 227 of 323 episodes (70%) are flagged for panel disagreement, reaching 81% for insertion. Quality grades are almost uniformly mixed: the clean share is 1% for bimanual manipulation (2 of 188), 0% for insertion and 2% for pick and place, which is difficult to reconcile with a score distribution in which 90% of episodes score at least 8.
Score is also unrelated to execution time. Across all episodes the Pearson correlation between duration and score is 0.07 (Figure 5); the visible clusters correspond to tasks, not to execution quality. A slow, hesitant demonstration and a fast, clean one can therefore receive the same score. Finally, score and label frequently disagree: 6 episodes labeled success score below 7, and 5 non-success episodes score 9 or higher.
4.4 / Case analysis
We inspected the evidence pages of the two most contradictory episodes (Figure 6, Table 4). Both state that the score is a provisional blend of 100% recorded view and 0% robot measurements, although the dataset includes full joint state and actions.
- Turn on the desk lamp (score 3.7, labeled success). The lamp is lit at the end and task completion scores 10. The total is pulled down by an arm-visibility facet of 0, because the scored camera is a wrist view that cannot show the robot’s arms. The low score reflects camera selection, not execution.
- Place the RAM stick into the yellow tray (score 10.0, labeled needs review). The written evidence states that the insertion attempt is clear but completion is not. Automated completion read 2.0 and was raised to 10 by video review; the service notes that review only examines low readings and can only raise them. Label agreement is 50%, and the task title does not match the observed behavior.
| Score | Label | Task | Observation |
|---|---|---|---|
| 3.7 | Success | Turn on the desk lamp | Completion 10; arm visibility 0 from wrist camera |
| 10.0 | Needs review | Place the RAM stick into the yellow tray | Completion raised 2.0 to 10; label agreement 50% |
| 5.2 | Needs review | Place the towel on the cabinet door | Lowest-scoring locker task |
| 5.3 | Needs review | Place the white medicine bottle into the first aid box | Small-object placement |
| 5.3 | Inconclusive | Place the clothes into the laundry basket | 20 s episode |
| 5.7 | Inconclusive | Place the white medicine bottles into the medical kit | Small-object placement |
4.5 / Summary of findings
| ID | Finding | Evidence | Consequence |
|---|---|---|---|
| F1 | Video-only scoring | 0% weight on robot measurements | Ignores recorded state and actions |
| F2 | Camera-dependent facets | 3.7 on a successful episode (wrist view) | Scores track camera choice |
| F3 | One-way correction | Review can only raise low readings | Upward bias; compressed top of scale |
| F4 | Low reviewer agreement | 70% flagged; 59 to 70% grade agreement | Verdicts are contested |
| F5 | Score and grade disagree | 90% score at least 8; about 1% clean | Unclear which signal to trust |
| F6 | Efficiency ignored | Duration vs score r = 0.07 | Slow and fast demos rank equally |
| F7 | Label vocabulary drift | Partial vs inconclusive across views | Brittle downstream automation |
| F8 | Task-title sprawl | 189 titles for about 10 tasks | Fragmented per-task statistics |
| F9 | No operator attribution | Every category not attributed | Operator scoring inactive |
Taken together, the scores are useful for triage: they surface, within minutes, the 20 to 60 episodes that merit human review, with readable evidence. They are not yet credible enough to filter training data or to certify datasets for third parties, and the decisive test of whether higher-scored episodes yield better policies has not been run.
5 / A telemetry-first verifier stack
We propose an evaluation layer in which a single ensemble of verifiers produces one verdict per episode, policy rollout or live task attempt, recorded with its confidence and evidence in a versioned ledger (Figure 7). The same verdicts drive data quality control, policy evaluation and runtime hand-off; takeovers and failures flow back into capture as labeled training data.
Read the architecture in Figure 7
- Capture: teleop recorder, MCAP + protobuf, all cameras, joint state, actions, F/T, tactile, operator + site IDs.
- Normalize: episodize, clock-align, task-name ontology, standard dataset export, provenance + rights tag.
- Verifier ensemble: telemetry rules (contact, force, pose, timing), multi-view VLM judge (all cameras), learned success detector, calibrated human panel, two-way correction, disagreement routed to a human.
- Evidence ledger: per-episode verdict + confidence + evidence spans, operator / task / site attribution, versioned and content-hashed, auditable; the insurer- and buyer-facing record.
- Data QC: keep / fix / drop per episode, operator scores + coaching, dataset readiness receipts for OEM data buyers.
- Policy evals: fixed simulation suites, fixed real-robot protocol, A/B on site tasks, reward for RL fine-tuning.
- Runtime ops: success / stuck detection, hand-off to remote operator, log takeover as labeled data, operator-minutes-per-task KPI.
5.1 / Design principles
- Telemetry first. Joint state, actions, gripper state, force/torque and tactile channels are cheaper and more objective than pixels; video is used where telemetry is silent (addresses F1).
- All cameras, camera-invariant facets. No facet may depend on which single camera was selected (F2).
- Two-way correction. Humans audit a random sample of high scores as well as low ones, and every correction is logged (F3).
- Disagreement as a routing signal. When verifiers disagree, the episode is routed to a calibrated human and the disagreement is stored (F4, F5).
- Efficiency as a facet. Time to completion relative to a per-task baseline enters execution quality (F6).
- Attribution and ontology at capture. Operator, site, robot, task (from a fixed ontology), outcome vocabulary and data-rights tags are written at recording time (F7 to F9).
5.2 / Verifier ensemble
Table 6 lists the verifier families. Their combination aims to move the operating point up and to the right of the credibility-availability frontier (Figure 8).
| Verifier | Signal | Detects | Credibility | Cost | Role |
|---|---|---|---|---|---|
| Telemetry rules | State, actions, gripper, F/T, timing | Drops, collisions, stalls, unsafe velocity | High (in scope) | Near zero | Hard gates |
| Multi-view VLM judge | All cameras | Semantic success, wrong object or placement | Medium | Low | Outcome and evidence text |
| Learned success detector | Video and telemetry | Task-specific success at scale | Medium-high | Low | Runtime hand-off; RL reward |
| Calibrated human panel | Replay and ledger | Ambiguous cases; ground truth | High | High | Disagreements; audits |
| Simulation replay | Recorded actions | Kinematic infeasibility; sync errors | Medium | Medium | Dataset certification |
5.3 / Metrics
| Metric | Definition | Consumer |
|---|---|---|
| Episode verdict | Success, partial or fail with calibrated confidence and evidence spans | All |
| Readiness | Share of hard gates passed (synchronization, completeness, telemetry present, rights) | Data QC; buyer receipts |
| Execution quality | Completion × efficiency (vs task baseline) × smoothness × safety events | Operator scoring; ranking |
| Operator score | Difficulty-adjusted execution quality per operator, with trend | Coaching; certification |
| Policy success rate | Verified successes over attempts on a fixed suite, with 95% CI | Policy evaluation |
| Operator minutes per task | Teleoperation and supervision minutes per verified completed task | Operations KPI |
| Verifier agreement | Cohen’s kappa between verifiers and against human audits | Health of the stack |
6 / Validation protocol
A scoring method should be trusted for training-data selection only if it passes the three tests in Table 8. The first is decisive: a score that does not predict policy performance has no value as a filter, however plausible its evidence text.
| Test | Procedure | Acceptance criterion |
|---|---|---|
| Predictive validity | Train the same policy (e.g., ACT or diffusion in LeRobot [4]) on equal-size top-ranked, bottom-ranked and random subsets; evaluate on a fixed simulated and real suite. | Top > random > bottom, significant at the 95% level |
| Human agreement | Expert panel labels a stratified random sample of 100 episodes, including high scorers. | Cohen’s kappa at least 0.6; no directional bias |
| Invariance | Re-score the same episodes from different single cameras and with telemetry withheld. | Stable verdicts across cameras; telemetry improves agreement |
Policy evaluation follows the same discipline: fixed task suites in simulation [7] and a written real-robot protocol per deployment site (task list, trial counts, reset procedure and success definitions), results reported with confidence intervals, and a shared policy-environment adapter in the style of XPolicyLab [3].
7 / Discussion and limitations
Implications
Verification is the layer through which data quality, policy quality and operational cost are all measured. For an operator of supervised robots, the central business metric, operator minutes per completed task, is only as trustworthy as the verifier that decides a task was completed. This argues for owning the verifier stack while using external services for triage and inspection (Table 9).
| Option | Strength | Gap | Use |
|---|---|---|---|
| Service A (audited) [8] | Operator, task and episode scoring; readable evidence | Video-only; one-way correction; no evals | Triage once data has operator IDs |
| Data-readiness service [10] | Readiness checks; annotation layers | Hosted only; early | Monitor |
| Commercial visualization platform | Mature visualization and fleet data | No verdicts or evals | Later, fleet operations |
| Open-source visualization tool | Inspection and queries | No verdicts | Inspection now |
| In-house stack | Telemetry verifiers, ledger, takeover data | Engineering effort | Build |
Limitations
We audit one dataset and one service version through its web interface; quality-grade counts were available per category rather than per episode; task families are keyword-derived; the trade-off figure is qualitative; and we have not yet trained policies to test predictive validity. Results may not transfer to other robots, tasks or vendors.
Data and anonymization
The audited episodes come from an access-gated public dataset [9]. We report aggregate statistics, and short quoted excerpts of published evidence; we do not reproduce dataset imagery or vendor interface images, and we anonymize the scoring vendor and the dataset source. The purpose of this audit is to characterize a class of methods, video-only demonstration scoring, rather than to rank a particular product. Future audits should use first-party recordings collected with explicit consent and attribution.
8 / Conclusion
Automated, video-only demonstration scoring produced high and compressed scores that disagreed with its own quality grades, depended on camera choice, were corrected only upward, and were contested by its own reviewers. Such scores help locate episodes worth inspecting but should not yet decide what enters a training set. A telemetry-first ensemble with two-way correction, disagreement routing and an auditable ledger, validated against downstream policy performance, offers a path to verdicts that are both available and credible.
References
[1] Y. Wan, X. Yue, Z. Liu, et al. No free checker: A survey of verifiers for robot policies. arXiv:2609.09250, 2026.
[2] RoboArena: Distributed real-world evaluation of generalist robot policies. arXiv:2506.18123, 2025. robo-arena.github.io.
[3] XPolicyLab: A unified standard and open ecosystem for robot policy evaluation and deployment. arXiv:2608.09892, 2026.
[4] LeRobot: An open-source library for end-to-end robot learning. arXiv:2602.22818, 2026.
[5] D. Niu, Z. Liu, Z. Wang, et al. T-Rex: Tactile-reactive dexterous manipulation. tactile-rex.github.io, 2026.
[6] DUET: Dual-robot understanding via efficient teaching. zhaoy37.github.io/Duet, 2026.
[7] B. Liu, Y. Zhu, C. Gao, et al. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. NeurIPS Datasets and Benchmarks, 2023.
[8] Service A. Commercial demonstration-quality scoring service (vendor anonymized); workspace accessed 2 October 2026.
[9] Public humanoid teleoperation sample, about 10 hours, access-gated release (source anonymized), 2026.
[10] Commercial data-readiness and annotation service (vendor anonymized), 2026.
A / Implementation plan
Figure 9 and Table 10 give a 16-week plan for the components in Section 5.
| Weeks | Deliverable | Exit criterion |
|---|---|---|
| 0 to 2 | Licensing review for third-party data; written questions to scoring vendors | Policy documented; vendor answers received |
| 1 to 7 | Recorder: MCAP, telemetry, operator, site and task IDs | 50 labeled episodes export to LeRobot and replay in simulation |
| 4 to 10 | Verifier v0: telemetry rules and all-camera VLM judge | Agreement with human audit measured |
| 6 to 10 | Predictive-validity experiment (Table 8) | Top, bottom and random results with CIs |
| 9 to 14 | Evidence ledger and operator scoring | Per-operator trends on first-party sessions |
| 12 to 18 | Runtime success and stuck detector | Used for hand-off on the first pilot |
B / Capture schema
Episodes are recorded as MCAP with protobuf messages and exported to LeRobot v3. Each episode contains all camera streams; measured joint state; commanded actions after safety clamps; raw operator XR input; gripper, force/torque and tactile channels where present; safety and watchdog events; takeover events (operator, time, reason code); and metadata for operator, site, robot, task (from a fixed ontology), outcome and data rights. The per-frame action-source flag in [9] is a useful precedent for recording whether a human or a policy was in control.
C / Reproducibility
The per-episode table used for all figures (score, duration, category, outcome label, task family and issue flag for 323 episodes) and the scripts that generate every figure and table are available from CosmicBrain AI on request. Figures use a colorblind-validated categorical palette; green, amber and red are reserved for outcome states.