Cosmic 0.5: A Universal Stack for Human-to-Humanoid Skill Transfer

The data bottleneck

A humanoid that can walk, balance and hold a lemon without crushing it is now a purchasable object. What is not purchasable is the thing that tells it when to close its fingers, how far to reach for the bowl, and what to do when the bag tips over. That knowledge lives in demonstration data, and demonstration data is the scarcest input in the industry.

The conventional answer is teleoperation: pay an operator to drive a robot through a task, thousands of times. It works, and it is the only method that produces perfectly on-distribution robot data. It is also linear in cost — every additional hour of data costs an additional hour of a human and an hour of a machine. Meanwhile, the same human performs the same task at home, on a phone camera, for free, several times a day.

Cosmic 0.5 is built on the position that these two sources should feed the same pipe. A VR teleoperation session, a monocular video of someone unpacking groceries, and an egocentric recording of two hands assembling a circuit board are, for the purposes of a manipulation policy, three noisy observations of the same underlying quantity: a human body moving through a task. The engineering problem is not choosing between them. It is building a representation stable enough that all three can be reduced to it, and a retargeting layer honest enough that what comes out the other side is executable on a real machine without breaking it.

The industry does not have a robotics data problem. It has a translation problem — the data already exists, in the wrong body.

System overview

Cosmic 0.5 is organised as three planes, connected by one representation and one artifact format.

The capture plane ingests human motion from whichever modality is available and is responsible for producing per-frame estimates with calibrated confidence. The motion plane fuses those estimates into a canonical motion — an embodiment-independent description of what the body did — then refines, filters and retargets it into robot-specific joint trajectories. The execution plane takes those trajectories, proves them against a model and a set of gates, and only then puts them on hardware.

The separation matters more than any individual component. Because the capture plane terminates at a canonical representation rather than at joint angles, adding a new input device does not require touching the retargeter. Because the execution plane consumes a bundled artifact rather than a live stream, the same trajectory that was previewed in simulation is bit-for-bit the trajectory that reaches the actuators.

The capture plane

Four modalities feed the stack. They differ in cost, in fidelity, and in what they can and cannot observe — and the fusion layer downstream is built around exactly those differences.

Monocular video

The cheapest and least cooperative input: a single RGB camera, unknown intrinsics, unknown scale, arbitrary viewpoint, frequent self-occlusion. It is also the only modality with effectively unlimited supply. A clip enters through the Studio, which runs pose estimation, produces a landmark overlay for inspection, and renders a simulated replay of the retargeted result beside the original — so a human reviewer can see, in one screen, whether the robot did what the person did.

Egocentric RGB-D with inertial reference

A head-mounted rig recording colour, depth and orientation simultaneously. Where monocular video must infer depth, this modality measures it: the viewer reports per-hand tracking confidence alongside metric wrist coordinates and a depth band for each hand — in the session below, the right hand at X 175mm Y 36mm Z 529mm with a working range of 529–674mm across both hands, at 99% and 95% confidence. The IMU panel carries the head pose that anchors those numbers to a frame.

This is the modality that captures the part of a task the body cannot show from outside: what the hands do at the moment of contact, from the viewpoint that actually matters. Sessions are annotated on a timeline, with an assisted pass that proposes natural-language descriptions of each segment — "the right hand reaches for the internal electronic component featuring a cooling fan and moves it towards the left case shell" — giving every trajectory a language label without a human writing one from scratch.

VR teleoperation

When a task needs to be performed rather than observed, an operator drives the robot directly through a WebXR headset. The teleoperation path is a different engineering problem from the offline ones: it is a control loop with a human inside it, and the dominant failure mode is latency, not accuracy.

Four decisions define it. Jitter is suppressed with adaptive One-Euro filtering rather than a moving average, so that smoothing does not impose a constant lag. End-to-end latency is measured and the operator's motion extrapolated forward to compensate for it. Arm poses are solved with warm-started differential inverse kinematics instead of a full nonlinear solve each frame. And XR input is decoupled from the control loop across threads, so that a slow headset frame cannot stall the robot. The stack targets the Unitree G1 with a choice of dexterous hands, in an upper-body mode with locked legs and a full-body mode with a learned balance policy.

Simulation

A library of more than fifty MuJoCo environments covering reaching, dual-arm coordination, block stacking, cutlery handling, dishwasher and cupboard interaction, and multi-stage grocery and food-preparation workflows. Simulation serves two roles here: it generates demonstrations where the real-world variant would be expensive or destructive, and it provides the deterministic replay surface on which every real trajectory is checked before it reaches hardware.

Canonical motion

The pivot of the whole system is a single decision: no capture modality is ever allowed to talk to a robot. Everything is reduced first to a canonical motion representation — SOMA — which describes the movement of a human body without reference to any particular machine.

The consequences are practical. A pose estimator can be swapped without touching retargeting: the provider is a named, recorded field in every run's metadata, so a clip processed six months ago carries the identity of the model that processed it. A new robot requires a new retarget profile, not a new pipeline. And a single recording can be re-emitted against a different body years later, because the archived artifact is the canonical motion, not the joint angles of whatever robot happened to be on the bench that week.

Every run is packaged as a bundle rather than a file. Alongside the canonical .npz, the bundle carries robot-ready joint trajectories as CSV, a BVH export for animation tooling, 2D and 3D keypoints from the pose stack, hand-detail diagnostics, per-frame fusion confidence, preview video, and a machine-readable run record naming every provider, parameter and timing in the run.

Fusion and filtering

Raw pose estimates are not usable as robot commands. They are jittery at the millimetre scale, they lose limbs behind occluders, and their error is not uniform — a wrist tracked at 99% confidence and a wrist inferred through a sleeve are the same data type carrying wildly different truth. The fusion and filtering chain exists to turn that into something an actuator can follow.

Weighting, not averaging

Fusion is confidence-weighted rather than uniform. A dedicated hand specialist [6] runs alongside the body pose stack and is folded into the canonical motion only where its own confidence clears threshold, with a short gap-fill window covering the frames where a hand passes briefly behind an object. Two properties keep the result stable: a minimum blend weight prevents any single source from being switched off entirely and snapping the trajectory, and fusion progress is held monotonic across modality switches, so that a hand-off between sources cannot cause the reconstruction to jump backwards.

Three filters, three jobs

Smoothing runs as a chain rather than a single pass, because the artefacts have different shapes. One-Euro [8] suppresses high-frequency jitter adaptively — heavier smoothing when the body is slow, lighter when it moves fast, which is precisely the trade-off a fixed low-pass filter gets wrong. A Savitzky–Golay pass preserves the peaks and inflections that a moving average would flatten, which matters because the moment of contact is a peak. An RTS smoother runs over the whole sequence with an explicit noise model, using the pose stack's own confidence as measurement noise, so low-confidence frames are pulled toward the dynamics estimate rather than believed.

Root and wrist are scaled independently of the body. The wrist is where the task happens, and the root is where drift accumulates; both are tuned separately from the torso for exactly that reason.

Refinement and occlusion

Filtering cannot recover what was never seen. When a person turns away from the camera, steps out of frame at the knees, or reaches behind their own torso, the limb data is not noisy — it is absent. Interpolating across the gap produces motion that is smooth and wrong: legs that drift, arms that take impossible shortcuts, a body that visibly lies about what it did.

Cosmic 0.5 handles this with a generative refinement stage running on the fused canonical motion, gated by a classifier that scores how much of the body is actually visible [5]. Above the full-body threshold, refinement is a polish pass. Below the partial-body threshold, occlusion infill is invoked and the hidden segments are generated to be continuous with the frames on either side. The classifier's job is to keep the generative stage away from data that does not need it — a system that always generates cannot be trusted to have observed.

The stage is steerable in plain language, which is unusual in a motion pipeline and is there for a specific reason: the failure modes are describable but not easily expressible as a loss. The standing instruction — "natural human motion consistent with the observed video, preserving body timing and hand motion; maintain standing up straight, in a comfortable relaxed and balanced manner" — encodes the priors that keep a reconstruction physically plausible for a bipedal machine. The occlusion instruction is narrower: "smoothly continue the visible motion through occluded segments, preserving timing, posture, and limb continuity from the surrounding visible frames."

Diffusion runs at fifty steps against a fixed seed. The fixed seed is a deliberate reproducibility choice: two runs of the same clip with the same parameters produce the same trajectory, which is a precondition for treating a bundle as an artifact rather than a sample.

Remark (scope of generated motion). Generated segments are not observations, and the report treats them differently. Occlusion infill produces motion that is plausible and continuous — it does not recover what the person actually did behind the occluder. Any downstream use that depends on ground truth should read the per-frame confidence artifact in the bundle, which marks these regions explicitly, rather than treating the trajectory as uniformly measured.

Retargeting

Retargeting is where an embodiment-independent motion becomes a specific machine's problem. A human shoulder has a range a G1 shoulder does not. A human finger closes on a lemon in a way a five-finger robot hand approximates rather than reproduces. A human shifts weight continuously; a bipedal robot with a fixed foot polygon does not get to.

Three constraints are enforced at this stage, and all three are enforced before anything is written to a bundle rather than checked afterwards. Joint limit clamping holds every retargeted joint inside the active profile's range. Velocity limiting caps the rate of change, which is what stops a fast human gesture from becoming an actuator command that would be rejected or, worse, followed. Feet stabilisation holds foot contacts steady against pose-estimation noise in the lower body — the single most common source of visibly wrong output, since a small error at the hip becomes a large error at the ankle.

The hand is handled as a separate profile from the body. This is the layer that makes the stack embodiment-agnostic in practice rather than in principle: one canonical motion, several hand profiles, several bodies.

Physics validation

A retargeted trajectory that respects joint and velocity limits is still not proof that a robot can execute it. Limits are necessary conditions, not sufficient ones — they say nothing about whether the resulting motion is dynamically feasible, whether the centre of mass stays over the support polygon, or whether two links pass through each other on the way.

The stack is explicit about this, and encodes the distinction as a ladder of replay modes [7] rather than a single "simulate" button. The default is kinematic replay with the base anchored: joints teleport exactly to the trajectory with no physics integration. This is a visual check — it answers "does this look like the motion we captured" and nothing more. Releasing the anchor lets the root follow the trajectory's own translation, which exposes drift the anchored view hides. Turning on physics integration is the mode that can actually falsify a trajectory, and it is treated as a debugging tool rather than a default because a physics failure at this stage is often a simulation-tuning artefact rather than a real one.

One further control deserves mention because it is a candid admission encoded in software: lower-body motion can be locked to the first frame. It is off by default, so that simulation shows the same waist and leg motion as the retargeted output. It exists for clips where the subject stands essentially still and pose-estimation noise causes the legs to drift — a case where showing the noise faithfully is less useful than suppressing it, and where the operator should be the one choosing.

Mission Control

Everything up to this point produces a file. Mission Control is the layer that decides whether that file is allowed to move two hundred kilograms of geared actuator, and it is the part of the stack where the engineering is least glamorous and most consequential.

Telemetry first

The console streams the robot's full 29-degree-of-freedom state — position, velocity and torque per joint, grouped by limb, over a rolling window. Torque is the channel that matters most: it is where an unexpected contact appears before it appears anywhere else, and it is the signal that distinguishes "the arm is moving" from "the arm is moving through something." A transport status line reports the live receive rate and message age, so a stale telemetry stream is visible as a number rather than inferred from a frozen viewport.

The gate chain

Playing a trajectory on hardware requires passing a sequence of independent gates, each of which fails closed. The bundle is loaded and its start pose is compared against where the robot actually is; a plan is generated to move from the current pose to the sequence start, because the most dangerous moment in a replay is the first frame, when a robot at rest is asked to become a robot mid-motion. A ghost preview renders the intended trajectory against the live model using the same execution path the real playback will take — not a separate visualiser that might disagree. Playback must be explicitly armed, and arming is bound to an environment variable on the machine, not a button in a browser, so that a console left open cannot be a loaded weapon. Collision avoidance and the control pipeline are separately selectable. A reflex and governor layer sits beneath all of it, and a hardware stop sits beside all of it.

Contact without sensors

Manipulation is a force problem wearing a position problem's clothes. Two trajectories can be identical in joint space and completely different in outcome depending on how hard the hand closed — and force is exactly what a position-only demonstration fails to record. Adding force-torque sensors to every joint solves it and costs more than the arm.

Cosmicbrain's work here follows FACTR 2 [3], and separates the problem into estimation and use. The estimation half learns the robot's own free-space inverse dynamics from a short contact-free motion — what torque this arm requires to move itself, absent the world — and then treats external contact as the residual between that prediction and the torque actually measured at the motors. The robot's existing motor current becomes a contact sensor. The second half puts that signal to work during training: the estimated torque is used to segment demonstrations into contact phases, and contact-relevant frames are up-sampled during behaviour cloning, on the reasoning that the few frames where the hand meets the object carry most of the information and are drowned out by the many frames of free-space transit.

The implementation is deliberately light — NumPy and PyTorch at the core, with dynamics and vision dependencies isolated behind adapters — and ships a synthetic demonstration that trains the estimator and recovers an injected contact on a CPU in minutes, with no robot and no downloaded weights. That property matters more than it sounds: a contact-estimation method that can only be evaluated on hardware cannot be iterated on quickly.

This is also why torque is a first-class channel in the telemetry view of Fig. 13 rather than a diagnostic afterthought. The same quantity that tells an operator something unexpected is happening is the quantity a policy learns contact from.

Policies and world models

Motion transfer produces trajectories. Trajectories become behaviour through a learning stack, and Cosmicbrain's is assembled from several lines of published research, implemented and integrated rather than invented here — a distinction this report keeps explicit.

Egocentric manipulation policies

An implementation of the Humanoid Policy ~ Human Policy line of work [1], which trains manipulation policies from an egocentric viewpoint without requiring wrist cameras — the practical significance being that a human demonstration and a robot demonstration can share an observation space. The training framework supports several algorithm families in parallel: action-chunking transformers with interchangeable visual backbones, diffusion policy, and a robotic diffusion transformer, with a MuJoCo replay harness for rolling out a policy before it touches hardware.

Dexterous data from everyday video

An implementation of Do As I Do [2], reconstructing hand–object interaction from ordinary monocular video and retargeting it onto multi-fingered robot hands. It orchestrates nine separate foundation models [4] — segmentation, 3D lifting, monocular geometry, hand reconstruction, point tracking and camera calibration — into a single path from RGB frames through a hand–object trajectory to a robot trajectory and finally to hardware commands, with a sampling-based optimiser handling retargeting. The repository is candid that it is not a turnkey system: mock backends let the pipeline be exercised without GPUs or weights, while real operation requires upstream models, a robot description file and asset-specific integration.

Direct hand control

A lighter path for cases where a person simply wants to drive a hand: real-time landmark tracking from a webcam mapped to a dual-arm platform with automatic inverse kinematics and openness-based gripper control, and to a twenty-four-joint dexterous hand with full finger and wrist mapping. Uploaded video can be processed offline into landmark data and an overlay render, which makes it a data-collection tool as well as a control one.

World model

The most exploratory component: a model that predicts the next egocentric frame conditioned on discrete hand actions, and can be driven interactively — step forward, step back, open or close either hand, and watch the predicted future change. It runs on-device today, and its predictions are visibly degraded compared to the observation it is conditioned on.

It is included in this report because of what it is for rather than what it currently produces. A policy that can imagine the consequence of closing its left hand before closing it is a policy that can plan; the egocentric capture stack described in §3 produces exactly the paired frame-and-action data such a model needs. The current model is an early result on a small number of sessions, and is presented as such.

Limits

A technical report that only describes what works is marketing. The following are the constraints an informed reader should hold Cosmic 0.5 against.

• The pipeline is offline, not real-time. A short clip taken through the full stack — pose, fusion, filtering, refinement, retargeting, simulation preview and packaging — takes tens of minutes of wall clock, dominated by the generative and rendering stages. The run in Fig. 18 processed an eight-second clip in just under twenty-five minutes. This is a batch data-generation system. The teleoperation path is the real-time path, and it is a separate one.
• No closed-loop benchmark is published here. This report contains no success-rate table, and the omission is deliberate. The stack has not yet been evaluated against a fixed, externally comparable task suite, and publishing internal numbers without one would be a claim the evidence does not support. The benchmark comes first; the numbers come with it.
• Kinematic preview is not a feasibility proof. As set out in §8, the default replay mode is a visual check. Only physics-integrated replay can falsify a trajectory, and it is not the default.
• Generated motion is not observed motion. Occlusion infill produces continuity, not truth. The bundle marks these regions in its confidence artifact; downstream consumers that ignore that marking will over-trust the trajectory.
• Several learning components are reimplementations. The egocentric policy framework, the everyday-video retargeting path and the force-estimation method follow published work by other research groups, credited in §15 and cited throughout. What is original here is the integration: one canonical representation, one artifact format and one gated execution path spanning all of them.
• Contact-rich manipulation remains the hard part. Reaching, transporting and placing rigid objects are the demonstrated regime. Deformables, in-hand reorientation, bimanual tasks with tight force coupling and any task where the critical contact is hidden from the camera are not solved by this stack today.
• The world model is early. Short horizons, small session counts, visible degradation. It is a research thread, not a component other parts of the system depend on.

Toward Cosmic 1.0

The version number is a statement about coverage rather than polish. Cosmic 0.5 is a complete path from human motion to gated hardware execution; what separates it from 1.0 is autonomy, evidence and speed.

A frozen benchmark (evidence)

A fixed closed-loop task suite, published with the protocol, so that results are comparable across versions and against other groups. This is the first blocker, not the last.

Streaming the offline stack (latency)

Collapsing fusion, filtering and refinement into a streaming path so that capture-to-trajectory runs closer to the clip's own duration than to tens of minutes.

Contact-aware policies on hardware (force)

Taking sensorless external-torque estimation from offline training into the live control loop, so that contact informs execution rather than only shaping the training set.

Profiles beyond one platform (bodies)

Retarget profiles for additional humanoids and hands, exercising the claim that the canonical representation is genuinely embodiment-independent.

Policy execution under the same gates (autonomy)

Moving from replaying a recorded bundle to executing a learned policy — through the identical arming, collision and governor chain, with no separate unguarded path.

Longer-horizon world models (prediction)

Extending action-conditioned prediction from single steps to horizons long enough to be useful for planning, trained on the growing egocentric corpus.

Commercial model

The stack supports two businesses, and they are the same asset sold twice. The first is dexterous data collection for humanoid manufacturers and foundation-model teams: the pipeline described in this report, delivered as curated demonstration data in the formats training stacks already consume. The second is humanoid labour delivered as a service to facilities: the same retargeting and execution stack, operated on a customer's floor and billed by robot-hour.

They compound. Data collection funds the operating hours; operating hours generate observations that improve the policies; better policies reduce the human supervision each robot-hour requires, which is the only path by which labour-as-a-service becomes a business with margin rather than a staffing agency with extra steps.

Acknowledgements

Cosmic 0.5 is an integration, and it stands on a considerable amount of work done elsewhere. Two components of the system shown in this report were built by people outside the core pipeline team, and are credited here in full.

Chronicles — the egocentric capture, hand-tracking, session-annotation and action-conditioned world-model tooling shown in Figs. 4 and 17 — is the work of Keval Shah. The operations console shown in Figs. 13 and 15, Mission Control, builds on open-source work by Patrick Deegan (MIT).

Three of the learning components described in §10 and §11 are independent implementations of methods published by other research groups: the egocentric policy framework follows [1], the everyday-video retargeting path follows [2], whose authors did not release code, and the sensorless force-estimation method follows [3]. What is claimed as original here is the integration — one canonical representation, one artifact format and one gated execution path spanning all of them.

References