One moment, six representations: the case for frame-level alignment

Alignment is the multiplier in multimodal physical data. It lets one hour serve perception, world modeling, policy learning, retargeting and simulation at once.

Multimodal is not aligned

Plenty of datasets carry RGB, depth, pose and language at once. Timestamp drift, inconsistent coordinate frames and unstable object identity still keep them from forming strict supervision.

Alignment has four components: temporal synchronization, spatial calibration, identity consistency and semantic phase alignment. Miss any one of them and cross-modal supervision collapses into parallel storage.

Fifty-four hours

EgoScale provides the clearest published evidence on what alignment is worth. Its structure is three stages: pre-train on 20,854 hours of action-labeled egocentric human video; mid-train on 54 hours of aligned play data — about 50 hours of human video and 4 hours of teleoperated robot demonstrations — covering the same 344 tabletop tasks with identical camera setups; then post-train on specific tasks. The resulting policy improved mean success rate by 54% over a no-pre-training baseline on a 22-DoF dexterous hand, and transferred to lower-DoF hands.

The ratio is the point. Fifty-four hours is about 0.26% of the pre-training corpus, and the robot-side portion is 4 hours — under 0.02%. The step that crosses the embodiment gap is roughly one four-hundredth of the data, and the part that needs a robot in the room is one five-thousandth.

Three consequences follow. Large-scale egocentric capture is being commoditized quickly. Paired data — human and robot, same task, same rig, strictly synchronized — cannot be crowdsourced, because it needs robot hardware, a teleoperation stack, task design and calibration in the same room at the same time. Four hours of it is not a small ask; it is a different kind of ask. And its value is not set by its own duration but by how many pre-training hours it activates.

Why the rest of the field keeps rediscovering this

The alignment problem shows up under several names. Work on human-to-robot data augmentation renders robot embodiments into human footage to close the visual gap, reporting real-world success gains between roughly 3% and 23% across gripper, dexterous and bimanual platforms. Work on active perception recovers synchronized camera and wrist trajectories because the misalignment between head motion and hand motion is what breaks egocentric pre-training. Video-alignment frameworks address viewpoint, action and visual alignment jointly before feeding a policy.

Different methods, same deficiency: the modalities exist, and they are not registered to one another. Whether the registration is done in rendering, in trajectory recovery, or at capture time, somebody has to do it. Doing it at capture time is cheaper and more accurate than doing it afterward.

What alignment buys in training

With strict alignment, cross-modal distillation becomes available: multi-view reconstruction teaches monocular depth, mocap teaches visual hand pose, instrumented contact teaches a visual contact predictor, and robot trajectories provide a backward check on whether a human representation preserved transferable constraints.

Expensive data becomes a teacher for cheap data at scale. That is the economics behind every layered data strategy worth running.

Turn the claim into measurements

Aligned is not an acceptance criterion. Five figures are: cross-sensor synchronization error at p50, p95 and p99 in milliseconds; calibration reprojection error in pixels with a validity window; object identity persistence across occlusion and across segments; inter-annotator agreement on task phase labels; and per-layer missing-data rate and confidence distribution.

All five can be written into a contract and independently checked. They are also the fastest way to find out whether a corpus described as multimodal actually is.

MOVAS-Ego 100K contains no aligned human–robot capture today; where an engagement needs it, it is scoped as custom collection.

Three checks for your own pipeline

None of these require a vendor. All three are computable from data most teams already hold. 1. Can you name the five alignment numbers for your own corpus? Synchronization error distribution, reprojection error, identity persistence, phase-label agreement, per- layer missing rate. Most teams can produce none of them on request. 2. Do you hold any strictly paired human-and-robot data? Not human data and robot data in the same catalog — the same task, same rig, same timeline. If the answer is no, the embodiment gap is being crossed by inference rather than by evidence. 3. Which layer is teaching which? A multimodal dataset with no declared teacher-target relationships is usually a storage decision rather than a training one.

1. Can you name the five alignment numbers for your own corpus?

Synchronization error distribution, reprojection error, identity persistence, phase-label agreement, per-layer missing rate. Most teams can produce none of them on request.

2. Do you hold any strictly paired human-and-robot data?

Not human data and robot data in the same catalog — the same task, same rig, same timeline. If the answer is no, the embodiment gap is being crossed by inference rather than by evidence.

3. Which layer is teaching which?

A multimodal dataset with no declared teacher-target relationships is usually a storage decision rather than a training one.

Six modalities are useful. Six aligned views of the same moment are transformative.

References

1. NVIDIA GEAR Lab, "EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data", arXiv 2602.16710, February 2026; project page, research.nvidia.com/labs/gear/egoscale. The 54% figure is relative to a no-pre-training baseline, not an absolute success rate.
2. "H2R: A Human-to-Robot Data Augmentation for Robot Pre-training from Videos", arXiv 2505.11920 — 3.3% to 23.3% real-world success gains across platforms.
3. "ActiveMimic: Egocentric Video Pretraining with Active Perception", arXiv 2606.06194, June 2026.
4. "Robot Learning from Human Videos: A Survey", arXiv 2604.27621, 2026 — on viewpoint, action and visual alignment frameworks.