The physical data engine: building the data layer for generalist robots

One of the core infrastructures of physical AI will be a system that continuously discovers, captures, structures, transfers, validates and actively fills gaps in physical experience.

Why an engine rather than a warehouse

Static datasets age. Model capability changes, embodiments change, and which tasks are weak changes with them.

An engine knows what it holds, what it lacks, what should be collected next, which material justifies expensive enrichment and which is already redundant. It is a closed loop, not an object store.

Six layers

Experience sources: internet, egocentric, stereo, multi-view, depth, robot. Physical parsing: hand, object, geometry, contact, action. State graph: transitions and coverage.

Enrichment: reconstruction, mocap, object pose, high-fidelity contact. Embodiment transfer: retargeting and morphology adaptation. Validation and feedback: simulation, benchmark and model outcomes feeding back into collection.

The active loop

Training exposes failure clusters — transparent objects, deformables, confined spaces, bimanual coordination, tool slip. The engine maps those failures onto a state taxonomy, searches the existing corpus, and where coverage is insufficient triggers discovery or targeted collection. New data is enriched and re-enters training.

This is also where the world-model turn has an operational consequence. As learned world models increasingly serve as environments for reinforcement learning and post-training, the data layer stops feeding only the policy and starts feeding the environment the policy trains in. The corpus is no longer a static input; a gap in it becomes a gap in the simulated world.

Vertical integration is real, and it does not close the question

The largest robot programmes are building their own collection capacity. One humanoid developer has announced an internet-scale egocentric pre-training effort backed by a partnership with an owner of more than 100,000 residential units, alongside a reported zero-shot transfer result from purely egocentric human video.

It would be dishonest to treat that as anything other than what it is: the strongest players will own their highest-volume data supply. The question it does not answer is which parts remain worth sourcing. Three categories keep resisting integration — information that is physically unobservable from the collection setup being scaled, such as ground truth behind occlusion and real force; work that requires organising the physical world differently, such as strictly paired human-and-robot capture; and rights chains for material that was not collected first-hand.

A data business that competes on volume against a vertically integrated buyer will lose. One that supplies what volume cannot produce has a different conversation.

Three stages, in order

Near term, comprehensible products: egocentric capture, reconstruction, mocap, custom collection. The competition here is execution and yield.

Medium term, aligned corpora, embodiment-ready transformation and benchmarked data packs. The competition is alignment capability and the evidence base behind it.

Long term, standing infrastructure: a capability target or a failure cluster goes in, and the physical experience most worth training on next comes out. The real measure at that point is how much capability boundary each dollar buys.

The stages add rather than replace, and the order cannot be skipped. Without near-term yield credibility the aligned products do not get a hearing, and without the medium-term evidence the infrastructure argument is a slide.

Three checks for your own pipeline

None of these require a vendor. All three are computable from data most teams already hold.

1. When a model fails, does anything in your data system change automatically?

If the path from failure cluster to collection brief is a meeting rather than a pipeline, the loop is not closed and the corpus will keep drifting away from the tasks that matter.

2. What can you not collect yourself?

Every serious programme should have this list written down. Occlusion ground truth, measured force and strictly paired capture appear on most of them.

3. Is your data budget an experiment line or an infrastructure line?

The answer determines the contract shape, the planning horizon and whether anyone is accountable for the corpus as an asset.

The physical data engine turns human experience into the next unit of robot capability.

References

1. "World Model for Robot Learning: A Comprehensive Survey", arXiv 2605.00080, 2026 — on learned world models as environments for reinforcement learning and post-training.
2. Figure, "Project Go-Big: Internet-Scale Humanoid Pretraining and Direct Human-to-Robot Transfer", company announcement, figure.ai. Vendor-reported.
3. Crunchbase News, "VCs Pour Billions Into Physical AI", 18 August 2026 — $47.4B across 521 deals in the first half of 2026, against $12B in the second half of 2025.
4. NVIDIA GEAR Lab, "EgoScale", arXiv 2602.16710, 19 February 2026.