The negative space of retargeting.
Every pipeline that converts human video into robot trajectories runs a physics check and throws away what fails. The discarded half is the densest signal in the pipeline.
Human-to-robot retargeting has a step almost everyone implements and almost nobody publishes: after wrist mapping, inverse kinematics and contact-preserving optimisation, the candidate trajectory is replayed in simulation against a specific end effector. If the replay diverges, the trajectory is dropped.
Treated as a filter, this step has one output: a cleaner corpus. Treated as a measurement, it has two — the trajectories that survived, and a precise statement about where the embodiment gap actually sits for a given robot on a given task.
The asymmetry
A single egocentric clip, retargeted across five reference end effectors, produces five independent outcomes. In our corpus the outcomes are rarely uniform: a bimanual dough-shaping clip that passes for three anthropomorphic hands will reject for a parallel-jaw gripper on aperture and for suction on deformability. A precision-insertion clip inverts that — the parallel gripper passes where a multi-finger hand rejects on clearance.
The outcome record therefore grows five times faster than the corpus itself, and the split between pass and reject varies by clip rather than being fixed. What matters is not that one side is larger. It is that “this clip is feasible” says one thing, while “this clip is feasible for these three hands and fails for these two, for these reasons, by these margins” says something a policy team can act on before committing compute.
| Clip | Shadow | Inspire | Allegro | Parallel-2F | Suction |
|---|---|---|---|---|---|
| Dough shapingdeformable, palmar | aperture | deformable | |||
| Connector seatingrigid, tool-mediated | geometry | ||||
| Tweezer insertionsub-mm clearance | clearance | scale | |||
| Filter replacementlong-horizon, mixed | |||||
| Shelf facingrigid, repetitive | |||||
| Laundry foldingdeformable, bimanual | grasp | deformable |
Rows are representative of their scene family, not averages. No clip in this set fails for every effector, and none passes for all but one — which is the point: the negative space is task-shaped, not effector-shaped.
Where this connects to cross-embodiment work
The standardisation effort around Open-X-Embodiment has made one thing routine: pooling demonstrations across robot bodies and training a policy that transfers between them. What that programme does not have, and cannot derive from the pooled positives alone, is a statement about which source demonstrations were never reachable for which target body.
A feasibility record is exactly that statement, and it has a second use beyond corpus selection. The outcomes are labelled training data for a feasibility predictor — given a human clip and a target URDF, estimate whether the retarget will survive replay, without running it. That model is cheap to train on a record of this shape and impossible to train without one. We have not built it; we are keeping the data that would let someone.
A closed vocabulary matters
Rejection reasons are only useful if they are comparable across clips and across embodiments. We use six categories and refuse to add a seventh without a structural argument:
Free-text failure notes are unusable at scale. A closed vocabulary makes the record queryable: show me every clip that fails for suction on deformability is a one-line filter, and the answer is a training-set decision.
Why this is hard to reproduce
None of the underlying methods are proprietary. Wrist mapping, IK and contact-preserving optimisation are published; the simulators are open. We make no claim on the algorithms and publish which ones we use.
What is not reproducible from public sources is the accumulated outcome. It exists only where the replays were run and the results were written down rather than dropped. A team that wants the same record has to rerun physics validation across every clip and every target embodiment — which is to say, they have to do the work, not download it.
What we think this implies for the category
Egocentric hours have stopped being scarce. A permissively licensed release now carries roughly as many hours as a large commercial corpus, and monthly collection capacity across the category runs into the hundreds of thousands of hours. The scarce quantities have moved: structured contact state, metric-scaled geometry, provenance that survives legal review — and the negative record, which is the only one of the four that gets more valuable the more of it you have and that no amount of additional collection produces on its own.
We publish the record alongside every retargeting engagement, and the rejection rate on every data card. Not because it flatters the corpus — it does not — but because a supplier that shows only what passed is asking to be taken on trust.
Questions this raises
A structured record of which retargeted trajectories a given robot end effector could not execute, and why. It is produced as a by-product of physics replay — the step most data pipelines use as a filter and then discard.
A policy trained only on feasible trajectories has no signal about where the feasible region ends. The negative record tells a team which human manipulations their embodiment cannot reproduce, before a training run discovers it.
The retargeting methods are public. The accumulated outcome record is not — it exists only where the replays were actually run and the results kept. Reproducing it means rerunning physics validation across every clip and every end effector.