MEVB
Movas Embodied Video Benchmark — a shared yardstick for the embodied AI community. Twelve core manipulation tasks, an open evaluation harness, sealed test splits and a public leaderboard.
Three Layers
Open Core
12 core manipulation tasks with ground-truth success metrics. Evaluation harness and public leaderboard released under a permissive licence.
Reproducible Evaluation
Standardised protocols, sealed test splits and multi-embodiment support — engineered for fair, reproducible cross-team comparison.
Enterprise Extensions
Optional commercial tier for teams needing extended task suites, private holdout sets, quarterly refreshes and SLA-backed evaluation infrastructure.
Task Suite
Difficulty rises from single-hand pick-and-place through bimanual coordination and tool use to long-horizon deformable-object tasks. The final two tasks measure generalisation rather than raw capability.
| ID | Task | Description | Tier |
|---|---|---|---|
| T01 | Cup-Pickup-Place | Pick a mug from the table and place it inside a marked zone. | Tier 1 |
| T02 | Drawer-Open-Close | Open a drawer fully, then close it. | Tier 1 |
| T03 | Door-Operation | Rotate a door handle and push the door open. | Tier 1 |
| T04 | Tray-Bimanual-Transport | Lift a loaded tray with both hands and move it between surfaces. | Tier 2 |
| T05 | Cap-Twist-Off | Stabilise a bottle with one hand and unscrew the cap with the other. | Tier 2 |
| T06 | Cloth-Fold-Bimanual | Fold a towel twice using both hands. | Tier 2 |
| T07 | Sponge-Wipe | Wipe a marked area of the table with a sponge. | Tier 3 |
| T08 | Spoon-Scoop-Transfer | Transfer granular material between bowls using a spoon. | Tier 3 |
| T09 | Single-Dish-Wash | Pick up a dish, rinse, sponge, and place on the drying rack. | Tier 4 |
| T10 | Shirt-Fold | Flatten a shirt, fold both sleeves inward, then fold from the hem. | Tier 4 |
| T11 | Generalization-Object | T01 performed with an unseen mug shape, colour and size. | Gen. |
| T12 | Generalization-Scene | T01 performed under changed lighting, tablecloth and background. | Gen. |
Evaluation Protocol
Success is milestone-based and binary per milestone. A task counts as successful only when every milestone is met; partial completion is recorded separately for diagnostic use.
TaskScore = 0.50 x SuccessRate
+ 0.30 x GeneralizationScore (L1 object + L2 scene + L3 instruction)
+ 0.20 x TimeEfficiency (1 - actual_time / budget_time)
OverallScore = weighted mean over 12 tasks
Tier 1 x1.0 Tier 2 x1.5 Tier 3 x2.0 Tier 4 x3.0
Splits Train 150 / Val 25 / Test-Public 15 / Test-Hidden 10 (per task)
Tracks Simulation 40% + Real-robot 60%