A joint research team from the University of Hong Kong, UC Berkeley, Tsinghua University, and Peking University has released RoboDojo, a comprehensive evaluation benchmark designed to assess generalist robot manipulation policies across both simulation and real-world environments. The benchmark addresses a critical gap in the field: existing evaluation methods either rely on simple, short-horizon tasks or fail to translate simulation performance to real-world deployment.
RoboDojo defines five core manipulation capabilities through 42 carefully designed tasks: generalization (adapting to unseen backgrounds, lighting, objects, and clutter), memory (leveraging historical information), long-horizon execution (multi-step manipulation sequences), fine-grained control (high-precision contact tasks like narrow-hole insertion), and open-ended instruction following (transferring learned skills to novel language commands and objects).
The evaluation platform consists of three components: a simulation platform for task construction, asset generation, parallel evaluation, and data collection; a real-world evaluation platform (RoboDojo-RealEval) emphasizing reproducibility through standardized hardware, workspace, lighting, and evaluation protocols; and XPolicyLab, a unified framework for policy development, training, evaluation, and deployment.
Results reveal that current generalist robot manipulation policies remain far from reliable. The best-performing model achieved an average success rate of only 8.80% in simulation and 12.8% in real-world deployment, compared to human teleoperation's near-perfect performance. The study found that different models exhibit uneven capability development: Spatial Forcing excels at generalization, X-VLA leads in fine manipulation, while Hy-Embodied-0.5-VLA performs best overall in long-horizon and memory tasks. However, none of these advantages span the full task set.
Key findings include that scene randomization significantly degrades model performance (Hy-Embodied-0.5-VLA dropped 92.9% under randomization), fine-grained control and memory remain critical weaknesses, and open-ended semantic instruction following shows the lowest success rates at just 1.67%. The benchmark establishes a public leaderboard and provides systematic diagnostic tools for the research community.
The paper is available on arXiv: https://arxiv.org/abs/2607.04434
