WorldOracle benchmark checks robot world models against what actually happened

A robot reaches for a piece of fruit, closes its gripper and misses. Given the same starting frame and the same action sequence, a world model produced a video in which the pick succeeded. The video looked plausible. The success it showed never happened.

The case comes from WorldOracle, a benchmark for action-conditioned world models (ACWMs) built by researchers from Princeton University, the University of Texas at Austin, the University of Toronto, NVIDIA and AgiBot. It opened to developers on October 16. World models are increasingly proposed as a way to train and test robots in generated video rather than on real hardware. That only works if the generated future matches the real one. WorldOracle measures how often it does.

How the benchmark works

The team first recorded real manipulation episodes, capturing both video and the actions the robot or person performed. A model under test receives the initial frame and the action sequence and generates a predicted video. The platform then scores that video against the real execution of the same actions.

The benchmark covers two robot embodiments, the AgiBot G2 and YAM, plus first-person video of human hands. Four tasks each involve a different kind of object: fruit picking (rigid), towel hanging (deformable), water pouring (fluid) and drawer pulling (articulated). Task coverage varies by embodiment. The human-hand setting is meant for models that accept hand-motion input, and those models are evaluated zero-shot, without adaptation data from the platform.

Failures are part of the test set on purpose. Missed grasps, drops and spills all appear, and if an operation failed in reality, predicting that failure counts as correct. According to the researchers, these samples reveal whether a model follows the given actions and their consequences, or defaults to a successful version of the task. The data also includes autonomous policy rollouts, with the pauses, jitter and corrections that distinguish them from human teleoperation.

The researchers say that among the embodied world-model benchmarks they compared, WorldOracle is the only one that combines action-sequence input, real execution data, failure cases, long-horizon interaction, multiple embodiments, multi-view evaluation and autonomous rollouts.

What the scores measure

A single number can hide the error that matters. The arm in a predicted video can track the real arm closely while the model gets the fruit's grasp state wrong. WorldOracle therefore uses four metrics, described on the project page:

• Physical State Correctness: a vision-language model judges physical-state criteria at the outcome keyframe of each segment.
• Interaction Spatial Alignment: checks whether the arm or hand and the manipulated object sit near their real positions, scoring arm and object separately.
• Key Interaction Area Fidelity: measures pixel, perceptual and semantic fidelity inside the regions of the object and the arm or hand.
• Multi-View Action Coherence: compares robot-arm alignment with the real execution across top, left and right camera views.

The first three are averaged with equal weight into a single-view score, WorldOracle-SV. For multi-view evaluation, the fourth is added and all four are weighted equally. Sub-scores are kept so developers can separate errors in state prediction, motion and local detail. To validate the metrics, the researchers asked 119 participants to compare 128 pairs of predicted videos. The metrics agreed with the human choice of the better prediction 88.1% to 93.2% of the time.

What the first results show

The team evaluated seven models, including DreamDojo, Cosmos 3, IRASim and GE-Sim 2.0. Two patterns stood out. Robot arm paths were predicted relatively well, but the positions and movements of manipulated objects could still deviate from reality. And models that did well over short horizons could degrade significantly over a full task.

The benchmark scores both the first 30 seconds of each trajectory and the complete sequence, because in a multi-step task each interaction leaves the starting state for the next. Rankings changed between the two. IRASim scored below Ctrl-World, DreamDojo and Cosmos 3 over the first 30 seconds, but above all three over the full sequence.

The researchers also state the benchmark's limits. It compares each prediction with one recorded execution. For fluids, deformable objects and stochastic contact, the same action can lead to several plausible outcomes, and one reference cannot capture that range. Training support also differs by setting: AgiBot G2 models use a benchmark-specific training set, YAM models use public data, and the human-hand setting is zero-shot. The team says scores across the three settings should not be used to judge embodiment difficulty or cross-embodiment generalisation.

The platform accepts open-weight and closed-source models. After registration and approval, developers pick a track, upload predicted videos following the submission guide, and receive composite scores, sub-scores and comparisons.

What it means for work

World models are being positioned to take over part of the slow, expensive work of robot development: collecting real trials, testing policies on hardware and checking whether a change helped. The fruit case shows the risk in that handoff. A model that shows success where the real robot failed would tell an engineer a policy works when it does not, and a policy trained on such futures would learn from outcomes that never occur. WorldOracle gives robotics teams a way to check this before relying on a model, with separate scores for the arm, the object and long sequences. It does not make world models trustworthy by itself. It measures how far they are from that, and only against one recorded outcome per run. For tasks involving cloth, liquids or uncertain contact, real-world trials remain the reference for now.