Open X-Embodiment: Robotic Learning Datasets and RT-X Models. Open X-Embodiment Collaboration (O'Neill, Rehman, Gupta, Maddukuri, Gupta, Padalkar and others), ICRA 2024.
Summary
Open X-Embodiment pools 60 existing robot datasets from 34 labs into one format: more than 1M real robot trajectories from 22 robot types, from single arms to quadrupeds.
The authors train two existing policies, RT-1-X (35M parameters) and RT-2-X (a 55B vision-language-action model), on a 12-dataset mixture from 9 manipulators, with every action coarsely mapped to a 7-DoF end-effector action.
RT-1-X raises mean success on 5 small-data domains by about 50% over each dataset's original method, and RT-2-X nearly triples success on Google Robot tasks whose skills are in the Bridge (WidowX) data but not in the Google Robot data (27.3% to 75.8%); in the two large-data domains only RT-2-X keeps up with single-robot training.
One-page summary
In class
Keep these in mind while the presenter talks. Tick them on page 2 of the PDF as they come up.
FAQ
No. The authors pooled 60 existing datasets from 34 research labs and converted them into one format (RLDS). Together they hold more than 1M real robot trajectories from 22 robot types, from single arms to bimanual robots and quadrupeds. The paper also counts 21 institutions (abstract, Sec. III, Sec. VI) next to the 34 labs, and does not explain the difference. (Abstract, Sec. III-A, Fig. 2)
No. Franka appears in the most datasets and scenes, while a few large datasets give a few robots most of the trajectories (Fig. 2). Pavel's audit of the 60 v1.0 rows in the public dataset sheet puts the top three robots (xArm, Kuka iiwa, Google Robot) at 85.6% of the listed trajectories, or 80.5% if QT-Opt counts only the 209,880 trajectories in the Octo-base-1.5 and OpenVLA statistics. Franka has 23 of the 60 datasets but 4.0% of the listed trajectories (5.4% with that count). These shares come from the audit, not from the paper. (Sec. III-B, Fig. 2; bushuyeu.com/review-oxe-empiricist)
No. The paper gives only "1M+". Pavel's audit sums the 60 v1.0 rows of the public dataset sheet to 1,403,481 trajectories, 580,392 of them from QT-Opt (Kuka). The Octo-base-1.5 and OpenVLA data statistics count only 209,880 QT-Opt trajectories, and with that count the total is 1,032,969. So "1M+" holds either way, but the metadata do not show why the two counts differ. (Sec. III-A; bushuyeu.com/review-oxe-empiricist)
No. Both were trained on a mixture of 12 datasets from 9 manipulators, among them RT-1, QT-Opt, Bridge and Language Table. The paper says this was all the data at the time of the experiments; the repository grew to 22 embodiments later. So when Sec. V-A says "the full X-embodiment dataset", it means this mixture. (Sec. IV-C, Sec. V-A)
RT-1-X is the 35M-parameter RT-1 (EfficientNet with FiLM, a USE instruction embedding, a 15-image history and a Transformer), trained on the robot mixture only and run locally. RT-2-X is RT-2-PaLI-X, a 55B vision-language-action model (5B in some ablations), co-fine-tuned on about a 1:1 mix of web vision-language data and the robot mixture, and queried on a cloud service. Both predict 8 action tokens with 256 bins each: 7 for end-effector movement and 1 for ending the episode. Neither is a new architecture: the paper uses RT-1 and RT-2 with minimal changes. (Sec. IV-B, Sec. IV-C, Tables I–II)
Only coarsely. Each dataset's actions are converted to a 7-DoF end-effector action (x, y, z, roll, pitch, yaw, gripper), normalized per dataset and binned, and one camera view per dataset is resized to a common resolution. Coordinate frames are not aligned, and a value can be an absolute or relative position or a velocity, so the same action vector can move two robots very differently. The paper lists only images and an instruction as inputs; no input tells the model which robot it controls. (Sec. IV-A, Sec. II)
The paper names them but gives no size threshold. Small: Kitchen Manipulation, Cable Routing, NYU Door Opening, AUTOLab UR5 and Robot Play, each tested with the robot and protocol of its own paper. Large: Bridge on the WidowX and the RT-1 data on the Google Robot. Only RT-1-X is tested on the small domains; both models are tested on the large ones. (Sec. V-A, Fig. 4, Table I)
The mean success rate of RT-1-X over the 5 small-data domains against two baselines: each dataset's Original Method and RT-1 trained on that dataset alone. The bar labels in Fig. 4 give means of 63% for RT-1-X, 41% for the Original Method and 44% for RT-1, so the gain is relative (about 54% and 43%) and about 20 points absolute. RT-1-X beats the Original Method on 4 of 5 domains; on AUTOLab UR5 it scores 45% against 53%. (Fig. 4, Sec. V-A, Sec. VI)
Not for RT-1-X, and Table I cannot tell for RT-2-X. RT-1-X scores below RT-1 trained on the single dataset in all three settings (27% vs 40%, 27% vs 30%, 73% vs 92%), which the authors read as underfitting. RT-2-X (55B) roughly matches or beats that RT-1 (50% vs 40%, 30% vs 30%, 91% vs 92%), but the table has no RT-2 trained on one dataset, so it does not separate pooling from model size, architecture and web pre-training. The caption also says RT-1-X is worse than the Original Method, but the table gives 27% against 13% on both Bridge settings. (Table I, Sec. V-A)
RT-2-X against RT-2, both 55B, on Google Robot tasks whose objects and skills are in the Bridge (WidowX) data but not in the Google Robot training data: 75.8% against 27.3%. Without Bridge in training, RT-2-X drops to 42.8%, which points to transfer from the WidowX data. On unseen objects, backgrounds and environments the two models are on par (RT-2 62%, RT-2-X 61%), although Sec. VI calls the result "~3× generalization improvements". (Sec. V-B, Table II rows (1)–(3), Sec. VI)
Yes, for the RT-2-X runs in the table. Rows (2) and (5) differ only in size (55B vs 5B, both with no image history): 75.8% against 14.5% on emergent skills and 61% against 30% on generalization. The paper instead compares rows (2) and (4), where size and history change together (75.8% vs 44.4%). Other single-change pairs show the effect of history (rows (4) and (5): 44.4% vs 14.5%) and of web pre-training (rows (7) and (6): 48.7% vs 0%). (Sec. V-C, Table II)
It does not test a robot outside the training mix, robots with very different sensing and actuation, or a rule for when transfer helps; Sec. VI lists these as open problems. RT-1-X is never tested out of distribution, and RT-2-X is not tested on the small-data domains. The paper reports 3,600 trials over 6 robots in total, but not the number of trials behind each success rate. (Sec. V, Sec. VI)
Glossary
Links
How this page was made. The video is an AI narration (Breeze TTS 2) of the full paper with the text on screen. The summary, questions and sketchnote were drafted with an AI assistant from the paper's text and checked against it; the references in brackets point to the paper. Paper license: CC BY 4.0. The sketchnote is an original drawing, not a copy of the paper's figures.