Open X-Embodiment read-along

Open X-Embodiment: Robotic Learning Datasets and RT-X Models. Open X-Embodiment Collaboration (O'Neill, Rehman, Gupta, Maddukuri, Gupta, Padalkar and others), ICRA 2024.

In short

Summary

Open X-Embodiment pools 60 existing robot datasets from 34 labs into one format: more than 1M real robot trajectories from 22 robot types, from single arms to quadrupeds.

The authors train two existing policies, RT-1-X (35M parameters) and RT-2-X (a 55B vision-language-action model), on a 12-dataset mixture from 9 manipulators, with every action coarsely mapped to a 7-DoF end-effector action.

RT-1-X raises mean success on 5 small-data domains by about 50% over each dataset's original method, and RT-2-X nearly triples success on Google Robot tasks whose skills are in the Bridge (WidowX) data but not in the Google Robot data (27.3% to 75.8%); in the two large-data domains only RT-2-X keeps up with single-robot training.

Sketchnote

One-page summary

Hand-drawn summary of Open X-Embodiment: Open X-Embodiment pools 60 existing robot datasets from 34 labs into one format: more than 1M real robot trajectories from 22 robot types, from single arms to quadrupeds. The authors train two existing policies, RT-1-X (35M parameters) and RT-2-X (a 55B vision-language-action model), on a 12-dataset mixture from 9 manipulators, with every action coarsely mapped to a 7-DoF end-effector action. RT-1-X raises mean success on 5 small-data domains by about 50% over each dataset's original method, and RT-2-X nearly triples success on Google Robot tasks whose skills are in the Bridge (WidowX) data but not in the Google Robot data (27.3% to 75.8%); in the two large-data domains only RT-2-X keeps up with single-robot training.
A one-page summary in sketchnote style. Download the PDF (page 1 the sketchnote, page 2 the listening sheet) to annotate it in Concepts or GoodNotes.

Listen for

In class

Keep these in mind while the presenter talks. Tick them on page 2 of the PDF as they come up.

  1. What "22 embodiments" and "60 datasets" count, and which 9 robots RT-X actually trains on
  2. What the shared action format aligns, and what it leaves unaligned
  3. What each result is compared against: the Original Method, RT-1 on one dataset, or RT-2
  4. Which comparisons isolate model size, and which also change the architecture or web pre-training
  5. Which experiments use RT-1-X and which use RT-2-X

Questions worth asking

  1. The inputs are images and an instruction only, and the coordinate frames are not aligned. What would an explicit robot input add, and how would you test it?
  2. The emergent skills move from Bridge (WidowX) to the Google Robot. What experiment would show transfer between other pairs of robots, or to a robot outside the mix?
  3. The paper gives no rule for when transfer helps. What would you measure on a new robot's dataset to predict it?

In class, on an iPad

  1. Before: watch or listen to the read-along (the chapters are below) and skim the questions below.
  2. Open the sketchnote PDF in Concepts or another note app. Page 1 is the summary drawing, with room to annotate. Page 2 is the listening sheet.
  3. During the talk: tick each "listen for" item as it comes up, and write one line in each box of the sheet.
  4. After: mark the one answer below that the talk changed for you, and ask your question.

Common questions and misunderstandings

FAQ

Was the Open X-Embodiment data collected for this paper?

No. The authors pooled 60 existing datasets from 34 research labs and converted them into one format (RLDS). Together they hold more than 1M real robot trajectories from 22 robot types, from single arms to bimanual robots and quadrupeds. The paper also counts 21 institutions (abstract, Sec. III, Sec. VI) next to the 34 labs, and does not explain the difference. (Abstract, Sec. III-A, Fig. 2)

Is the data spread evenly over the 22 robots?

No. Franka appears in the most datasets and scenes, while a few large datasets give a few robots most of the trajectories (Fig. 2). Pavel's audit of the 60 v1.0 rows in the public dataset sheet puts the top three robots (xArm, Kuka iiwa, Google Robot) at 85.6% of the listed trajectories, or 80.5% if QT-Opt counts only the 209,880 trajectories in the Octo-base-1.5 and OpenVLA statistics. Franka has 23 of the 60 datasets but 4.0% of the listed trajectories (5.4% with that count). These shares come from the audit, not from the paper. (Sec. III-B, Fig. 2; bushuyeu.com/review-oxe-empiricist)

Is "1M+ trajectories" an exact count?

No. The paper gives only "1M+". Pavel's audit sums the 60 v1.0 rows of the public dataset sheet to 1,403,481 trajectories, 580,392 of them from QT-Opt (Kuka). The Octo-base-1.5 and OpenVLA data statistics count only 209,880 QT-Opt trajectories, and with that count the total is 1,032,969. So "1M+" holds either way, but the metadata do not show why the two counts differ. (Sec. III-A; bushuyeu.com/review-oxe-empiricist)

Were RT-1-X and RT-2-X trained on all 22 robots?

No. Both were trained on a mixture of 12 datasets from 9 manipulators, among them RT-1, QT-Opt, Bridge and Language Table. The paper says this was all the data at the time of the experiments; the repository grew to 22 embodiments later. So when Sec. V-A says "the full X-embodiment dataset", it means this mixture. (Sec. IV-C, Sec. V-A)

How do RT-1-X and RT-2-X differ?

RT-1-X is the 35M-parameter RT-1 (EfficientNet with FiLM, a USE instruction embedding, a 15-image history and a Transformer), trained on the robot mixture only and run locally. RT-2-X is RT-2-PaLI-X, a 55B vision-language-action model (5B in some ablations), co-fine-tuned on about a 1:1 mix of web vision-language data and the robot mixture, and queried on a cloud service. Both predict 8 action tokens with 256 bins each: 7 for end-effector movement and 1 for ending the episode. Neither is a new architecture: the paper uses RT-1 and RT-2 with minimal changes. (Sec. IV-B, Sec. IV-C, Tables I–II)

Are the robots' action spaces unified?

Only coarsely. Each dataset's actions are converted to a 7-DoF end-effector action (x, y, z, roll, pitch, yaw, gripper), normalized per dataset and binned, and one camera view per dataset is resized to a common resolution. Coordinate frames are not aligned, and a value can be an absolute or relative position or a velocity, so the same action vector can move two robots very differently. The paper lists only images and an instruction as inputs; no input tells the model which robot it controls. (Sec. IV-A, Sec. II)

Which domains count as small-data and which as large-data?

The paper names them but gives no size threshold. Small: Kitchen Manipulation, Cable Routing, NYU Door Opening, AUTOLab UR5 and Robot Play, each tested with the robot and protocol of its own paper. Large: Bridge on the WidowX and the RT-1 data on the Google Robot. Only RT-1-X is tested on the small domains; both models are tested on the large ones. (Sec. V-A, Fig. 4, Table I)

What does "50% higher" compare?

The mean success rate of RT-1-X over the 5 small-data domains against two baselines: each dataset's Original Method and RT-1 trained on that dataset alone. The bar labels in Fig. 4 give means of 63% for RT-1-X, 41% for the Original Method and 44% for RT-1, so the gain is relative (about 54% and 43%) and about 20 points absolute. RT-1-X beats the Original Method on 4 of 5 domains; on AUTOLab UR5 it scores 45% against 53%. (Fig. 4, Sec. V-A, Sec. VI)

Does pooled data help in the large-data domains?

Not for RT-1-X, and Table I cannot tell for RT-2-X. RT-1-X scores below RT-1 trained on the single dataset in all three settings (27% vs 40%, 27% vs 30%, 73% vs 92%), which the authors read as underfitting. RT-2-X (55B) roughly matches or beats that RT-1 (50% vs 40%, 30% vs 30%, 91% vs 92%), but the table has no RT-2 trained on one dataset, so it does not separate pooling from model size, architecture and web pre-training. The caption also says RT-1-X is worse than the Original Method, but the table gives 27% against 13% on both Bridge settings. (Table I, Sec. V-A)

What does the "about 3×" emergent-skills result compare?

RT-2-X against RT-2, both 55B, on Google Robot tasks whose objects and skills are in the Bridge (WidowX) data but not in the Google Robot training data: 75.8% against 27.3%. Without Bridge in training, RT-2-X drops to 42.8%, which points to transfer from the WidowX data. On unseen objects, backgrounds and environments the two models are on par (RT-2 62%, RT-2-X 61%), although Sec. VI calls the result "~3× generalization improvements". (Sec. V-B, Table II rows (1)–(3), Sec. VI)

Does Table II show that a bigger model transfers more?

Yes, for the RT-2-X runs in the table. Rows (2) and (5) differ only in size (55B vs 5B, both with no image history): 75.8% against 14.5% on emergent skills and 61% against 30% on generalization. The paper instead compares rows (2) and (4), where size and history change together (75.8% vs 44.4%). Other single-change pairs show the effect of history (rows (4) and (5): 44.4% vs 14.5%) and of web pre-training (rows (7) and (6): 48.7% vs 0%). (Sec. V-C, Table II)

What did the paper not test?

It does not test a robot outside the training mix, robots with very different sensing and actuation, or a rule for when transfer helps; Sec. VI lists these as open problems. RT-1-X is never tested out of distribution, and RT-2-X is not tested on the small-data domains. The paper reports 3,600 trials over 6 robots in total, but not the number of trials behind each success rate. (Sec. V, Sec. VI)

Chapters

Video

  1. 0:00Abstract
  2. 1:331. Introduction
  3. 6:052. Related work
  4. 8:593. The Open X-Embodiment repository
  5. 12:504. RT-X design
  6. 19:465. Experimental results
  7. 29:046. Discussion, future work, and open problems
  8. 31:06Figures and tables

Key terms

Glossary

Embodiment
A robot type with its own body, sensors and actuators; OXE has 22, from single arms to quadrupeds.
X-embodiment training
Training one policy on data from several robot types at once.
Positive transfer
Data from other robots raises success on a robot above training on its own data only.
Original Method
The model each dataset's creators trained on that dataset alone; the main baseline.
RLDS
The episode format (serialized TFRecord files) that OXE stores all 60 datasets in.
VLA
Vision-language-action model: a vision-language model fine-tuned to output robot actions as tokens.
Co-fine-tuning
Fine-tuning on robot data mixed with web vision-language data; about 1:1 for RT-2-X.
Action bins
The 7 end-effector dimensions and 1 end-of-episode dimension, each split into 256 uniform bins and predicted as a class.
Emergent skills
Here: Google Robot tasks whose objects and skills are in Bridge (WidowX) data but not in the Google Robot data.