DROID read-along

DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. Khazatsky, Pertsch, Nair, Balakrishna, Dasari, Karamcheti and others, RSS 2024.

In short

Summary

DROID is a robot manipulation dataset of 76k successful demonstrations (350 hours) on a shared Franka Panda hardware design, collected by 50 people with 18 robots at 13 institutions, in 564 scenes, 86 tasks and 52 buildings.

Its point is diversity: real homes, offices and labs, cameras that collectors move between sessions (1,417 viewpoints), and tasks that collectors choose themselves.

Co-training a diffusion policy with a 40k-trajectory, language-annotated subset of DROID and in-domain task demonstrations beats co-training on Open X-Embodiment by 22 points of success in-distribution and 17 points out of distribution, and a size-matched test on three tasks supports the value of scene diversity.

Sketchnote

One-page summary

Hand-drawn summary of DROID: DROID is a robot manipulation dataset of 76k successful demonstrations (350 hours) on a shared Franka Panda hardware design, collected by 50 people with 18 robots at 13 institutions, in 564 scenes, 86 tasks and 52 buildings. Its point is diversity: real homes, offices and labs, cameras that collectors move between sessions (1,417 viewpoints), and tasks that collectors choose themselves. Co-training a diffusion policy with a 40k-trajectory, language-annotated subset of DROID and in-domain task demonstrations beats co-training on Open X-Embodiment by 22 points of success in-distribution and 17 points out of distribution, and a size-matched test on three tasks supports the value of scene diversity.
A one-page summary in sketchnote style. Download the PDF (page 1 the sketchnote, page 2 the listening sheet) to annotate it in Concepts or GoodNotes.

Listen for

In class

Keep these in mind while the presenter talks. Tick them on page 2 of the PDF as they come up.

  1. What is diverse in DROID, and what is fixed (one robot model, one gripper model)?
  2. How the collection protocol creates diversity on purpose
  3. What the baselines are: in-domain only, and OXE co-training
  4. Which test separates diversity from dataset size
  5. Whether any experiment uses no in-domain demonstrations

Questions worth asking

  1. DROID varies scenes and viewpoints but not robots. When is that the better trade than OXE's 22 embodiments?
  2. Every evaluated manipulation policy uses in-domain demonstrations. What would a fair zero-shot test of DROID look like?
  3. The cameras move between sessions. Does that buy viewpoint robustness, and how would you measure it?

In class, on an iPad

  1. Before: watch or listen to the read-along (the chapters are below) and skim the questions below.
  2. Open the sketchnote PDF in Concepts or another note app. Page 1 is the summary drawing, with room to annotate. Page 2 is the listening sheet.
  3. During the talk: tick each "listen for" item as it comes up, and write one line in each box of the sheet.
  4. After: mark the one answer below that the talk changed for you, and ask your question.

Common questions and misunderstandings

FAQ

Is DROID 76k episodes or about 92k?

76k is the number of successful demonstrations. About 16k more episodes were labelled "not successful"; they are in the release but do not count toward the 76k, and the policy experiments leave them out. (Sec. III-B, Sec. V-A)

Was it collected at 13 institutions or 18 labs?

The paper gives both numbers. Section III says 18 robots and 50 data collectors across 13 institutions, and that the setup was built 18 times; the introduction says 18 research labs. Read 13 as institutions and 18 as robots (setups). (Sec. I, Sec. III, Sec. III-A)

Does DROID contain several kinds of robot?

No. All episodes use the same hardware design: a Franka Panda arm with a Robotiq 2F-85 gripper, two exterior Zed 2 cameras, a Zed Mini wrist camera and a Quest 2 for teleoperation, built 18 times. DROID is diverse in scenes, tasks, objects and viewpoints, not in embodiments, which is the opposite trade to Open X-Embodiment. (Sec. III-A, Fig. 2)

Does the paper show that a DROID policy works zero-shot in a new scene?

No. All evaluated manipulation policies use in-domain demonstrations. The baselines train on those demonstrations alone or mix them 50/50 with OXE; the DROID variants mix them 50/50 with DROID data. The authors collect 50–150 demonstrations per task (Toasting and Close Waffle Maker share one multi-task policy). Training for new scenes without in-domain data is listed as an open question. (Sec. V-A, Sec. V-B, Sec. VI, Appendix F-B)

Where does "20% better" come from?

The introduction summarizes the improvement as 20% on average; Figure 8 gives the exact gains: averaged over 6 tasks in 4 locations, co-training with DROID beats the next best method (co-training with OXE) by 22 points of absolute success in-distribution and by 17 points out of distribution. Each task setting and method has 10 rollouts. (Sec. I, Sec. V-B, Fig. 8)

Is the gain from more data or from more diverse data?

Both, and the paper separates them once. Two subsets of the same size (7,362 trajectories) from the training pool are compared on three out-of-distribution tasks: the 20 scenes with the most demonstrations, and a uniform random sample of 7,362 successful demonstrations. The diverse sample does better, and training with the full 40k-trajectory pool matches or beats both on the three tasks. (Sec. V-C, Fig. 10, Appendix F-B)

Does the paper propose a new policy or model?

No. It uses an existing diffusion policy from Robomimic: a ResNet-50 encoder per exterior camera, a frozen DistilBERT instruction embedding, proprioception, and absolute end-effector actions. The contribution is the data, the collection platform and the analysis. (Sec. V-A, Appendix F)

What counts as a "scene"?

A substantial change of the robot's workspace, such as moving from the stove to the sink in one kitchen. New objects or moved cameras do not count. The collection software logged 2,080 scene IDs; after removing duplicates the authors report 564, which they call a conservative count. (Sec. IV(c), Appendix D)

Why do the collectors move the cameras?

To get viewpoint diversity. The software regularly prompts "scene augmentations": nudging the robot, moving and re-calibrating the exterior cameras, changing the lighting, adding or removing objects. DROID has 1,417 camera viewpoints. The paper argues this helps with new camera angles; its Cook Lentils out-of-distribution test adds a camera shift together with distractors, but no experiment isolates the effect of the camera. (Sec. III-B, Sec. IV, Sec. V-A, Appendix E)

Where do the language instructions come from?

Collectors select or type the feasible tasks for each scene, and before each episode the software samples one of them at random, so collectors do not pick only easy tasks. After collection, crowd workers write up to three instructions per episode. The paper's policies condition on an instruction through a frozen DistilBERT embedding. (Sec. III-B, Fig. 11, Sec. V-A)

Why was Language Table removed from the OXE baseline?

Its scenes and tasks are repetitive and its raw size was hard for the authors' training pipeline. It is about 5% of the Octo training mixture that the OXE baseline otherwise follows. (Sec. V-B)

Chapters

Video

  1. 0:00Abstract
  2. 1:281. Introduction
  3. 6:182. Related work
  4. 13:063. DROID data collection setup
  5. 23:334. DROID dataset analysis
  6. 32:525. Experiments
  7. 47:416. Discussion
  8. 49:34Appendix B. DROID data features
  9. 50:33Appendix C. Scene type classification
  10. 51:48Appendix D. Identifying unique scenes
  11. 53:55Appendix E. Evaluation procedure
  12. 57:28Appendix F. Diffusion policy details
  13. 1:02:05Appendix G. Automatic camera calibration

Key terms

Glossary

Co-training
Training on two datasets at once; here each batch is half task demonstrations and half DROID.
In-distribution / OOD
Evaluation like the task demos (with small position noise) / with changes such as distractors or a new object.
Scene
A distinct robot workspace; 564 in DROID after removing duplicate scene IDs.
Viewpoint
A placement of the exterior cameras; DROID has 1,417.
OXE
Open X-Embodiment: pooled robot datasets spanning 22 embodiments. DROID's OXE baseline co-trains on a curated Octo-based subset without Language Table.
Diffusion policy
A policy that generates action sequences by denoising; the paper's fixed learner.
Teleoperation
A person drives the robot, here with Quest 2 controllers, to record demonstrations.