DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset. Khazatsky, Pertsch, Nair, Balakrishna, Dasari, Karamcheti and others, RSS 2024.
Summary
DROID is a robot manipulation dataset of 76k successful demonstrations (350 hours) on a shared Franka Panda hardware design, collected by 50 people with 18 robots at 13 institutions, in 564 scenes, 86 tasks and 52 buildings.
Its point is diversity: real homes, offices and labs, cameras that collectors move between sessions (1,417 viewpoints), and tasks that collectors choose themselves.
Co-training a diffusion policy with a 40k-trajectory, language-annotated subset of DROID and in-domain task demonstrations beats co-training on Open X-Embodiment by 22 points of success in-distribution and 17 points out of distribution, and a size-matched test on three tasks supports the value of scene diversity.
One-page summary
In class
Keep these in mind while the presenter talks. Tick them on page 2 of the PDF as they come up.
FAQ
76k is the number of successful demonstrations. About 16k more episodes were labelled "not successful"; they are in the release but do not count toward the 76k, and the policy experiments leave them out. (Sec. III-B, Sec. V-A)
The paper gives both numbers. Section III says 18 robots and 50 data collectors across 13 institutions, and that the setup was built 18 times; the introduction says 18 research labs. Read 13 as institutions and 18 as robots (setups). (Sec. I, Sec. III, Sec. III-A)
No. All episodes use the same hardware design: a Franka Panda arm with a Robotiq 2F-85 gripper, two exterior Zed 2 cameras, a Zed Mini wrist camera and a Quest 2 for teleoperation, built 18 times. DROID is diverse in scenes, tasks, objects and viewpoints, not in embodiments, which is the opposite trade to Open X-Embodiment. (Sec. III-A, Fig. 2)
No. All evaluated manipulation policies use in-domain demonstrations. The baselines train on those demonstrations alone or mix them 50/50 with OXE; the DROID variants mix them 50/50 with DROID data. The authors collect 50–150 demonstrations per task (Toasting and Close Waffle Maker share one multi-task policy). Training for new scenes without in-domain data is listed as an open question. (Sec. V-A, Sec. V-B, Sec. VI, Appendix F-B)
The introduction summarizes the improvement as 20% on average; Figure 8 gives the exact gains: averaged over 6 tasks in 4 locations, co-training with DROID beats the next best method (co-training with OXE) by 22 points of absolute success in-distribution and by 17 points out of distribution. Each task setting and method has 10 rollouts. (Sec. I, Sec. V-B, Fig. 8)
Both, and the paper separates them once. Two subsets of the same size (7,362 trajectories) from the training pool are compared on three out-of-distribution tasks: the 20 scenes with the most demonstrations, and a uniform random sample of 7,362 successful demonstrations. The diverse sample does better, and training with the full 40k-trajectory pool matches or beats both on the three tasks. (Sec. V-C, Fig. 10, Appendix F-B)
No. It uses an existing diffusion policy from Robomimic: a ResNet-50 encoder per exterior camera, a frozen DistilBERT instruction embedding, proprioception, and absolute end-effector actions. The contribution is the data, the collection platform and the analysis. (Sec. V-A, Appendix F)
A substantial change of the robot's workspace, such as moving from the stove to the sink in one kitchen. New objects or moved cameras do not count. The collection software logged 2,080 scene IDs; after removing duplicates the authors report 564, which they call a conservative count. (Sec. IV(c), Appendix D)
To get viewpoint diversity. The software regularly prompts "scene augmentations": nudging the robot, moving and re-calibrating the exterior cameras, changing the lighting, adding or removing objects. DROID has 1,417 camera viewpoints. The paper argues this helps with new camera angles; its Cook Lentils out-of-distribution test adds a camera shift together with distractors, but no experiment isolates the effect of the camera. (Sec. III-B, Sec. IV, Sec. V-A, Appendix E)
Collectors select or type the feasible tasks for each scene, and before each episode the software samples one of them at random, so collectors do not pick only easy tasks. After collection, crowd workers write up to three instructions per episode. The paper's policies condition on an instruction through a frozen DistilBERT embedding. (Sec. III-B, Fig. 11, Sec. V-A)
Its scenes and tasks are repetitive and its raw size was hard for the authors' training pipeline. It is about 5% of the Octo training mixture that the OXE baseline otherwise follows. (Sec. V-B)
Video
Glossary
Links
How this page was made. The video is an AI narration (Breeze TTS 2) of the full paper with the text on screen. The summary, questions and sketchnote were drafted with an AI assistant from the paper's text and checked against it; the references in brackets point to the paper. Paper license: CC BY 4.0. The sketchnote is an original drawing, not a copy of the paper's figures.