Teaching a household robot in VR

The 2026 BEHAVIOR Challenge tests robots on 100 household tasks in a simulator and provides 20,000 recorded demonstrations by people. Our robot has two arms and a learned controller that chooses movements from camera images and robot information. We saved failed attempts and the state of the virtual scene, like a saved game, so an expert could resume near a mistake.

On the older tasks, a reviewer marked the failure in a policy rollout. On the new tasks, the follower ran from demonstration phase starts and an automatic detector marked failed phases. The challenge’s task score measures goal conditions true at the end, so completing one pick-up is not equivalent to completing a task.

The operator used a Quest 3S or Pico 4 Enterprise headset with NVIDIA CloudXR streaming the simulator view. The right controller’s B button handed control from the policy to inverse-kinematics controllers that followed the operator’s hands. Pressing B again restored the policy controllers. A red border identified human-control frames.

Control ran at 30 Hz and physics at 120 Hz. These are update rates; we did not measure end-to-end VR latency. The console recorded ten seconds of policy continuation after hand-back. On the new tasks, human corrections started directly from the failed state without a policy in the loop, a variant closer to relabeling learner states than to gated online HG-DAgger.

A training example pairs what the robot saw with the movement the teacher supplied. Our recording saves camera images, robot movements and a marker of who controlled each moment. The processing then keeps human corrections as examples in the robot’s action format. The speaking notes explain the technical details for questions.

This format connects the VR interface to the training stack. An incorrect hand-back label can turn the learner’s own actions into purported expert examples.

With replay starting five seconds before the failure mark, people often prevented the mistake. In the team notebook, 29 of 33 place recordings began with the can already held, and only one included a human re-grasp. The default moved to three seconds after the mark on October 1, after 78 of the initial 82 recordings had been collected.

Human time limited collection: 78 recordings took four days on one task. The 82-recording batch became 115 intervention episodes. The planned policy-training pilot on this human takeover dataset was canceled, so its effect on policy performance was not tested.

On the new tasks, automatic experts corrected saved failure states. Privileged per-skill scripts produced 4,166 recordings across 48 tasks. For selected skills, an AI agent reviewed the video and a person made the acceptance decision: 1,028 clips were reviewed, with 855 published and 173 denied. Other groups used the automatic detector gate; 116 early recordings bypassed the video review.

We also used Comet’s older-task policy, pt50, as the expert. It completed 754 of 3,961 attempts (19%). The first round kept 500 pick-up and place corrections; the submitted follower’s recipe lists 499. Another 254 corrections were collected later and were not trained on.

A successful final state does not establish that the recorded actions form a useful demonstration.

With a 10% share of stronger-policy corrections and 1,000 fine-tuning steps, held-out pick-up phase success rose from 37.6% to 44.6% against a matched demonstrations-only control: +7.0 percentage points, 95% bootstrap interval over tasks [+1.0, +12.7], on 383 pick-up phases. This result uses a privileged phase detector and does not establish an end-to-end task-score gain.

The human takeover dataset was not tested in that policy-training pilot. Separately, 66 of the 111 human recordings on new tasks were encoded for the submitted phase-end detector, with 45 in its training split and 21 in validation. Scripted recoveries hurt the policy; a waist-joint mismatch was an important explanation, with training duration and fumbling also confounded.

On the 49 new tasks’ submitted instances, the follower completed zero full tasks in 490 rollouts, with mean Q 0.031. The team’s self-reported overall submission was Q 0.092 and 1.8% task success; all 18 successes came from pt50 on older tasks. These values describe the October submission and were checked in the November 3 report.

Make control ownership and hand-back explicit. Choose the replay point for the behavior being collected, prevention or recovery. Compare per-joint action labels with demonstrations before training. Use the same failure snapshots for a controlled study of takeover authority and delay, then run another collection-and-training round with the updated learner.

The companion DAgger and HG-DAgger paper review explains the two methods and proposes the user study.

Source: “Harvesting Failures for Imitation: Human-Gated and Privileged DAgger in the 2026 BEHAVIOR Challenge,” team report, November 3, 2026, Sections 3–7 and Tables 1–3. The full report remains an internal draft. The report’s authors checked the numbers against the team’s records; dates and comparators are retained here. Videos are the team’s own simulator recordings. AI tools helped draft and construct this page and the slides.