DAgger and HG-DAgger: paper review

HG-DAgger: Interactive Imitation Learning with Human Experts. Michael Kelly, Chelsea Sidrane, Katherine Driggs-Campbell and Mykel J. Kochenderfer. Stanford University and University of Illinois Urbana-Champaign. ICRA 2019.

A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. Stéphane Ross, Geoffrey J. Gordon and J. Andrew Bagnell. Carnegie Mellon University. AISTATS 2011.

This paper review covers the problem, the two approaches, novelty, evidence, usable lessons, weaknesses and a proposed VR user study. The Google Slides deck has explanatory speaking notes for every slide, with timing, visual cues and transitions. The printable notes add a glossary and questions with short answers.

Imagine a demonstration where the robot grasps a can and then places it in a bin. If the robot misses the grasp, its hand stays empty. It needs a correction for that situation. Learning only from successful demonstrations, called behavioral cloning, may leave that correction unseen. DAgger collects expert answers in situations the robot encounters during its own practice.

DAgger: let the robot practice, ask an expert what action it should take, keep those examples together with earlier demonstrations, and train again. A policy is the robot’s learned controller: the part that chooses a movement. The next round uses the improved controller.

HG-DAgger: the human chooses when to take over and keeps full control until handing back. Only human-control intervals add new action labels. This gives the human a consistent feedback signal during a correction.

DAgger explains why teaching on the robot’s own situations can reduce errors that build on earlier mistakes. Its result depends on repeated practice and suitable expert answers. HG-DAgger makes teaching easier to organize around human control: a person chooses when to take over and keeps full control while correcting. Takeover events also help calibrate a warning signal based on disagreement among learned controllers. That signal estimates risk; it does not prove that an action is safe.

In Super Tux Kart, DAgger reached zero falls per lap after 15 training rounds in the reported evaluation (Ross et al., Section 5.1). HG-DAgger was tested in simulated driving and on a physical car navigating simulated obstacles. For the first 5,000 physical-car samples, the paper reports zero collisions and zero road departures for HG-DAgger, compared with one collision and six departures for cloning, and one of each for DAgger (Kelly et al., Table II).

The car result is a small feasibility demonstration. The samples are sequential control samples, not independent driving trials, and the result does not establish general driving safety.

The robot needs an example for what just went wrong. The teacher needs predictable control while showing the correction. In XR, the handover button and a clear indication of who controls the robot influence the examples that get recorded.

DAgger requires repeated collection and training, expert labels on difficult states, and a way to manage unsafe rollouts. HG-DAgger assumes the human detects danger soon enough. Its explanation involving perceived lag is not isolated in a controlled user study, its physical evaluation is small, and the risk-threshold rule is a heuristic.

Use saved simulator failures to compare two interfaces: the person takes full control, or shares control with the robot. Separately test more or less delay between hand movement and robot response. Measure whether corrections work, how long they take, and how difficult they feel. Then test whether robots learn better from the recordings. This is a proposed experiment; the reviewed papers did not run it.

The separate BEHAVIOR project presentation explains our implemented VR recovery loop and its measured outcomes.

Sources: the two linked primary papers, especially DAgger Sections 2–5 and HG-DAgger Sections II–IV. Figure credit: Kelly et al., 2019. AI tools helped draft and construct the slides and this page; claims were checked against the papers.