2026-09-21

BEHAVIOR-1K 2026 data composition

This page describes the 100-task benchmark from the files the evaluator reads. Those files are its constants, the robot configuration it loads, the goal definitions it scores against, and all 20,000 recorded demonstrations. The facts come first. The four places where the challenge documentation disagrees with the code come after. Two of them fall on the 50 tasks added for 2026, which nobody has run.

100 tasks20,000 demos1,018 BDDL definitions23 submissions from 2025

The data

What the benchmark is

These facts come before any of the questions that follow. Every figure here comes from the evaluator's own constants, the robot configuration it loads, or the recorded demonstrations. None comes from the challenge documentation.

The corpus

100

tasks

50 carried from 2025, 50 new, none dropped

20,000

teleop demos

exactly 200 per task, no exceptions

341s

median demo

10,233 steps at 30 Hz

scenes

4 new, hosting 24 tasks exclusively

0.089

2025 field-mean Q

unreported tasks counted as 0; 0.236 over reported ones

tasks at zero

183 rollouts run, none scored above 0

The new half is shorter, not longer. Carried tasks run 397 s on average. The 50 new tasks run 306 s, which is 23 % less. Length is one of the stronger difficulty predictors, so the 2026 set may be easier per task than the 2025 set. That is the opposite of the usual direction for a second-year benchmark.

Ten of the 20,000 episodes run under 40 seconds. The shortest is 148 steps, or 4.9 s, in putting_away_Halloween_decorations, where the median episode runs 7.7 minutes. Its six video streams span 4.93 s and match the recorded length exactly, so the recording is short rather than truncated. Ten episodes in 20,000 move no statistic. Filter on length if you sample episodes individually.

What the robot is, sees and does

Robot	R1Pro, a bimanual mobile manipulator	eval/r1pro.yaml
Control rate	30 Hz actions over 120 Hz physics — one action every 33 ms	eval_utils.py:177,179
Cameras	three — head at 720×720, each wrist at 480×480, RGB and depth	eval_utils.py:10,11
Proprioception	base velocity, per-arm joint position and velocity, end-effector pose, gripper and trunk state	r1pro.yaml proprio_obs
Action	23 dimensions: base 0:3, torso 3:7, left arm 7:14, left gripper 14, right arm 15:22, right gripper 22	eval_utils.py ACTION_QPOS_INDICES
Control mode	arms and trunk take absolute joint positions; the base takes a velocity, capped at 0.75 m/s and 1.0 rad/s	r1pro.yaml controller_config
Grasping	assisted — an object attaches when the gripper closes near it	r1pro.yaml grasping_mode

How it is scored

Q-score	`len(satisfied) / (len(satisfied) + len(unsatisfied))` over the goal predicates, read at episode end	behavior_task.py:410
Goal predicates	1 to 12 per task, median 3; 66 of 100 goal blocks contain a quantifier that expands at runtime	problem0.bddl
Episode timeout	`int(length × 1.5)` — from 3,224 steps on the shortest task to 39,090 on the longest	eval_utils.py:13, evaluator.py:163
Test instances	40 per task, ids 301–340: the first 20 public, the last 20 hidden	eval_utils.py:14–17
Tiebreakers	simulated time, base distance and end-effector displacement, each normalised as human ÷ robot, so above 1 beats the human demonstration	task.jsonl, score_utils.py

One consequence to carry into everything below. Q is a ratio whose denominator is the task's goal-predicate count, and that count is often 1. Partial credit is available on paper across the benchmark, and absent in practice on a quarter of it.

Data fit

How well the corpus suits each candidate

The corpus is fixed. These three methods need different things from it, and it serves them unequally. fits means the data suits the method as it is. workable means usable at a cost. outside means outside what the method has been shown to handle.

What the data suppliesverified from code and demos	Diffusion PolicyDDPM, 278M, per-task	π0.5 baseflow matching, ~3B, multi-task	Comet pt50π0.5 fine-tuned on 2025
200 demos per task, every task	fitsexactly its own setting — 200 proficient-human demos on every Robomimic task	fits20,000 total across tasks	fitstrained on `range(200)`, all of them
Median episode 10,264 steps shortest task 2,150	outsideits longest benchmark caps at 700 steps. The shortest BEHAVIOR task is 3.1× that; the median is 14.7×, and 37× Push-T	workabledesigned for long horizons, but its own reported evaluations are far shorter	workabledemonstrated at this length, at Q 0.25 held-out
23-dimension action mixed position and velocity	outsideits benchmarks run 2–14 dims, all single-mode control	fitsmodel action dim is 32; 23 is zero-padded to fit	fitssame
979,200 px rendered per step 720² head + 2×480² wrists, RGB and depth	outsidetrained at 84²–240²; the render is 69× its Square setting	workablethe serving wrapper resizes RGB to 224² and keeps depth at 720, so most of the render is discarded before the model sees it	workablesame wrapper
Per-segment skill language `annotations/`, all 100 tasks; but no one-sentence instruction for the new 50	workabledoes not use language, so neither the annotations nor the missing instruction change anything for it	fitsdense phase text is exactly what a language-conditioned model can use — if the loader reads it	workablesame, and openpi’s loader looks for an `orchestrators/` directory the release does not publish
100 tasks	outsideone policy per task in every published result — 100 training runs	fitsmulti-task by construction	workabletrained on the 50 carried tasks; has never seen the 50 new ones or the 4 new scenes
210.9M frames, 1,953 hours	outsideone BEHAVIOR task alone is 2.05M frames, 26× the whole Square dataset	workablewithin scale, but 3.27 TB gates who can train on it	workabletrained on the 2025 subset at batch 256 for 50k steps
30 Hz control	workableits real-robot setting is 10 Hz; the rate behind its simulated ablations is not stated in the paper	fitschunk 32 spans 1.07 s at this rate	fitschunk 32, matching

Diffusion Policy fits the corpus least well, and is the most useful control. Five of the eight rows fall outside what it has been shown to handle. The one row that matches exactly is 200 demonstrations per task. It is also the only one of the three that does not use language, so the missing instructions cost it nothing. It is small enough to train per task on one workstation GPU. That makes it a poor submission candidate and a useful control: it differs from the published Diffusion Policy results in task regime only, and from the π0.5 arm in architecture only.

Comet pt50 fits the corpus best, and is the most constrained. Its training set was behavior-1k/2025-challenge-demos. The 50 tasks it saw carry into 2026 with their demonstration lengths unchanged. Not one carried task moved. Half the 2026 benchmark is therefore in distribution for it. The other half is not, and neither are the four scenes that hold only new tasks.

The resolution result has a counter-intuitive cause. The evaluator renders 979,200 RGB pixels per step, and the serving wrapper resizes them to 224². Extra detail reaching the policy therefore cannot explain the measured benefit of full-resolution rendering. The model sees the same 224² image either way. The likely mechanism is downsampling quality. A 720² render reduced to 224² is better antialiased than a native 224² render. This is testable, and nobody has tested it.

What looks wrong

Four things that do not behave as documented

Ordered by how much each one should change a decision. Every claim below cites the file it came from. None rests on a docstring or a README.

Task durations do not match the demos, for the new 50 only
All 50 carried-over tasks agree with the evaluator's demo length to within a second. Of the 50 new ones, two do. Median error 13.3 %, worst +31.6 % on cook_brussels_sprouts. Scoring is unaffected. evaluator.py:163 reads length from task.jsonl and never reads this field. But any plan built on duration is wrong on the half of the benchmark that nobody has run. The table below gives each task.
source task_data.json vs datasets/2026-challenge-task-instances/metadata/task.jsonl · evaluator.py:163
24 tasks have exactly one goal predicate, so Q is binary
Q-score is len(satisfied)/(len(satisfied)+len(unsatisfied)). A one-predicate task has no partial credit. Q is 0 or 1. This includes turning_on_radio, the task that every published baseline is tuned on. The median task across the benchmark has 3 goal predicates.
source behavior_task.py:410 · bddl3/bddl/activity_definitions/*/problem0.bddl
The new 50 tasks have no high-level instruction, though the dataset is richly annotated
All 50 carried-over tasks carry an instruction string in task_data.json. None of the 50 new tasks carry one. meta/tasks.parquet holds only a task index, and the per-episode tasks field holds the bare snake_case name. That is 100 distinct values across all 20,000 episodes, not 20,000 annotations. The dataset does supply language, in a directory that none of those three files point to. annotations/ is the largest directory in the release, at 20,002 files, one JSON per episode for all 100 tasks. Each file carries per-segment skill text with frame ranges. For vacuuming_floors, one of the new 50: "move to" [0,416] on ["vacuum"], then "pick up from" [416,592] on ["vacuum","floors"], each tagged with a skill type. The gap is therefore the one-sentence task instruction, not language in general. An earlier version of this page stated that no language existed for the new 50 tasks. That was wrong. The check covered only meta/.
source task_data.json · meta/tasks.parquet · meta/episodes/chunk-*/file-*.parquet · annotations/task-0069/episode_00690030.json · HF list_repo_files
The five tasks nobody scored on in 2025 share one predicate
wash_dog_toys, clean_a_patio, clean_a_trumpet, cook_cabbage and make_pizza hold Q = 0.000 in all 23 submissions, and not one rollout of them ever scored above zero. Far fewer teams tried them than that makes it sound. Each was reported by only 3 to 6 of the 23 submissions, because most submissions are partial and an unreported task is recorded as 0.000. What survives is still a clean result: across the 183 rollouts that were actually run on these five tasks, by the two teams that reported the whole benchmark and four others, every single one scored 0.000. The privileged track, whose submissions may read ground-truth simulator state, is thinner evidence than it appears here: one privileged submission reported two of the five, both at zero, and none reported the other three. Three of the five are scored on covered, which appears in 60 % of the zero-score tasks against 5 % of the solved ones. covered is a particle-system state, such as dust, water and stains. The field did not fail at long horizons on these tasks. It failed at fluids.
source docs/challenge_submissions/*.json · goal predicates from problem0.bddl

Finding 01, in full

Duration, task by task

stated is task_data.json:duration, an integer field. actual is task.jsonl:length ÷ 30 Hz, the figure the evaluator uses. Because stated holds whole seconds, it cannot be more precise than ±1 s. Rows inside that band are therefore marked as agreeing, rather than shown a rounding artefact. timeout is the step cap the evaluator derives, int(length × 1.5). Sorted by disagreement.

50 / 50

carried tasks agree

within one second, every one

2 / 50

new tasks agree

halve_an_egg, unloading_the_car

26.6s

median new-task error

13.3 % of the true length

27 / 21

longer / shorter

than the website states; 52 agree

filter show

Taskid used by `--task-name`	Stated`task_data.json` duration	Actual`task.jsonl` length÷30	Δ show far the real length differs — ▲ longer, ▼ shorter, ● agrees	Δ %share of true length	Timeoutstep cap, `length×1.5`
`cook_brussels_sprouts`	687	522.1	▼−164.9	−31.6%	23,492
`setting_the_table`	727	593.6	▼−133.4	−22.5%	26,712
`tidying_bathroom`	541	433.4	▼−107.6	−24.8%	19,504
`tidying_living_room`	316	418.9	▲+102.9	+24.6%	18,848
`setup_a_bar_for_a_cocktail_party`	350	447.3	▲+97.3	+21.8%	20,128
`rearrange_your_room`	515	428.5	▼−86.5	−20.2%	19,282
`stacking_wood`	447	531.3	▲+84.3	+15.9%	23,909
`freeze_fruit`	339	420.6	▲+81.6	+19.4%	18,928
`make_gift_bags_for_baby_showers`	242	321.7	▲+79.7	+24.8%	14,476
`thawing_frozen_food`	348	274.3	▼−73.7	−26.9%	12,344
`put_together_a_basic_pruning_kit`	448	375.5	▼−72.5	−19.3%	16,896
`store_batteries`	328	256.8	▼−71.2	−27.7%	11,556
`putting_dirty_dishes_in_sink`	337	405.0	▲+68.0	+16.8%	18,222
`dispose_of_glass`	240	306.5	▲+66.5	+21.7%	13,792
`putting_away_toys`	438	376.6	▼−61.4	−16.3%	16,945
`organizing_school_stuff`	426	486.4	▲+60.4	+12.4%	21,887
`sorting_books_on_shelf`	200	258.0	▲+58.0	+22.5%	11,609
`bringing_paper_to_recycling`	318	373.8	▲+55.8	+14.9%	16,821
`laying_tile_floors`	520	478.4	▼−41.6	−8.7%	21,526
`make_rose_centerpieces`	111	151.1	▲+40.1	+26.5%	6,798
`dispose_of_batteries`	441	480.9	▲+39.9	+8.3%	21,642
`re_shelving_library_books`	281	317.6	▲+36.6	+11.5%	14,292
`cook_a_frozen_pie`	323	288.9	▼−34.1	−11.8%	13,001
`store_honey`	193	225.6	▲+32.6	+14.5%	10,150
`sorting_bottles_cans_and_paper`	312	342.1	▲+30.1	+8.8%	15,395
`sweeping_garage`	126	149.1	▲+23.1	+15.5%	6,709
`organizing_art_supplies`	228	206.5	▼−21.5	−10.4%	9,291
`clean_your_rusty_garden_tools`	484	505.3	▲+21.3	+4.2%	22,736
`installing_a_scanner`	122	142.0	▲+20.0	+14.1%	6,387
`scrubbing_bathroom_floor`	123	105.1	▼−17.9	−17.0%	4,730
`installing_smoke_detectors`	68	85.6	▲+17.6	+20.6%	3,853
`installing_a_modem`	63	80.4	▲+17.4	+21.6%	3,619
`cook_broccolini`	137	153.9	▲+16.9	+11.0%	6,926
`boxing_food_after_dinner`	240	223.3	▼−16.7	−7.5%	10,046
`cleaning_up_branches_and_twigs`	434	417.6	▼−16.4	−3.9%	18,790
`vacuuming_floors`	65	80.4	▲+15.4	+19.2%	3,617
`collecting_aluminum_cans`	325	340.1	▲+15.1	+4.4%	15,304
`clean_a_keyboard`	142	129.2	▼−12.8	−9.9%	5,814
`installing_a_fax_machine`	148	135.9	▼−12.1	−8.9%	6,115
`carrying_out_garden_furniture`	358	368.6	▲+10.6	+2.9%	16,588
`cook_a_brisket`	251	244.5	▼−6.5	−2.7%	11,002
`composting_waste`	115	121.3	▲+6.3	+5.2%	5,457
`make_cabinet_doors`	120	114.8	▼−5.2	−4.5%	5,163
`polishing_shoes`	363	357.9	▼−5.1	−1.4%	16,106
`packing_meal_for_delivery`	300	295.3	▼−4.7	−1.6%	13,289
`clean_up_broken_glass`	286	289.7	▲+3.7	+1.3%	13,037
`turning_out_all_lights_before_sleep`	337	333.5	▼−3.5	−1.0%	15,009
`store_produce`	316	317.4	▲+1.4	+0.4%	14,283
`setting_mousetraps`	339	339.9	●—	—	15,294
`unloading_the_car`	378	377.4	●—	—	16,985
`halve_an_egg`	212	212.6	●—	—	9,565
`putting_shoes_on_rack`	258	257.5	●—	—	11,589
`clean_boxing_gloves`	275	274.5	●—	—	12,353
`cleaning_up_plates_and_food`	457	456.5	●—	—	20,544
`cook_cabbage`	471	471.5	●—	—	21,217
`moving_boxes_to_storage`	487	486.5	●—	—	21,894
`collecting_childrens_toys`	640	639.5	●—	—	28,779
`hanging_pictures`	80	79.6	●—	—	3,581
`attach_a_camera_to_a_tripod`	130	130.4	●—	—	5,867
`picking_up_trash`	176	175.6	●—	—	7,901
`bringing_water`	315	314.6	●—	—	14,157
`outfit_a_basic_toolbox`	355	354.6	●—	—	15,956
`chopping_wood`	358	358.4	●—	—	16,127
`clean_a_patio`	402	402.4	●—	—	18,106
`clearing_food_from_table_into_fridge`	436	435.6	●—	—	19,602
`putting_away_Halloween_decorations`	460	459.6	●—	—	20,682
`getting_organized_for_work`	522	522.4	●—	—	23,506
`make_pizza`	640	639.6	●—	—	28,780
`boxing_books_up_for_storage`	808	807.6	●—	—	36,341
`chop_an_onion`	213	213.3	●—	—	9,599
`wash_a_baseball_cap`	278	278.3	●—	—	12,524
`putting_up_Christmas_decorations_inside`	457	457.3	●—	—	20,578
`turning_on_radio`	72	71.7	●—	—	3,224
`picking_up_toys`	630	629.7	●—	—	28,335
`storing_food`	662	662.3	●—	—	29,803
`assembling_gift_baskets`	869	868.7	●—	—	39,090
`canning_food`	766	765.8	●—	—	34,463
`preparing_lunch_box`	275	274.8	●—	—	12,367
`spraying_fruit_trees`	278	278.2	●—	—	12,518
`cook_hot_dogs`	305	304.8	●—	—	13,717
`sorting_vegetables`	397	396.8	●—	—	17,855
`freeze_pies`	415	415.2	●—	—	18,682
`bringing_in_wood`	451	451.2	●—	—	20,303
`carrying_in_groceries`	476	475.8	●—	—	21,412
`slicing_vegetables`	495	494.8	●—	—	22,267
`rearranging_kitchen_furniture`	298	298.1	●—	—	13,414
`setting_the_fire`	304	303.9	●—	—	13,677
`putting_dishes_away_after_cleaning`	365	365.1	●—	—	16,430
`tidying_bedroom`	368	367.9	●—	—	16,556
`wash_dog_toys`	374	374.1	●—	—	16,834
`can_meat`	395	394.9	●—	—	17,770
`sorting_household_items`	527	526.9	●—	—	23,711
`loading_the_car`	641	640.9	●—	—	28,839
`clean_up_your_desk`	714	713.9	●—	—	32,126
`make_microwave_popcorn`	108	107.9	●—	—	4,856
`clean_a_trumpet`	177	176.9	●—	—	7,960
`set_up_a_coffee_station_in_your_kitchen`	209	208.9	●—	—	9,399
`spraying_for_bugs`	216	216.0	●—	—	9,719
`hiding_Easter_eggs`	254	254.0	●—	—	11,429
`cook_bacon`	256	256.0	●—	—	11,519

The timeout column is the one that matters. The evaluator derives it from length rather than duration, so every episode receives the correct budget. This field damages planning, not scoring. The damage falls entirely on the 50 tasks with the least available information.

The metric

Where Q-score points actually live

Q is the fraction of goal predicates satisfied at episode end, averaged over tasks. The predicate structure of each task therefore is the scoring surface. That surface is shallower than it appears. 24 tasks are all-or-nothing.

fewest predicates

24 tasks; Q is 0 or 1

median predicates

one predicate = 33% of Q

most predicates

setup_a_bar_for_a_cocktail_party

tasks with quantifiers

true denominator set at runtime

Goal predicateBDDL state checked at episode end	Tasksof 100 that use it	Share
`inside`	54
`ontop`	40
`open`	23
`nextto`	16
`covered`	13
`real`	9
`toggled_on`	7
`cooked`	6
`attached`	4
`contains`	4

Two predicates carry the benchmark. inside and ontop appear in 54 and 40 tasks. A policy that places one object inside or on top of another reaches more of the scoring surface than any other single skill. covered appears in only 13 tasks, and the entire 2025 field scored zero on it.

Progress

Which predicates pay out, and how far anyone gets

Each of the 23 submissions (listed under Provenance) reports a Q for every individual rollout. Q is satisfied ÷ total. A value on an N-predicate task therefore decodes to the number of predicates that rollout achieved. 3,256 rollouts decode this way.

Getting some credit versus finishing

This chart covers the 21 tasks whose goal uses a single predicate type, so achievement attributes to that type without ambiguity. The bar runs from the completion rate to the any-credit rate. A long bar means policies start the job often and finish it rarely.

inside

14.2% → 66.7%

ontop

37.7% → 66.4%

cooked

64.0% → 66.0%

toggled_on

57.2% → 57.2%

real

23.2% → 55.4%

attached

18.2% → 18.2%

covered

9.9% → 17.9%

completed every predicatesatisfied at least onebar between the two = the partial-credit zone

How deep into the chain rollouts get

All scored tasks, grouped by how many goal predicates they have. Each row gives the share of rollouts that satisfied 0, 1, 2 and so on up to N. Darker means further along. The grey block on the left is complete failure.

N = 1

68.0%32.0%

918

N = 2

54.7%12.1%33.2%

404

N = 3

47.9%13.4%20.8%17.8%

365

N = 4

45.2%28.1%

922

N = 5

43.2%20.9%18.9%12.2%

148

N = 6

22.4%20.3%14.6%13.0%15.1%

192

N = 7

52.5%

160

N = 8

40.7%17.4%16.3%15.1%

0 predicatesfirstmiddleall Nright-hand figure is the rollout count

inside supplies the partial credit. ontop is where tasks finish. Both earn something on about two thirds of rollouts. But ontop completes on 37.7 % against 14.2 % for inside. The field can place objects on things. It can start to place them in things, and seldom finishes.

covered and attached rarely start. Fewer than 19 % of rollouts earn any credit on either. For attached, the any-credit rate and the completion rate are the same number. It never succeeds partly. These two are the particle and fastening states, and they set the floor of the benchmark.

What this cannot tell you. Q gives the count of satisfied predicates, never which ones. On a mixed-type task, 1 of 4 does not name which of the four. Satisfaction order needs per-frame replay. A replay harness of this kind already does this, recording satisfied and unsatisfied predicate indices per frame and timestamping each change. It has run on two tasks only. Extending it across the benchmark is a GPU job, and it is the only way to find which predicate comes first.

The same numbers as a table

Predicatesingle-type tasks only	Tasks	Rollouts	Any credit	Completed	Mean Q
`inside`	7	472	66.7%	14.2%	0.338
`ontop`	2	223	66.4%	37.7%	0.508
`cooked`	1	50	66.0%	64.0%	0.65
`toggled_on`	1	180	57.2%	57.2%	0.572
`real`	1	56	55.4%	23.2%	0.371
`attached`	2	132	18.2%	18.2%	0.182
`covered`	7	374	17.9%	9.9%	0.139

Difficulty

What predicts a hard task

Rank correlation against the 2025 field-mean Q, over the 50 carried tasks. Those are the only tasks with measured outcomes. All four correlations are negative. The question is which one is strongest.

Predictorknown before any rollout	Spearman ρvs 2025 field-mean Q	What it measures
end-effector displacement	-0.467	how far the hands travel
base distance	-0.384	how far the robot drives
demo length	-0.372	how long it takes
goal predicates	-0.215	how many conditions

Manipulation, not duration. The distance the hands must travel predicts difficulty better than the length of the task, and better still than the number of conditions. The tiebreaker metrics that the challenge already records are therefore the best available difficulty index. They are known in advance for all 100 tasks, including the 50 that nobody has attempted. Goal-predicate count is both the weakest predictor of the four and the least reliable, because the count is a floor on the 66 tasks whose goals are quantified. See Method.

Read every number here as a bound, not a measurement. The outcome variable is the field-mean Q, and most 2025 submissions are partial. A task that few teams reported carries mostly zeros whatever its difficulty, and the count of submissions reporting a task correlates with its field-mean Q at ρ 0.824, higher than any predictor in the table. Teams chose which tasks to run, and that choice is inside the outcome. The four predictors are still the best ordering available for the unrun 50, because they need no rollout at all.

The five tasks that never scored, on any rollout

Task	Secondsmean demo	PredicatesQ denominator	Goal predicate types
`clean_a_trumpet`	177	1	`covered`
`wash_dog_toys`	374	4	`covered`
`clean_a_patio`	402	1	`covered`
`cook_cabbage`	472	4	`contains` `real`
`make_pizza`	640	2	`ontop` `real`

Provenance

The 23 submissions this page leans on

Every 2025 figure here comes from these files, archived in the fork at docs/challenge_submissions/. Twenty-three submissions come from eighteen teams. Five teams submitted to both test sets. The privileged track could read ground-truth simulator state.

Team	Affiliation	Track	Test set	Qoverall, 50 tasks	Task SRfully solved
Robot Learning Collective	Independent	standard	public	0.2605	0.112
Robot Learning Collective	Independent	standard	hidden	0.2599	0.124
Comet	NVIDIA Research	standard	hidden	0.2514	0.114
SimpleAI Robot	Beijing Simple AI Technology Co Ltd	standard	public	0.1943	0.140
Comet	NVIDIA Research	standard	public	0.1830	0.144
The North Star	Huawei CRI EAI Team	standard	public	0.1702	0.128
SimpleAI Robot	Beijing Simple AI Technology Co Ltd	standard	hidden	0.1591	0.108
The North Star	Huawei CRI EAI Team	standard	hidden	0.1204	0.076
Embodied Intelligence	Independent	privileged	public	0.1110	0.062
Embodied Intelligence	Independent	privileged	hidden	0.0947	0.052
RAPPER	GIST	privileged	public	0.0750	0.052
tobi	Alzonova	standard	public	0.0717	0.036
MR	MR	privileged	public	0.0512	0.034
RACΞL	CMU	standard	public	0.0140	0.014
Ahri+EFFL+MLV	Postech	standard	public	0.0100	0.010
Merlin Labs	Independent	standard	public	0.0090	0.006
LYQRobotics	Independent	standard	public	0.0080	0.008
ACT	Xiamen University	standard	public	0.0037	0.002
StarVLA	Independent	standard	public	0.0019	0.000
Cloud-Data	Cloud Data Technology Co Ltd	standard	public	0.0000	0.000
EntropyMaximum	Independent	standard	public	0.0000	0.000
Magikid	Magikid	standard	public	0.0000	0.000
RobotSimArk	1	standard	public	0.0000	0.000

Most of these submissions are partial. Only 3 of the 23 report all 50 tasks; the median reports 11, and 18 report fewer than 20. Unreported tasks score 0 and stay in the denominator, which is the rule the challenge states. So the mean of 0.089 across all 50 tasks measures coverage as much as capability. Over the tasks each submission actually reports, the same mean is 0.236.

Discussion

Comet did not improve on the hidden set

Comet is the only 2025 team whose Q rose from the public set to the hidden set, by 37 %. The published files settle why. Its public submission reports rollouts for 19 of the 50 tasks. Its hidden submission reports all 50. A task with no rollouts scores 0 and stays in the denominator, so the two numbers are averages over different amounts of the benchmark.

Team	Tasks reportedpublic → hidden		Q-scorepublic → hidden
Comet	19	50	0.1830	0.2514	+37%
Robot Learning Collective	50	50	0.2605	0.2599	-0%
SimpleAI Robot	15	15	0.1943	0.1591	-18%
The North Star	13	13	0.1702	0.1204	-29%
Embodied Intelligence	11	10	0.1110	0.0947	-15%

A missing task is not a task that scored zero. Both appear as 0.000 in per_task_scores, so the distinction is only visible in per_rollout_scores. A task that was run and failed keeps its ten rollout records, each holding 0.0. Robot Learning Collective's public file carries rollouts for all 50 tasks including its 9 zeros, and Comet's own hidden file carries rollouts for all 50 including its 8 zeros. The 31 tasks absent from Comet's public file carry no rollouts at all.

The same 19 tasks, measured on both sets

Cometthe 19 tasks both submissions report	Public	Hidden	Change
Q-score	0.4816	0.3978	-17%
Task success rate	0.3789	0.2789	-26%
Tasks improved / fell / matched	4 improved, 14 fell, 1 matched

Comet fell on the hidden set, like every other team. On the tasks both submissions report, Q drops 17 % and the task success rate drops 26 %. That sits between SimpleAI Robot at −18 % and Embodied Intelligence at −15 %, measured the same way on each team's own reported set. Comet is mid-field, not an exception. The 37 % rise is the arithmetic of 31 unreported tasks entering the average, and those 31 average 0.1617 on the hidden set.

The two tasks cited as evidence of new partial credit were never reported on the public set. An earlier version of this page read collecting_childrens_toys 0.000→0.583 and storing_food 0.000→0.400 as a policy gaining partial credit. Both are in the unreported 31, and so are 23 of the 27 tasks that appear to have improved. Only four improved among the tasks reported on both sets. The rise is coverage, not behaviour.

For anyone using the released checkpoints, 0.1830 is not a policy-quality figure. It is 19 tasks of score divided by 50. Over the tasks that submission actually reports, the public figure is 0.4816. A reproduction that runs all 50 public tasks is measuring something neither number measures, so comparing it against 0.1830 or against the hidden-set 0.2514 will mislead in opposite directions. Robot Learning Collective took the top hidden-set score at 0.2599, from a submission that reports all 50 tasks on both sets.

Method

How the predicates were counted, and why the count is a floor

Goal predicates were counted by parsing the (:goal …) block of each problem0.bddl and taking the leaf predicates. For a goal like turning_on_radio's, that is exact:

(:goal (and (toggled_on ?radio_receiver.n.01_1)))

One predicate written, one predicate scored. But 66 of the 100 goal blocks contain a quantifier. A quantifier is a template rather than a predicate. picking_up_trash reads:

(:goal (and
  (forall (?can__of__soda.n.01 - can__of__soda.n.01)
    (inside ?can__of__soda.n.01 ?ashcan.n.01_1))))

That is every can of soda is inside the ashcan. The parse counts one. The :objects block in the same file declares three cans. At runtime goal_status therefore holds three ground predicates, and binning two of them scores 0.67. That is partial credit which the written block does not show. This is ordinary BDDL semantics rather than a defect, which is why it appears here and not in the findings.

Two consequences for the numbers on this page. Every predicate count marked + is a floor. The denominator can also differ between instances of the same task, because each of the 40 test instances uses a different scene layout. One instance may hold three cans and another five, scored out of three and out of five.

The fix, if the numbers must be exact. Load each task instance in the simulator and read goal_status. That is about 100 scene loads at 150 to 300 s each, so 4 to 8 hours on one GPU. It would replace every + with a measured number, and allow the predicate correlation to be computed on correct counts.

Reference

All 100 tasks

Sorted shortest first. Every column states what it holds and where it comes from. Read preds against Q 2025. The first gives how finely a task can pay out. The second gives whether anyone collected.

filter show

Taskid used by `--task-name`	Secmean demo, `length`÷30 Hz	PredsQ denominator; `+` expands at runtime	Base mmetres the base drove	EEF mmetres both hands moved	Q 2025field mean; blank = never run	Spreadlongest demo ÷ shortest	Demoorganizers’ recording
`turning_on_radio`	72	1	6	7	0.448	4.5	watch ›
`hanging_pictures`	80	1+	7	6	0.057	8.4	watch ›
`vacuuming_floors`	80	1	5	6	—	3.8	watch ›
`installing_a_modem`	80	4+	3	8	—	3.2	watch ›
`installing_smoke_detectors`	86	1	9	10	—	2.9	watch ›
`scrubbing_bathroom_floor`	105	1	5	16	—	5.3	watch ›
`make_microwave_popcorn`	108	2	9	12	0.252	3.1	watch ›
`make_cabinet_doors`	115	1	4	10	—	3.4	watch ›
`composting_waste`	121	2	6	15	—	2.0	watch ›
`clean_a_keyboard`	129	1	4	13	—	6.3	watch ›
`attach_a_camera_to_a_tripod`	130	1	11	14	0.048	3.8	watch ›
`installing_a_fax_machine`	136	2+	10	13	—	4.2	watch ›
`installing_a_scanner`	142	2	6	15	—	6.8	watch ›
`sweeping_garage`	149	2	19	9	—	2.7	watch ›
`make_rose_centerpieces`	151	4	9	20	—	2.6	watch ›
`cook_broccolini`	154	5+	11	12	—	3.0	watch ›
`picking_up_trash`	176	1+	16	20	0.354	7.1	watch ›
`clean_a_trumpet`	177	1	12	19	0.000	2.5	watch ›
`organizing_art_supplies`	206	5	12	25	—	2.5	watch ›
`set_up_a_coffee_station_in_your_kitchen`	209	6	17	20	0.044	3.8	watch ›
`halve_an_egg`	213	5	14	32	—	2.6	watch ›
`chop_an_onion`	213	4	17	17	0.039	3.2	watch ›
`spraying_for_bugs`	216	1+	22	19	0.065	2.9	watch ›
`boxing_food_after_dinner`	223	5+	16	29	—	2.1	watch ›
`store_honey`	226	1	12	23	—	3.5	watch ›
`cook_a_brisket`	244	3	14	28	—	2.6	watch ›
`hiding_Easter_eggs`	254	3+	22	23	0.131	4.9	watch ›
`cook_bacon`	256	2+	21	20	0.120	1.9	watch ›
`store_batteries`	257	1+	14	29	—	2.4	watch ›
`putting_shoes_on_rack`	258	6+	27	19	0.299	3.4	watch ›
`sorting_books_on_shelf`	258	10+	10	24	—	2.5	watch ›
`thawing_frozen_food`	274	9+	18	30	—	3.0	watch ›
`clean_boxing_gloves`	274	1+	29	22	0.020	3.5	watch ›
`preparing_lunch_box`	275	5+	21	31	0.051	3.2	watch ›
`spraying_fruit_trees`	278	2	24	29	0.059	2.0	watch ›
`wash_a_baseball_cap`	278	1+	29	22	0.083	1.9	watch ›
`cook_a_frozen_pie`	289	2	19	34	—	2.5	watch ›
`clean_up_broken_glass`	290	1+	29	28	—	3.1	watch ›
`packing_meal_for_delivery`	295	2+	20	38	—	1.9	watch ›
`rearranging_kitchen_furniture`	298	4+	26	31	0.064	3.8	watch ›
`setting_the_fire`	304	5+	32	24	0.043	2.7	watch ›
`cook_hot_dogs`	305	1+	26	31	0.141	3.4	watch ›
`dispose_of_glass`	306	1+	18	32	—	2.4	watch ›
`bringing_water`	315	2+	26	20	0.206	2.6	watch ›
`store_produce`	317	2+	26	42	—	2.4	watch ›
`re_shelving_library_books`	318	1+	20	36	—	2.9	watch ›
`make_gift_bags_for_baby_showers`	322	3+	17	40	—	2.7	watch ›
`turning_out_all_lights_before_sleep`	334	2+	40	34	—	1.9	watch ›
`setting_mousetraps`	340	3+	32	26	0.275	4.3	watch ›
`collecting_aluminum_cans`	340	1+	18	42	—	2.6	watch ›
`sorting_bottles_cans_and_paper`	342	12+	27	46	—	2.9	watch ›
`outfit_a_basic_toolbox`	355	7	32	38	0.023	3.3	watch ›
`polishing_shoes`	358	6+	17	41	—	2.2	watch ›
`chopping_wood`	358	8	38	35	0.090	3.3	watch ›
`putting_dishes_away_after_cleaning`	365	2+	33	60	0.049	2.1	watch ›
`tidying_bedroom`	368	3+	33	25	0.149	5.4	watch ›
`carrying_out_garden_furniture`	369	2+	61	25	—	1.6	watch ›
`bringing_paper_to_recycling`	374	3	35	49	—	2.0	watch ›
`wash_dog_toys`	374	4+	47	32	0.000	2.9	watch ›
`put_together_a_basic_pruning_kit`	376	4	19	37	—	2.4	watch ›
`putting_away_toys`	377	1+	35	57	—	2.5	watch ›
`unloading_the_car`	377	2+	34	39	—	2.3	watch ›
`can_meat`	395	4+	32	54	0.001	2.1	watch ›
`sorting_vegetables`	397	5+	34	69	0.056	3.1	watch ›
`clean_a_patio`	402	1	37	23	0.000	2.6	watch ›
`putting_dirty_dishes_in_sink`	405	2+	57	44	—	1.7	watch ›
`freeze_pies`	415	4+	35	58	0.014	3.1	watch ›
`cleaning_up_branches_and_twigs`	418	2+	41	50	—	2.2	watch ›
`tidying_living_room`	419	4+	21	37	—	2.6	watch ›
`freeze_fruit`	421	4+	25	67	—	2.2	watch ›
`rearrange_your_room`	428	2+	24	37	—	2.1	watch ›
`tidying_bathroom`	433	4	19	40	—	2.4	watch ›
`clearing_food_from_table_into_fridge`	436	4+	32	33	0.013	1.9	watch ›
`setup_a_bar_for_a_cocktail_party`	447	12+	23	50	—	2.0	watch ›
`bringing_in_wood`	451	1+	36	32	0.110	2.1	watch ›
`cleaning_up_plates_and_food`	456	4+	36	34	0.094	2.4	watch ›
`putting_up_Christmas_decorations_inside`	457	7+	41	39	0.086	2.0	watch ›
`putting_away_Halloween_decorations`	460	4+	47	40	0.148	127.7	watch ›
`cook_cabbage`	472	4	39	44	0.000	2.6	watch ›
`carrying_in_groceries`	476	4+	37	40	0.050	1.8	watch ›
`laying_tile_floors`	478	5+	65	44	—	1.7	watch ›
`dispose_of_batteries`	481	2+	47	33	—	3.4	watch ›
`organizing_school_stuff`	486	6	26	52	—	2.8	watch ›
`moving_boxes_to_storage`	486	4	37	19	0.383	2.1	watch ›
`slicing_vegetables`	495	7+	41	40	0.030	2.9	watch ›
`clean_your_rusty_garden_tools`	505	5	69	48	—	3.0	watch ›
`cook_brussels_sprouts`	522	5+	35	63	—	2.4	watch ›
`getting_organized_for_work`	522	10	47	44	0.003	2.9	watch ›
`sorting_household_items`	527	7+	51	54	0.019	2.8	watch ›
`stacking_wood`	531	6+	86	43	—	3.3	watch ›
`setting_the_table`	594	4+	56	85	—	2.0	watch ›
`picking_up_toys`	630	3+	47	42	0.041	4.2	watch ›
`collecting_childrens_toys`	640	4+	58	50	0.128	2.9	watch ›
`make_pizza`	640	2	57	78	0.000	2.6	watch ›
`loading_the_car`	641	4	48	41	0.036	2.4	watch ›
`storing_food`	662	4+	56	58	0.052	1.6	watch ›
`clean_up_your_desk`	714	8+	60	74	0.015	2.3	watch ›
`canning_food`	766	9+	64	92	0.001	2.5	watch ›
`boxing_books_up_for_storage`	808	1+	72	50	0.011	2.4	watch ›
`assembling_gift_baskets`	869	4+	80	79	0.051	2.4	watch ›