ALFWorld
What it measures
Section titled “What it measures”TextWorld household tasks: the agent is told “put a clean mug in the cabinet” and has to act through a text interface — go to, take, heat, put — until the game says the goal is reached. Solved means the goal was reached within 50 actions; there is no partial credit per task.
Study on 100 ids from the official train split; each pool below is scored once per checkpoint. valid_unseen is ALFWorld’s own out-of-distribution evaluation split; valid_seen and valid_train are its two other evaluation splits.
Results
Section titled “Results”Last export 2026-10-19T17:07:34Z · 8 methods · 171 published points, each recomputable from its per-task verdicts.
train · 100 tasks · H0 0.530 · study split
Section titled “train · 100 tasks · H0 0.530 · study split”Score — study split, 100 tasks per round
| method | r0 (H0) | r1 | r2 | r3 | r4 | r5 | r6 | r7 | r8 | r9 | r10 | record |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
sica_ci | 0.530 | 0.890 | 0.930 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | ↗ |
ahe | 0.530 | 0.920 | 1.000 | 1.000 | — | — | — | — | — | — | — | ↗ |
dgm | 0.530 | 0.820 | 0.890 | 0.940 | 0.950 | 0.950 | — | — | — | — | — | ↗ |
harnessx | 0.530 | 0.710 | 0.790 | 0.910 | 0.970 | 0.970 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | ↗ |
hyperagents | 0.530 | 0.740 | 0.900 | 0.990 | — | — | — | — | — | — | — | ↗ |
tthe | 0.530 | 0.650 | 0.670 | 0.700 | 0.700 | 0.740 | 0.740 | 0.840 | 0.760 | 0.830 | 0.930 | ↗ |
dream_rsi | 0.530 | 0.760 | 0.810 | 0.990 | 1.000 | 1.000 | 1.000 | 0.600 | — | — | — | ↗ |
meta_harness | 0.530 | 0.250 | 0.610 | 0.490 | 0.610 | 0.670 | 0.160 | 0.690 | 0.700 | 0.660 | — | ↗ |
The r0 (H0) column is one run — the same harness on the same tasks — not a constant: its record holds the trajectories and the per-task verdicts.
Cost — task-agent: millions of tokens the harness spent acting, per round
| method | r1 | r2 | r3 | r4 | r5 | r6 | r7 | r8 | r9 | r10 | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| in | out | in | out | in | out | in | out | in | out | in | out | in | out | in | out | in | out | in | out | |
sica_ci | 1.18 | 0.01 | 0.13 | 0.00 | reuse | — | reuse | — | reuse | — | reuse | — | reuse | — | reuse | — | reuse | — | reuse | — |
ahe | 17.64 | 0.06 | 3.56 | 0.03 | 3.56 | 0.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
dgm | 15.93 | 0.06 | 13.31 | 0.05 | 9.79 | 0.05 | 10.57 | 0.04 | 3.71 | 0.02 | — | — | — | — | — | — | — | — | — | — |
harnessx | 23.05 | 0.08 | 20.15 | 0.07 | 12.83 | 0.05 | 9.58 | 0.05 | 9.04 | 0.04 | 2.37 | 0.02 | 2.35 | 0.02 | 2.18 | 0.02 | 2.17 | 0.02 | 2.18 | 0.02 |
hyperagents | 19.13 | 0.07 | 19.90 | 0.09 | 0.28 | 0.00 | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
tthe | 14.78 | 0.06 | 17.51 | 0.07 | 22.68 | 0.08 | 12.84 | 0.07 | 15.58 | 0.07 | 15.53 | 0.07 | 15.43 | 0.07 | 19.75 | 0.07 | 16.09 | 0.07 | 13.63 | 0.06 |
dream_rsi | 19.50 | 0.07 | 18.04 | 0.06 | 0.04 | 0.00 | reuse | — | reuse | — | reuse | — | 18.19 | 0.07 | — | — | — | — | — | — |
meta_harness | 11.60 | 0.06 | 12.67 | 0.05 | 6.03 | 0.05 | 12.95 | 0.06 | 27.72 | 0.10 | 5.81 | 0.11 | 26.52 | 0.10 | 20.75 | 0.08 | 27.96 | 0.10 | — | — |
Cost — improver: millions of tokens the method spent editing, per round
| method | r1 | r2 | r3 | r4 | r5 | r6 | r7 | r8 | r9 | r10 | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| in | cache | out | in | cache | out | in | cache | out | in | cache | out | in | cache | out | in | cache | out | in | cache | out | in | cache | out | in | cache | out | in | cache | out | |
sica_ci | 0.04 | 8.7 | 0.08 | 0.04 | 11.3 | 0.07 | 0.11 | 8.2 | 0.07 | 0.05 | 11.2 | 0.08 | 0.05 | 8.4 | 0.07 | 0.02 | 14.2 | 0.07 | 0.05 | 17.4 | 0.08 | 0.07 | 20.4 | 0.08 | 0.08 | 25.0 | 0.08 | 0.04 | 13.7 | 0.08 |
ahe | 0.10 | 29.3 | 0.18 | 0.09 | 24.8 | 0.16 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
dgm | 0.15 | 19.0 | 0.15 | — | — | — | 0.18 | 51.2 | 0.27 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
harnessx | 0.03 | 9.5 | 0.08 | 0.04 | 10.4 | 0.08 | 0.13 | 9.5 | 0.07 | 0.06 | 14.1 | 0.07 | 0.03 | 11.1 | 0.07 | 0.05 | 14.2 | 0.08 | 0.12 | 8.7 | 0.07 | 0.05 | 11.0 | 0.07 | 0.09 | 15.8 | 0.08 | 0.16 | 12.1 | 0.07 |
hyperagents | 0.11 | 29.7 | 0.23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |
tthe | 0.08 | 10.4 | 0.07 | 0.04 | 9.5 | 0.08 | 0.12 | 6.1 | 0.05 | 0.06 | 17.2 | 0.07 | 0.06 | 10.9 | 0.08 | 0.05 | 10.9 | 0.07 | 0.07 | 15.5 | 0.08 | 0.04 | 10.9 | 0.08 | 0.10 | 11.8 | 0.08 | 0.04 | 19.4 | 0.09 |
dream_rsi | 0.03 | 11.1 | 0.08 | 0.06 | 14.1 | 0.09 | 0.04 | 12.6 | 0.07 | 0.13 | 9.7 | 0.06 | 0.03 | 10.9 | 0.07 | — | — | — | — | — | — | 0.02 | 10.1 | 0.09 | — | — | — | — | — | — |
meta_harness | 0.15 | 5.0 | 0.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0.21 | 17.8 | 0.18 |
valid_unseen · 134 tasks · H0 0.403 · 57 points
Section titled “valid_unseen · 134 tasks · H0 0.403 · 57 points”Score — valid_unseen, 134 tasks per point
| method | r0 (H0) | r1 | r2 | r3 | r4 | r5 | r6 | r7 | r8 | r9 | r10 | record |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
sica_ci | 0.403 | 0.970 | 0.896 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000† | ↗ |
ahe | 0.403 | 0.993 | 1.000 | 1.000† | — | — | — | — | — | — | — | ↗ |
dgm | 0.403 | 0.851 | 0.843 | 0.963 | 0.978 | 0.985† | — | — | — | — | — | ↗ |
harnessx | 0.403 | 0.590 | 0.701 | 0.940 | 0.970 | 0.970 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000† | ↗ |
hyperagents | 0.403 | 0.687 | 1.000 | 1.000† | — | — | — | — | — | — | — | ↗ |
tthe | 0.403 | 0.560 | 0.627 | 0.619 | 0.582 | 0.679 | 0.716 | 0.866 | 0.776 | 0.910 | 0.933† | ↗ |
dream_rsi | 0.403 | 0.619 | 0.791 | 1.000 | 1.000 | 1.000 | 1.000† | 0.478 | — | — | — | ↗ |
meta_harness | 0.403 | 0.231 | 0.478 | 0.366 | 0.485 | 0.597 | 0.097 | 0.590 | 0.545† | 0.590 | — | ↗ |
Cost — task-agent: millions of tokens per measured point (an evaluation point runs no improver, so there is no improver column here)
| method | r1 (M) | r2 (M) | r3 (M) | r4 (M) | r5 (M) | r6 (M) | r7 (M) | r8 (M) | r9 (M) | r10 (M) |
|---|---|---|---|---|---|---|---|---|---|---|
sica_ci | 0.8 | 2.2 | reuse | reuse | reuse | reuse | reuse | reuse | reuse | reuse |
ahe | 8.6 | 4.2 | 4.2 | — | — | — | — | — | — | — |
dgm | 19.9 | 19.8 | 10.2 | 13.2 | 2.3 | — | — | — | — | — |
harnessx | 33.6 | 31.1 | 13.4 | 11.3 | 11.7 | 4.2 | 4.0 | 3.9 | 3.8 | 3.8 |
hyperagents | 28.8 | 12.4 | 0.2 | — | — | — | — | — | — | — |
tthe | 18.0 | 26.4 | 35.1 | 13.8 | 18.2 | 18.0 | 19.9 | 28.0 | 19.3 | 16.4 |
dream_rsi | 29.6 | 22.1 | reuse | reuse | reuse | reuse | 24.8 | — | — | — |
meta_harness | 13.9 | 16.4 | 8.3 | 16.6 | 35.7 | 7.4 | 36.0 | 29.0 | 37.3 | — |
valid_seen · 140 tasks · H0 0.414 · 57 points
Section titled “valid_seen · 140 tasks · H0 0.414 · 57 points”Score — valid_seen, 140 tasks per point
| method | r0 (H0) | r1 | r2 | r3 | r4 | r5 | r6 | r7 | r8 | r9 | r10 | record |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
sica_ci | 0.414 | 0.936 | 0.886 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | ↗ |
ahe | 0.414 | 0.893 | 1.000 | 1.000† | — | — | — | — | — | — | — | ↗ |
dgm | 0.414 | 0.814 | 0.829 | 0.929 | 0.957 | 0.971 | — | — | — | — | — | ↗ |
harnessx | 0.414 | 0.557 | 0.643 | 0.850 | 0.971 | 0.971 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | ↗ |
hyperagents | 0.414 | 0.629 | 0.914 | 1.000† | — | — | — | — | — | — | — | ↗ |
tthe | 0.414 | 0.593 | 0.600 | 0.679 | 0.671 | 0.707 | 0.700 | 0.821 | 0.800 | 0.871 | 0.914 | ↗ |
dream_rsi | 0.414 | 0.714 | 0.864 | 1.000 | 1.000 | 1.000 | 1.000 | 0.550 | — | — | — | ↗ |
meta_harness | 0.414 | 0.136 | 0.471 | 0.421 | 0.507 | 0.593 | 0.186 | 0.643 | 0.579 | 0.657 | — | ↗ |
Cost — task-agent: millions of tokens per measured point (an evaluation point runs no improver, so there is no improver column here)
| method | r1 (M) | r2 (M) | r3 (M) | r4 (M) | r5 (M) | r6 (M) | r7 (M) | r8 (M) | r9 (M) | r10 (M) |
|---|---|---|---|---|---|---|---|---|---|---|
sica_ci | 1.0 | 0.7 | reuse | reuse | reuse | reuse | reuse | reuse | reuse | reuse |
ahe | 25.9 | 3.4 | 3.4 | — | — | — | — | — | — | — |
dgm | 22.1 | 22.1 | 13.4 | 15.0 | 1.8 | — | — | — | — | — |
harnessx | 38.1 | 34.6 | 20.6 | 13.5 | 13.1 | 3.4 | 3.4 | 3.3 | 3.2 | 3.3 |
hyperagents | 31.4 | 25.3 | 0.4 | — | — | — | — | — | — | — |
tthe | 21.5 | 29.6 | 33.0 | 18.9 | 22.7 | 22.5 | 22.9 | 25.8 | 21.6 | 18.2 |
dream_rsi | 28.1 | 20.4 | 0.0 | reuse | reuse | reuse | 25.1 | — | — | — |
meta_harness | 16.6 | 21.5 | 8.2 | 20.0 | 41.7 | 7.3 | 38.6 | 33.4 | 41.2 | — |
valid_train · 200 tasks · H0 0.525 · 57 points
Section titled “valid_train · 200 tasks · H0 0.525 · 57 points”Score — valid_train, 200 tasks per point
| method | r0 (H0) | r1 | r2 | r3 | r4 | r5 | r6 | r7 | r8 | r9 | r10 | record |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
sica_ci | 0.525 | 0.940 | 0.930 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | ↗ |
ahe | 0.525 | 0.905 | 1.000 | 1.000 | — | — | — | — | — | — | — | ↗ |
dgm | 0.525 | 0.790 | 0.840 | 0.960 | 0.930 | 0.950 | — | — | — | — | — | ↗ |
harnessx | 0.525 | 0.620 | 0.655 | 0.845 | 0.955 | 0.965 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | ↗ |
hyperagents | 0.525 | 0.705 | 0.940 | 0.995 | — | — | — | — | — | — | — | ↗ |
tthe | 0.525 | 0.590 | 0.630 | 0.680 | 0.630 | 0.645 | 0.680 | 0.755 | 0.735 | 0.810 | 0.890 | ↗ |
dream_rsi | 0.525 | 0.660 | 0.815 | 0.990 | 1.000 | 1.000 | 1.000 | 0.480 | — | — | — | ↗ |
meta_harness | 0.525 | 0.190 | 0.580 | 0.485 | 0.585 | 0.645 | 0.155 | 0.665 | 0.635 | 0.610 | — | ↗ |
Cost — task-agent: millions of tokens per measured point (an evaluation point runs no improver, so there is no improver column here)
| method | r1 (M) | r2 (M) | r3 (M) | r4 (M) | r5 (M) | r6 (M) | r7 (M) | r8 (M) | r9 (M) | r10 (M) |
|---|---|---|---|---|---|---|---|---|---|---|
sica_ci | 1.9 | 0.8 | reuse | reuse | reuse | reuse | reuse | reuse | reuse | reuse |
ahe | 32.3 | 6.3 | 6.3 | — | — | — | — | — | — | — |
dgm | 32.2 | 30.7 | 17.6 | 22.4 | 5.9 | — | — | — | — | — |
harnessx | 47.4 | 46.2 | 27.8 | 19.4 | 19.3 | 6.5 | 6.6 | 6.2 | 6.1 | 6.1 |
hyperagents | 40.0 | 29.1 | 0.1 | — | — | — | — | — | — | — |
tthe | 32.5 | 42.6 | 45.6 | 24.6 | 32.0 | 30.9 | 33.8 | 40.2 | 31.0 | 26.7 |
dream_rsi | 41.3 | 30.4 | 0.1 | reuse | reuse | reuse | 38.8 | — | — | — |
meta_harness | 24.1 | 28.2 | 11.7 | 28.3 | 53.5 | 10.7 | 52.9 | 43.9 | 58.2 | — |
Run it
Section titled “Run it”- study: 100 ids from
train(train100.csv), 10 rounds, whole set per round - measure: every saved round on each split. The budget is 48 machine-wide slots;
--jobsis how many of them one invocation may use. The checkpoint sweeps ran two shards at--jobs 24, which is what keeps the two cards balanced: one call-level balancer in front of both servers, so every model call goes to whichever card is free. - score: the environment’s own verifier; per-task verdicts are kept, so a score recomputes from
per_task
# studyHG_ALFWORLD_STUDY_IDS=train100.csv HG_ALFWORLD_POOL=valid_unseen \ python3 driver.py --harness loop --dataset alfworld --split declared --side train \ --rounds 10 --trials 1 --jobs 24 --improver dsh \ --method-entrypoint 'python3 methods/<method>/run.py' --run-id <id> --runs-root runs
# one saved checkpoint on one splitHG_ALFWORLD_STUDY_IDS=train100.csv HG_ALFWORLD_POOL=valid_unseen \ python3 driver.py --harness loop --dataset alfworld --split declared --side eval \ --from-run <id> --from-round <N> --trials 1 --jobs 24 --improver none \ --run-id <id>-ckpt-r<N>-valid_unseen --runs-root runsTrajectories
Section titled “Trajectories”Published on HuggingFace; until the repository is announced, ask in an issue.
Contribute
Section titled “Contribute”Bring your own setup — a different base harness, model or improver is welcome as long as it is declared and H0 is measured on it. Format and verification: Contributing.