ALFWorld

What it measures

TextWorld household tasks: the agent is told “put a clean mug in the cabinet” and has to act through a text interface — go to, take, heat, put — until the game says the goal is reached. Solved means the goal was reached within 50 actions; there is no partial credit per task.

Study on 100 ids from the official train split; each pool below is scored once per checkpoint. valid_unseen is ALFWorld’s own out-of-distribution evaluation split; valid_seen and valid_train are its two other evaluation splits.

Results

all splitstrain100 tasks · H0 0.530valid_train200 tasks · H0 0.525valid_seen140 tasks · H0 0.414valid_unseen134 tasks · H0 0.403

stackall stacks

reset

Last export 2026-10-23T13:37:42Z · 7 methods · 174 published points, each recomputable from its per-task verdicts.

train · 100 tasks · H0 0.530 · study split

Score — study split, 100 tasks per round

method	r0 (H0)	r1	r2	r3	r4	r5	r6	r7	r8	r9	r10	record
sica_ci	0.530	0.890	0.930	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	↗
ahe	0.530	0.920	1.000	1.000	—	—	—	—	—	—	—	↗
harnessx	0.530	0.710	0.790	0.910	0.970	0.970	1.000	1.000	1.000	1.000	1.000	↗
tthe	0.530	0.650	0.670	0.700	0.700	0.740	0.740	0.840	0.760	0.830	0.930	↗
dgm	0.530	0.870	0.850	0.570	0.770	0.960	0.950	0.680	0.990	1.000	1.000	↗
dream_rsi	0.530	0.530	0.970	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	↗
hyperagents	0.530	0.820	0.810	0.760	0.990	0.920	0.770	0.940	0.960	0.990	0.980	↗

The r0 (H0) column is one run — the same harness on the same tasks — not a constant: its record holds the trajectories and the per-task verdicts.

Cost — task-agent: millions of tokens the harness spent acting, per round

task-agent tokens per round — millions of tokens (M); in = uncached prompt tokens, out = generated tokens; 0.00 = the round made no model calls at all (the harness solved the tasks with its own solver, and the traces record every step without one reply record); n/r = the round ran but the record carries no counts

method	r1		r2		r3		r4		r5		r6		r7		r8		r9		r10
	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out
sica_ci	1.18	0.01	0.13	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
ahe	17.64	0.06	3.56	0.03	3.56	0.03	—	—	—	—	—	—	—	—	—	—	—	—	—	—
harnessx	23.05	0.08	20.15	0.07	12.83	0.05	9.58	0.05	9.04	0.04	2.37	0.02	2.35	0.02	2.18	0.02	2.17	0.02	2.18	0.02
tthe	14.78	0.06	17.51	0.07	22.68	0.08	12.84	0.07	15.58	0.07	15.53	0.07	15.43	0.07	19.75	0.07	16.09	0.07	13.63	0.06
dgm	15.83	0.05	18.73	0.06	12.67	0.06	0.70	0.00	6.44	0.03	5.96	0.03	19.70	0.07	2.63	0.03	4.47	0.03	4.05	0.03
dream_rsi	8.21	0.05	1.06	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
hyperagents	15.72	0.06	13.63	0.07	18.99	0.07	3.53	0.03	11.44	0.05	11.32	0.05	4.59	0.05	4.36	0.04	3.11	0.03	2.72	0.03

Cost — improver: millions of tokens the method spent editing, per round

improver tokens per round — millions of tokens (M); in = uncached prompt, cache = prompt read from cache, out = generated; ↳ = another child of the submission counted to its left: one improver call produced several candidates, so its spend appears once, not once per child; a dash means no improver usage is on record for that round – either the method edited nothing or the round’s usage was not recorded – never a zero

method	r1			r2			r3			r4			r5			r6			r7			r8			r9			r10
	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out
sica_ci	0.04	8.7	0.08	0.04	11.3	0.07	0.11	8.2	0.07	0.05	11.2	0.08	0.05	8.4	0.07	0.02	14.2	0.07	0.05	17.4	0.08	0.07	20.4	0.08	0.08	25.0	0.08	0.04	13.7	0.08
ahe	0.24	31.8	0.24	0.27	27.7	0.19	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—	—
harnessx	0.10	10.0	0.09	0.12	11.7	0.10	0.13	9.5	0.07	0.20	19.7	0.11	0.14	13.6	0.09	0.17	18.4	0.10	0.12	8.7	0.07	0.15	13.3	0.09	0.15	16.5	0.09	0.16	12.1	0.07
tthe	0.12	10.7	0.08	0.10	10.8	0.11	0.12	6.1	0.05	0.18	20.1	0.10	0.12	11.4	0.09	0.13	12.3	0.09	0.16	16.8	0.10	0.16	13.0	0.10	0.12	11.8	0.08	0.17	25.3	0.14
dgm	0.24	31.4	0.22	↳	↳	↳	↳	↳	↳	0.23	45.1	0.26	↳	↳	↳	↳	↳	↳	0.19	30.8	0.20	↳	↳	↳	↳	↳	↳	0.29	20.5	0.18
dream_rsi	0.16	25.7	0.18	0.15	15.9	0.12	0.13	6.8	0.07	0.13	11.6	0.10	0.11	9.8	0.06	0.05	2.2	0.03	0.05	1.4	0.03	0.11	13.9	0.09	0.09	4.1	0.05	0.05	1.4	0.03
hyperagents	0.21	41.9	0.24	↳	↳	↳	↳	↳	↳	0.15	39.5	0.25	↳	↳	↳	↳	↳	↳	0.20	36.4	0.26	↳	↳	↳	↳	↳	↳	0.27	25.3	0.17

valid_unseen · 134 tasks · H0 0.403 · 59 points

Score — valid_unseen, 134 tasks per point

method	r0 (H0)	r1	r2	r3	r4	r5	r6	r7	r8	r9	r10	record
sica_ci	0.403	0.970	0.896	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	↗
ahe	0.403	0.993	1.000	1.000	—	—	—	—	—	—	—	↗
harnessx	0.403	0.590	0.701	0.940	0.970	0.970	1.000	1.000	1.000	1.000	1.000	↗
tthe	0.403	0.560	0.627	0.619	0.582	0.679	0.716	0.866	0.776	0.910	0.918	↗
dgm	0.403	0.836	0.843	0.463	0.739	1.000	0.910	0.560	0.978	1.000	1.000	↗
dream_rsi	0.403	—	0.933	0.978	0.978	0.978	0.985	0.985	0.978	0.978	0.978	↗
hyperagents	0.403	0.851	0.799	0.649	1.000	1.000	0.873	0.993	—	—	—	↗

Cost — task-agent: millions of tokens per measured point (in / out; an evaluation point runs no improver, so there is no improver column here)

task-agent tokens per measured point — millions of tokens (M); in = uncached prompt tokens, out = generated tokens; 0.00 = the point made no model calls (the harness solved the tasks with its own solver); n/s = only a total was recorded for this point

method	r1		r2		r3		r4		r5		r6		r7		r8		r9		r10
	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out
sica_ci	0.80	0.01	2.19	0.01	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
ahe	8.51	0.05	4.20	0.03	4.20	0.03	—	—	—	—	—	—	—	—	—	—	—	—	—	—
harnessx	33.49	0.11	31.00	0.11	13.36	0.07	11.21	0.06	11.68	0.06	4.12	0.03	3.99	0.03	3.82	0.03	3.81	0.03	3.84	0.03
tthe	17.89	0.08	26.32	0.10	34.96	0.12	13.74	0.09	18.14	0.10	17.94	0.10	19.86	0.09	27.87	0.11	19.24	0.09	17.47	0.08
dgm	23.62	0.08	23.14	0.08	15.44	0.07	0.00	0.00	5.16	0.04	13.38	0.05	25.18	0.10	4.67	0.04	5.22	0.04	5.08	0.04
dream_rsi	—	—	1.29	0.01	0.23	0.00	0.31	0.00	0.22	0.00	0.16	0.00	0.16	0.00	0.20	0.00	0.22	0.00	0.22	0.00
hyperagents	20.16	0.08	19.84	0.10	31.01	0.11	3.40	0.04	10.32	0.05	11.11	0.06	4.78	0.05	—	—	—	—	—	—

valid_seen · 140 tasks · H0 0.414 · 58 points

Score — valid_seen, 140 tasks per point

method	r0 (H0)	r1	r2	r3	r4	r5	r6	r7	r8	r9	r10	record
sica_ci	0.414	0.936	0.886	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	↗
ahe	0.414	0.893	1.000	1.000	—	—	—	—	—	—	—	↗
harnessx	0.414	0.557	0.643	0.850	0.971	0.971	1.000	1.000	1.000	1.000	1.000	↗
tthe	0.414	0.593	0.600	0.679	0.671	0.707	0.700	0.821	0.800	0.871	0.914	↗
dgm	0.414	0.857	0.886	0.457	0.779	0.993	0.943	0.593	1.000	1.000	1.000	↗
dream_rsi	0.414	—	—	0.979	0.986	0.979	0.986	0.979	0.971	0.986	0.979	↗
hyperagents	0.414	0.857	0.793	0.707	0.993	0.921	0.829	0.936	—	—	—	↗

Cost — task-agent: millions of tokens per measured point (in / out; an evaluation point runs no improver, so there is no improver column here)

task-agent tokens per measured point — millions of tokens (M); in = uncached prompt tokens, out = generated tokens; 0.00 = the point made no model calls (the harness solved the tasks with its own solver); n/s = only a total was recorded for this point

method	r1		r2		r3		r4		r5		r6		r7		r8		r9		r10
	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out
sica_ci	0.97	0.01	0.73	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
ahe	25.84	0.09	3.32	0.03	3.32	0.03	—	—	—	—	—	—	—	—	—	—	—	—	—	—
harnessx	37.98	0.12	34.50	0.11	20.54	0.08	13.44	0.06	13.03	0.06	3.40	0.03	3.32	0.03	3.23	0.03	3.19	0.03	3.24	0.03
tthe	21.43	0.09	29.49	0.10	32.90	0.11	18.84	0.10	22.56	0.10	22.40	0.10	22.79	0.09	25.75	0.10	21.51	0.09	18.17	0.07
dgm	23.57	0.08	20.79	0.07	20.02	0.08	1.85	0.01	4.98	0.03	8.46	0.04	28.81	0.10	3.15	0.03	3.93	0.03	3.69	0.03
dream_rsi	—	—	—	—	0.07	0.00	0.06	0.00	0.07	0.00	0.06	0.00	0.07	0.00	0.24	0.00	0.06	0.00	0.11	0.00
hyperagents	20.34	0.08	19.21	0.10	28.88	0.10	3.76	0.04	16.24	0.07	12.98	0.06	5.80	0.06	—	—	—	—	—	—

valid_train · 200 tasks · H0 0.525 · 57 points

Score — valid_train, 200 tasks per point

method	r0 (H0)	r1	r2	r3	r4	r5	r6	r7	r8	r9	r10	record
sica_ci	0.525	0.940	0.930	1.000	1.000	1.000	1.000	1.000	1.000	1.000	1.000	↗
ahe	0.525	0.905	1.000	1.000	—	—	—	—	—	—	—	↗
harnessx	0.525	0.620	0.655	0.845	0.955	0.965	1.000	1.000	1.000	1.000	1.000	↗
tthe	0.525	0.590	0.630	0.680	0.630	0.645	0.680	0.755	0.735	0.810	0.890	↗
dgm	0.525	0.845	0.855	0.605	0.850	0.975	0.930	0.600	0.985	—	—	↗
dream_rsi	0.525	—	0.950	0.970	0.980	0.970	0.975	0.970	0.970	0.970	0.970	↗
hyperagents	0.525	0.835	0.775	0.690	0.975	0.935	0.775	0.970	—	—	—	↗

Cost — task-agent: millions of tokens per measured point (in / out; an evaluation point runs no improver, so there is no improver column here)

task-agent tokens per measured point — millions of tokens (M); in = uncached prompt tokens, out = generated tokens; 0.00 = the point made no model calls (the harness solved the tasks with its own solver); n/s = only a total was recorded for this point

method	r1		r2		r3		r4		r5		r6		r7		r8		r9		r10
	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out
sica_ci	1.90	0.02	0.77	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00	0.00
ahe	32.15	0.12	6.24	0.05	6.24	0.05	—	—	—	—	—	—	—	—	—	—	—	—	—	—
harnessx	47.25	0.16	46.01	0.16	27.68	0.11	19.27	0.09	19.22	0.09	6.44	0.05	6.59	0.05	6.11	0.05	6.04	0.05	6.09	0.05
tthe	32.41	0.13	42.45	0.15	45.40	0.16	24.52	0.13	31.88	0.15	30.74	0.15	33.67	0.14	39.99	0.16	30.91	0.13	26.60	0.11
dgm	30.38	0.11	31.62	0.11	24.73	0.11	1.37	0.01	10.95	0.06	16.64	0.07	41.65	0.15	5.91	0.06	—	—	—	—
dream_rsi	—	—	1.20	0.01	0.66	0.00	0.63	0.00	0.66	0.00	0.55	0.00	0.61	0.00	0.62	0.00	0.65	0.00	0.38	0.00
hyperagents	30.49	0.12	26.89	0.14	40.41	0.14	6.38	0.07	19.82	0.09	21.28	0.10	7.36	0.08	—	—	—	—	—	—

Run it

• study: 100 ids from train (train100.csv), 10 rounds, whole set per round
• measure: every saved round on each split. The budget is 48 machine-wide slots; --jobs is how many of them one invocation may use. The checkpoint sweeps ran two shards at --jobs 24, which is what keeps the two cards balanced: one call-level balancer in front of both servers, so every model call goes to whichever card is free.
• score: the environment’s own verifier; per-task verdicts are kept, so a score recomputes from per_task

# study
HG_ALFWORLD_STUDY_IDS=train100.csv HG_ALFWORLD_POOL=valid_unseen \
  python3 driver.py --harness loop --dataset alfworld --split declared --side train \
  --rounds 10 --trials 1 --jobs 24 --improver dsh \
  --method-entrypoint 'python3 methods/<method>/run.py' --run-id <id> --runs-root runs

# one saved checkpoint on one split
HG_ALFWORLD_STUDY_IDS=train100.csv HG_ALFWORLD_POOL=valid_unseen \
  python3 driver.py --harness loop --dataset alfworld --split declared --side eval \
  --from-run <id> --from-round <N> --trials 1 --jobs 24 --improver none \
  --run-id <id>-ckpt-r<N>-valid_unseen --runs-root runs

Trajectories

Published on HuggingFace; until the repository is announced, ask in an issue.

Contribute

Bring your own setup — a different base harness, model or improver is welcome as long as it is declared and H0 is measured on it. Format and verification: Contributing.