AppWorld

What it measures

A simulated world of apps — e-mail, calendar, contacts, a file system, shopping, a music player — behind REST-like APIs. A task is a natural-language goal that must be reached by calling those APIs, and it counts as solved only when the official evaluator finds the world state in the target state: a right-looking answer that never touched the world is a failure.

Study on 90 ids from the official train split, scored every round on the training tasks the method was allowed to learn from; each round is also scored on the official dev split (57 tasks), which never enters the search and is the pool a round is chosen on. test_normal (168) and test_challenge (417) are the eval side: measured once, on the round that was chosen.

Results

all splitstrain90 tasks · H0 0.011dev57 tasks · H0 0.000test_normal168 taskstest_challenge417 tasks

stackall stacks

reset

Last export 2026-10-23T13:37:35Z · 7 methods · 140 published validation points, each recomputable from its per-task verdicts.

train · 90 tasks · H0 0.011 · study split

Score — train, 90 tasks per round

method	r0 (H0)	r1	r2	r3	r4	r5	r6	r7	r8	r9	r10	r11	r12	r13	r14	r15	r16	r17	r18	r19	r20
sica_ci	0.011*	0.200	0.289	0.344	0.389	0.356	0.478	0.544	0.478	0.533	0.656	0.544	0.589	0.633	0.678	0.744	0.722	0.733	0.778	0.722	0.811
ahe	0.011*	0.111	0.133	0.278	0.411	0.489	0.489	0.511	0.533	0.000	0.567	0.533	0.511	0.456	0.489	0.467	0.489	0.456	0.556	0.500	0.556
dgm	0.011*	0.078	0.100	0.189	0.400	0.311	0.322	0.122	0.200	0.333	0.344	0.367	0.400	0.356	0.367	0.322	0.378	0.356	0.411	0.356	0.433
harnessx	0.011*	0.144	0.300	0.233	0.300	0.011	0.300	0.322	0.311	0.578	0.533	0.533	0.622	0.444	0.622	0.700	0.678	0.567	0.700	0.633	0.700
hyperagents	0.011*	0.156	0.189	0.189	0.222	0.244	0.278	0.000	0.144	0.200	0.178	0.322	0.433	0.467	0.533	0.122	0.122	0.556	0.633	0.378	0.667
tthe	0.011*	0.144	0.167	0.156	0.267	0.367	0.367	0.367	0.367	0.367	0.344	0.344	0.344	0.322	0.356	0.356	0.367	0.289	0.289	0.289	0.389
dream_rsi	0.011*	0.067	0.000	0.144	0.167	0.233	0.278	0.456	0.489	0.500	0.522	0.511	0.489	0.500	0.511	0.533	0.533	0.456	0.456	0.489	0.511

Cost — task-agent: millions of tokens the harness spent acting, per round

task-agent tokens per round — millions of tokens (M); in = uncached prompt tokens, out = generated tokens; 0.00 = the round made no model calls at all (the harness solved the tasks with its own solver, and the traces record every step without one reply record); n/r = the round ran but the record carries no counts

method	r1		r2		r3		r4		r5		r6		r7		r8		r9		r10		r11		r12		r13		r14		r15		r16		r17		r18		r19		r20
	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out
sica_ci	14.55	0.22	13.17	0.18	10.38	0.22	9.03	0.15	9.66	0.15	11.99	0.18	10.75	0.18	9.81	0.19	8.80	0.19	11.10	0.18	12.03	0.20	11.95	0.21	11.79	0.18	10.76	0.18	11.07	0.16	11.01	0.16	11.03	0.19	10.13	0.18	5.87	0.08	9.04	0.14
ahe	11.47	0.20	10.92	0.18	14.80	0.23	10.90	0.20	11.16	0.24	11.19	0.24	11.21	0.23	10.92	0.21	13.31	0.17	9.27	0.20	10.11	0.15	11.12	0.17	12.65	0.17	12.62	0.17	11.96	0.15	12.29	0.15	12.48	0.16	10.79	0.15	11.84	0.16	12.05	0.15
dgm	7.62	0.17	14.37	0.24	8.13	0.15	9.02	0.14	14.97	0.31	15.39	0.27	16.92	0.27	23.62	0.37	18.11	0.23	16.92	0.23	18.10	0.27	14.67	0.23	16.67	0.19	14.67	0.20	10.04	0.15	16.54	0.21	17.03	0.20	8.98	0.17	12.87	0.21	11.64	0.20
harnessx	9.51	0.15	15.83	0.25	12.53	0.20	15.83	0.25	7.44	0.07	15.83	0.25	12.52	0.21	11.72	0.17	10.35	0.20	12.48	0.20	11.89	0.19	10.66	0.23	10.36	0.21	11.06	0.21	10.62	0.22	10.58	0.22	12.95	0.24	11.30	0.23	9.88	0.22	11.30	0.23
hyperagents	11.96	0.29	14.64	0.34	10.08	0.19	15.65	0.28	15.16	0.26	7.77	0.22	5.01	0.21	8.75	0.23	21.52	0.21	9.15	0.19	9.87	0.11	10.67	0.17	7.91	0.13	6.66	0.14	19.18	0.35	19.22	0.35	7.98	0.20	7.87	0.19	12.84	0.12	7.98	0.17
tthe	14.34	0.21	13.93	0.22	14.55	0.23	14.41	0.21	12.87	0.21	12.87	0.21	12.87	0.21	12.87	0.21	12.87	0.21	11.83	0.20	11.74	0.21	11.74	0.21	14.32	0.21	14.90	0.23	14.90	0.23	12.78	0.19	13.24	0.23	13.24	0.23	13.24	0.23	13.29	0.24
dream_rsi	9.31	0.13	3.88	0.05	11.94	0.13	10.79	0.14	12.55	0.12	12.86	0.12	15.85	0.16	10.46	0.14	10.50	0.14	10.84	0.14	12.04	0.17	11.50	0.15	11.70	0.16	11.64	0.13	11.08	0.14	11.27	0.16	11.20	0.14	12.18	0.19	12.53	0.15	12.27	0.18

Cost — improver: millions of tokens the method spent editing, per round

improver tokens per round — millions of tokens (M); in = uncached prompt, cache = prompt read from cache, out = generated; ↳ = another child of the submission counted to its left: one improver call produced several candidates, so its spend appears once, not once per child; a dash means no improver usage is on record for that round – either the method edited nothing or the round’s usage was not recorded – never a zero

method	r1			r2			r3			r4			r5			r6			r7			r8			r9			r10			r11			r12			r13			r14			r15			r16			r17			r18			r19			r20
	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out
sica_ci	0.16	25.6	0.12	0.19	23.3	0.14	0.22	18.5	0.09	0.19	14.2	0.07	0.21	32.3	0.10	0.20	13.2	0.06	0.17	16.6	0.09	0.23	21.7	0.08	0.17	20.1	0.10	0.22	27.9	0.12	0.31	23.9	0.08	0.17	11.4	0.06	0.18	9.9	0.05	0.24	21.4	0.08	0.28	39.9	0.09	0.19	13.7	0.06	0.24	36.9	0.11	0.21	16.4	0.06	0.29	62.2	0.14	0.09	8.2	0.07
ahe	0.32	28.0	0.16	0.28	33.9	0.20	0.30	31.8	0.20	0.26	19.0	0.12	0.40	25.7	0.14	0.29	22.6	0.12	0.33	29.8	0.14	0.31	22.5	0.16	0.30	28.9	0.15	0.23	12.1	0.08	0.41	46.5	0.21	0.37	30.8	0.12	0.33	26.5	0.14	0.39	35.5	0.15	0.42	48.3	0.19	0.43	32.2	0.12	0.47	43.6	0.17	0.38	42.3	0.16	0.43	32.7	0.14	0.47	43.2	0.15
dgm	0.38	51.4	0.31	↳	↳	↳	↳	↳	↳	0.45	66.9	0.33	↳	↳	↳	↳	↳	↳	0.51	57.2	0.28	↳	↳	↳	↳	↳	↳	0.49	61.6	0.28	0.54	72.2	0.30	↳	↳	↳	↳	↳	↳	0.55	57.7	0.22	↳	↳	↳	0.56	73.8	0.27	↳	↳	↳	↳	↳	↳	0.61	59.9	0.24	↳	↳	↳
harnessx	0.10	11.0	0.10	0.15	18.5	0.10	0.22	23.3	0.09	—	—	—	0.14	10.7	0.08	—	—	—	0.17	13.8	0.07	0.16	12.5	0.08	0.22	30.4	0.11	0.23	24.7	0.09	—	—	—	0.24	32.9	0.10	0.20	29.4	0.11	—	—	—	0.20	18.6	0.07	0.22	23.5	0.09	0.21	16.4	0.07	—	—	—	0.17	13.7	0.06	—	—	—
hyperagents	0.38	40.3	0.28	↳	↳	↳	↳	↳	↳	0.17	26.8	0.11	0.52	67.4	0.37	↳	↳	↳	↳	↳	↳	0.59	76.4	0.32	↳	↳	↳	↳	↳	↳	0.18	18.2	0.10	0.36	32.8	0.16	↳	↳	↳	0.60	72.3	0.31	↳	↳	↳	—	—	—	0.53	49.9	0.24	↳	↳	↳	↳	↳	↳	0.58	63.9	0.27
tthe	0.16	25.7	0.13	0.15	14.0	0.11	0.12	6.3	0.04	0.14	13.7	0.07	0.17	19.1	0.10	0.12	6.0	0.04	0.16	12.7	0.07	0.22	27.0	0.09	0.10	2.9	0.02	0.20	14.7	0.09	0.19	19.4	0.10	0.15	14.1	0.07	0.20	18.1	0.08	0.22	33.3	0.11	0.19	17.7	0.08	0.22	29.1	0.13	0.22	25.6	0.09	0.14	10.5	0.06	0.19	18.5	0.08	0.21	23.8	0.11
dream_rsi	0.13	12.2	0.09	0.30	55.6	0.17	0.16	18.9	0.09	0.19	20.8	0.10	0.19	16.3	0.08	0.18	13.9	0.07	0.19	20.3	0.09	0.14	14.5	0.07	0.24	17.3	0.06	0.18	6.7	0.04	0.20	31.0	0.12	0.21	21.4	0.06	0.07	7.3	0.06	0.11	6.6	0.06	0.07	2.4	0.02	0.21	22.3	0.07	0.11	8.1	0.05	0.14	15.6	0.09	0.12	10.4	0.08	0.18	22.5	0.09

dev · 57 tasks · H0 0.000 · validation split, the pool a round is chosen on

Score — dev, 57 tasks per point

method	r0 (H0)	r1	r2	r3	r4	r5	r6	r7	r8	r9	r10	r11	r12	r13	r14	r15	r16	r17	r18	r19	r20
sica_ci	0.000*	0.140	0.281	0.351	0.368	0.351	0.439	0.404	0.316	0.368	0.474	0.509	0.421	0.579	0.596	0.491	0.509	0.509	0.509	0.474	0.561
ahe	0.000*	0.123	0.228	0.351	0.351	0.404	0.439	0.456	0.439	0.000	0.579	0.421	0.579	0.456	0.474	0.439	0.474	0.333	0.474	0.386	0.386
dgm	0.000*	0.053	0.105	0.246	0.368	0.351	0.316	0.070	0.193	0.439	0.404	0.386	0.351	0.421	0.386	0.281	0.439	0.404	0.298	0.298	0.316
harnessx	0.000*	0.070	0.228	0.123	0.281	0.000	0.246	0.281	0.404	0.474	0.404	0.474	0.421	0.281	0.439	0.509	0.526	0.561	0.544	0.439	0.439
hyperagents	0.000*	0.175	0.175	0.088	0.140	0.193	0.281	0.000	0.211	0.263	0.175	0.246	0.421	0.421	0.474	0.123	0.123	0.456	0.474	0.333	0.544
tthe	0.000*	0.035	0.175	0.158	0.228	0.316	0.404	0.386	0.298	0.316	0.228	0.263	0.263	0.333	0.351	0.316	0.351	0.404	0.368	0.333	0.351
dream_rsi	0.000*	0.053	0.000	0.053	0.035	0.158	0.246	0.386	0.368	0.368	0.316	0.439	0.404	0.333	0.263	0.298	0.421	0.491	0.316	0.333	0.439

Cost — task-agent: millions of tokens the harness spent acting on the dev point (in / out)

task-agent tokens per measured point — millions of tokens (M); in = uncached prompt tokens, out = generated tokens; one point is one measurement

method	r0		r1		r2		r3		r4		r5		r6		r7		r8		r9		r10		r11		r12		r13		r14		r15		r16		r17		r18		r19		r20
	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out	in	out
sica_ci	9.47*	0.16*	11.32	0.20	8.10	0.11	6.18	0.12	5.56	0.09	5.70	0.09	7.06	0.12	6.95	0.13	5.61	0.11	5.51	0.10	6.82	0.11	6.63	0.12	7.55	0.14	7.44	0.11	7.56	0.12	8.19	0.12	5.83	0.08	6.06	0.09	5.69	0.08	3.43	0.05	6.56	0.09
ahe	9.47*	0.16*	7.38	0.12	7.80	0.13	6.89	0.10	6.49	0.10	5.50	0.10	6.41	0.11	6.78	0.11	6.92	0.13	8.59	0.10	5.67	0.10	7.41	0.08	7.64	0.09	7.62	0.09	7.88	0.11	8.44	0.10	7.42	0.10	7.72	0.09	8.15	0.10	8.69	0.12	9.79	0.11
dgm	9.47*	0.16*	5.13	0.10	9.41	0.16	4.39	0.09	5.65	0.10	5.52	0.11	8.02	0.12	9.89	0.16	16.34	0.22	9.82	0.11	9.92	0.14	12.29	0.15	10.24	0.15	11.19	0.13	11.85	0.15	5.79	0.10	10.25	0.13	11.44	0.12	4.76	0.11	8.92	0.17	6.12	0.12
harnessx	9.47*	0.16*	7.55	0.11	7.92	0.15	6.64	0.11	8.36	0.15	3.97	0.04	7.64	0.14	6.68	0.13	6.31	0.13	7.41	0.13	6.62	0.11	6.53	0.10	6.92	0.13	6.07	0.12	6.80	0.13	7.65	0.15	7.66	0.17	8.45	0.16	7.26	0.16	7.50	0.13	7.69	0.15
hyperagents	9.47*	0.16*	8.30	0.22	9.10	0.21	5.81	0.10	11.00	0.19	9.57	0.15	4.37	0.12	3.01	0.15	4.08	0.11	11.23	0.12	5.66	0.11	6.01	0.11	5.76	0.10	4.79	0.08	4.37	0.07	13.27	0.22	12.89	0.21	4.52	0.08	6.44	0.11	7.53	0.08	4.88	0.08
tthe	9.47*	0.16*	9.38	0.14	9.57	0.14	9.03	0.14	8.84	0.15	8.01	0.13	7.41	0.12	7.66	0.13	7.83	0.13	7.82	0.13	7.49	0.13	8.17	0.13	7.36	0.13	9.17	0.13	8.87	0.14	9.44	0.15	9.99	0.17	9.11	0.15	8.51	0.14	8.54	0.14	7.91	0.14
dream_rsi	9.47*	0.16*	5.98	0.08	2.85	0.04	7.17	0.09	6.32	0.08	7.85	0.08	7.20	0.06	10.43	0.10	8.33	0.09	7.68	0.09	7.80	0.08	8.20	0.08	7.75	0.09	8.79	0.10	7.72	0.10	7.33	0.09	9.61	0.13	7.80	0.08	7.48	0.09	8.36	0.09	7.82	0.10

Cost — improver: millions of tokens the method spent editing that produced the checkpoint this point validated

improver tokens per round — millions of tokens (M); in = uncached prompt, cache = prompt read from cache, out = generated; ↳ = another child of the submission counted to its left: one improver call produced several candidates, so its spend appears once, not once per child; a dash means no improver usage is on record for that round – either the method edited nothing or the round’s usage was not recorded – never a zero; these are the round’s train-side numbers: a dev point runs no improver, it only prices the checkpoint that round produced

method	r1			r2			r3			r4			r5			r6			r7			r8			r9			r10			r11			r12			r13			r14			r15			r16			r17			r18			r19			r20
	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out	in	cache	out
sica_ci	0.16	25.6	0.12	0.19	23.3	0.14	0.22	18.5	0.09	0.19	14.2	0.07	0.21	32.3	0.10	0.20	13.2	0.06	0.17	16.6	0.09	0.23	21.7	0.08	0.17	20.1	0.10	0.22	27.9	0.12	0.31	23.9	0.08	0.17	11.4	0.06	0.18	9.9	0.05	0.24	21.4	0.08	0.28	39.9	0.09	0.19	13.7	0.06	0.24	36.9	0.11	0.21	16.4	0.06	0.29	62.2	0.14	0.09	8.2	0.07
ahe	0.32	28.0	0.16	0.28	33.9	0.20	0.30	31.8	0.20	0.26	19.0	0.12	0.40	25.7	0.14	0.29	22.6	0.12	0.33	29.8	0.14	0.31	22.5	0.16	0.30	28.9	0.15	0.23	12.1	0.08	0.41	46.5	0.21	0.37	30.8	0.12	0.33	26.5	0.14	0.39	35.5	0.15	0.42	48.3	0.19	0.43	32.2	0.12	0.47	43.6	0.17	0.38	42.3	0.16	0.43	32.7	0.14	0.47	43.2	0.15
dgm	0.38	51.4	0.31	↳	↳	↳	↳	↳	↳	0.45	66.9	0.33	↳	↳	↳	↳	↳	↳	0.51	57.2	0.28	↳	↳	↳	↳	↳	↳	0.49	61.6	0.28	0.54	72.2	0.30	↳	↳	↳	↳	↳	↳	0.55	57.7	0.22	↳	↳	↳	0.56	73.8	0.27	↳	↳	↳	↳	↳	↳	0.61	59.9	0.24	↳	↳	↳
harnessx	0.10	11.0	0.10	0.15	18.5	0.10	0.22	23.3	0.09	—	—	—	0.14	10.7	0.08	—	—	—	0.17	13.8	0.07	0.16	12.5	0.08	0.22	30.4	0.11	0.23	24.7	0.09	—	—	—	0.24	32.9	0.10	0.20	29.4	0.11	—	—	—	0.20	18.6	0.07	0.22	23.5	0.09	0.21	16.4	0.07	—	—	—	0.17	13.7	0.06	—	—	—
hyperagents	0.38	40.3	0.28	↳	↳	↳	↳	↳	↳	0.17	26.8	0.11	0.52	67.4	0.37	↳	↳	↳	↳	↳	↳	0.59	76.4	0.32	↳	↳	↳	↳	↳	↳	0.18	18.2	0.10	0.36	32.8	0.16	↳	↳	↳	0.60	72.3	0.31	↳	↳	↳	—	—	—	0.53	49.9	0.24	↳	↳	↳	↳	↳	↳	0.58	63.9	0.27
tthe	0.16	25.7	0.13	0.15	14.0	0.11	0.12	6.3	0.04	0.14	13.7	0.07	0.17	19.1	0.10	0.12	6.0	0.04	0.16	12.7	0.07	0.22	27.0	0.09	0.10	2.9	0.02	0.20	14.7	0.09	0.19	19.4	0.10	0.15	14.1	0.07	0.20	18.1	0.08	0.22	33.3	0.11	0.19	17.7	0.08	0.22	29.1	0.13	0.22	25.6	0.09	0.14	10.5	0.06	0.19	18.5	0.08	0.21	23.8	0.11
dream_rsi	0.13	12.2	0.09	0.30	55.6	0.17	0.16	18.9	0.09	0.19	20.8	0.10	0.19	16.3	0.08	0.18	13.9	0.07	0.19	20.3	0.09	0.14	14.5	0.07	0.24	17.3	0.06	0.18	6.7	0.04	0.20	31.0	0.12	0.21	21.4	0.06	0.07	7.3	0.06	0.11	6.6	0.06	0.07	2.4	0.02	0.21	22.3	0.07	0.11	8.1	0.05	0.14	15.6	0.09	0.12	10.4	0.08	0.18	22.5	0.09

* round 0 is the shared H0 — one state, measured once and reused, the same way the train side reuses it via --h0-from. Every method prints that one measurement at r0, so the column reads as what it is: a single checkpoint, not seven.

test_normal · 168 tasks

method	chosen on dev	dev (57)	test_normal	test_challenge	eval (585)
hyperagents	r20	0.544	0.458	0.314	0.356
ahe	r10	0.579	0.488	0.259	0.325
dream_rsi	r17	0.491	0.304	0.264	0.275
sica_ci	r14	0.596	0.441	0.206	0.274
harnessx	r17	0.561	0.393	0.199	0.255
dgm	r9	0.439	0.333	0.158	0.209
tthe	r6	0.404	0.232	0.125	0.156

test_challenge · 417 tasks

method	chosen on dev	dev (57)	test_normal	test_challenge	eval (585)
hyperagents	r20	0.544	0.458	0.314	0.356
ahe	r10	0.579	0.488	0.259	0.325
dream_rsi	r17	0.491	0.304	0.264	0.275
sica_ci	r14	0.596	0.441	0.206	0.274
harnessx	r17	0.561	0.393	0.199	0.255
dgm	r9	0.439	0.333	0.158	0.209
tthe	r6	0.404	0.232	0.125	0.156

Contribute

Bring your own setup — a different base harness, model or improver is welcome as long as it is declared and H0 is measured on it. Format and verification: Contributing.