Benchmarks

{/* Generated by hg-work/rsi10/export-site.py; last export 2026-10-19T17:07:34Z. */}

Each page here is one benchmark. A benchmark with published measurements carries the same five things: what it measures and how its tasks are split, the scores this platform measured, the exact configuration those runs used, where the full trajectories live, and how to submit a result of your own. A benchmark whose adapter is merged but which has not produced a valid point yet says so on its own page, and publishes no number.

A score is published only when it can be traced to per-task verdicts: a point that measured 139 of a pool’s 140 tasks is excluded and named, never averaged in. The frozen reference (H0, the base harness on the same tasks) is printed with every table, because a raw number without it does not say whether a method helped.

benchmark	tasks	split	published points	status
ALFWorld	100 study / 134 eval	study train, eval valid_unseen, plus valid_seen and valid_train	57 on valid_unseen, 57 on valid_seen, 57 on valid_train	measured
AppWorld	732 released	study train, dev dev, test test_normal + test_challenge	—	in progress
LoCoMo	10 conversations / 1,986 questions	none — the benchmark declares no study/eval split	—	in progress

How to read any table here

The rules below are the same on every benchmark page, so they are stated once:

• H0 (round 0) is the base harness, before any method touched it; the distance from that line is what a method added. Every table prints it.
• — means not measured, never zero. † marks the round the method kept, scored by the exam run that chose it; the checkpoint sweep does not re-measure one state twice. A line in a chart breaks where a round was not measured — it is not a straight line through unmeasured rounds.
• Every published point covers its whole pool with n_invalid = 0. A point that measured fewer tasks, or hit an invalid/verifier failure, is excluded and listed under Excluded with its task count; it is never averaged in.
• A move smaller than the noise is not evidence. The same-arm replicate spread on 20 ALFWorld tasks is ~6.5 pp (SD 1.3 tasks), so 20-task arms move in 5-point steps; on 134 tasks the standard error is ~4.3 pp.
• Rows are only comparable within one setup. A different agent model, base harness or improver is a different measurement; the setup is printed above each table.

An in progress page is not a result and is not counted anywhere: it exists so that the question “does this platform cover AppWorld?” has an answer that distinguishes “the adapter is merged” from “we measured it”. Integration status is tracked in the repository’s registry/dataset-blockers.json.

Missing a benchmark you care about? A dataset page is the first contribution — see Contributing.