ravage

XBEN Frozen Referee Run — Start Here

This is the official full third run of all 104 XBEN cases. The runner completed the ordered set once, without code changes, retries, or diagnostic replacement runs.

Result

Status	Count
Passed	85
Failed	16
Error	1
Timeout	2
Total	104

• Solve rate: 81.73%
• Exact model cost: $55.758722
• Cost per valid flag: $0.655985
• Model requests: 1,151
• Unmatched model attempts: 0
• Run status: complete; termination reason: completed

Minimum Reviewer Path

Read these files in order:

1. README.md — frozen baseline, contract, and evidence layout.
2. third-run/README.md — run-specific details and non-pass inventory.
3. official-third-run-results.tsv — one authoritative row per case.
4. second-vs-third-results.tsv — status changes from the previous run.
5. third-run/report.json — authoritative machine-readable score and cost.
6. third-run/preflight-source.json — clean commits, exact command, model configuration hash, and tool-image provenance.
7. third-run/preflight.json — enforced comparison contract and all 104 cases.
8. AUDIT.md — independent consistency and artifact-coverage checks.
9. third-run/POST_RUN_SEAL.md — post-run checksum provenance.
10. archives/README.md — raw archive verification and extraction.

Integrity

From the repository root:

shasum -a 256 -c docs/xben-run-evidence/20260712/archives/archives.sha256
shasum -a 256 -c docs/xben-run-evidence/20260712/evidence-artifacts.sha256

The 12 MB compressed archive expands to the complete 171 MB raw evidence tree: all per-case audit databases, transcripts, events, stdout, Docker logs, operator logs, source provenance, target teardown, and network-policy records. It also retains the randomized exact flags required to audit every pass.

Frozen Inputs

• Ravage commit: 358ceef1d11159f725ec1f355b098722a2fd7186
• XBEN benchmark commit: a16cb1ba0701b20d63913846536fc7b0672082af
• Model snapshot: gpt-5.4-2026-03-05
• Tool image: sha256:d6f2c889ef0beb5da60accfe5771ccac930a544512f95a719b394e1f71c86f51
• Contract: MAPTA/AWE-style description-only black box
• Scoring: randomized exact flag
• Tool network: target-only-v1

The operational run was staged under /private/tmp so execution could not dirty the frozen repository. This committed bundle is its durable publication copy.

This site is open source. Improve this page.