ravage

Start Here

Do not start by extracting the raw archives. The raw files are audit backing: per-case transcripts, run metadata, operator logs, audit databases, and hashes. They are bundled under archives/ so the Git commit stays reviewable.

Score Summary

First-run checklist rollup:

• Solved: 85 / 104
• Score: 81.73%
• Failed: 18
• Errored: 1
• Open: 0

Authoritative first-run source: first-run/xben-mapta-benchmark-checklist.md

Official full second run result:

• Solved: 84 / 104
• Score: 80.77%
• Failed: 13
• Errored: 7
• Timeout: 0
• Quota error: 0

Authoritative second-run source: second-run/report.json

That file is a visible copy of the original raw report stored inside archives/second-run-raw.tar.gz at docs/xben-run-evidence/20260710/second-run/raw/run/report.json.

The second run is one solved case lower than the first-run checklist rollup.

Minimum Reviewer Path

Read these files in order:

1. START_HERE.md
2. second-run/preflight-source.json
3. second-run/report.json
4. official-second-run-results.tsv
5. first-run-checklist-results.tsv
6. first-vs-second-results.tsv
7. AUDIT.md
8. archives/README.md
9. local-operator-logs/README.md
10. second-run/README.md
11. errored-reruns/README.md

That gives the score, the clean source preflight, the exact command, the case-level results, the audit review notes, and the classification of the official errored cases.

Preflight Evidence

second-run/preflight-source.json is the key preflight evidence file. It records that the official full second run started from a clean repository at:

• Commit: e5bc553555acb2f17b1cf643bfb711790a35d993
• Tree: 8a02438401f6553a49282ce6caf577f6a75f76f9
• Subject: first commit

It also records the exact full-run command and runtime environment.

The full run artifacts were copied into this repository after the run, bundled, and hashed:

• Full evidence manifest: evidence-artifacts.sha256
• Archive manifest: archives/archives.sha256
• Second-run raw manifest: second-run/raw-artifacts.sha256

Where The Evidence Is

• First-run checklist copy: first-run/xben-mapta-benchmark-checklist.md
• First-run one-row-per-case summary: first-run-checklist-results.tsv
• Local rolling operator logs copied from /Users/oha/ravage/logs: archives/supporting-raw.tar.gz
• Parsed local-log table: local-operator-logs-results.tsv
• Local-log mismatches against the first checklist: local-logs-vs-first-checklist-mismatches.tsv
• Official second-run report: second-run/report.json
• Official second-run one-row-per-case summary: official-second-run-results.tsv
• First-vs-second comparison: first-vs-second-results.tsv
• Source preflight: second-run/preflight-source.json
• Benchmark/MAPTA-AWE preflight: second-run/benchmark-preflight.json
• Official second-run raw archive: archives/second-run-raw.tar.gz
• Supporting raw archive: archives/supporting-raw.tar.gz
• Per-case operator logs inside archive: docs/xben-run-evidence/20260710/second-run/raw/operator-logs/XBEN-*/benchmark.log
• Per-case run metadata inside archive: docs/xben-run-evidence/20260710/second-run/raw/run/XBEN-*/run.json
• Per-case agent transcript inside archive: docs/xben-run-evidence/20260710/second-run/raw/run/XBEN-*/workspace/transcript.jsonl
• Per-case tool/event trace inside archive: docs/xben-run-evidence/20260710/second-run/raw/run/XBEN-*/workspace/events.jsonl

What The Raw Files Mean

For a passed case, reviewers can inspect:

• docs/xben-run-evidence/20260710/second-run/raw/run/XBEN-XXX-24/run.json
• docs/xben-run-evidence/20260710/second-run/raw/operator-logs/XBEN-XXX-24/benchmark.log
• docs/xben-run-evidence/20260710/second-run/raw/run/XBEN-XXX-24/workspace/events.jsonl
• docs/xben-run-evidence/20260710/second-run/raw/run/XBEN-XXX-24/workspace/transcript.jsonl

For a failed or errored case, reviewers should inspect:

• official-second-run-results.tsv
• docs/xben-run-evidence/20260710/second-run/raw/run/XBEN-XXX-24/run.json
• docs/xben-run-evidence/20260710/second-run/raw/operator-logs/XBEN-XXX-24/benchmark.log
• errored-reruns/README.md if it was one of the 7 official errors

Those raw paths are inside archives/second-run-raw.tar.gz; see archives/README.md for verification and extraction commands.

Diagnostic Reruns

Diagnostic reruns are stored separately under errored-reruns/. They are not the official score. They explain what happened to the 7 official errored cases when rerun later.

Important: the official score remains 84 / 104. Diagnostic reruns are context, not a replacement score.

This site is open source. Improve this page.