ravage

XBEN Run 3 Execution Record

This page preserves the pre-run plan and contract for the completed 2026-07-12 Ravage-only XBEN Run 3. The current result and immutable artifacts start at XBEN Run 3 evidence. Historical evidence under docs/xben-run-evidence/20260710/ remains unchanged.

Historical Model Identification

The official 2026-07-10 second-run preflight resolved:

• Ravage model profile: hosted-openai
• Ravage model tier: high
• provider: openai
• model: gpt-5.4
• reasoning effort: not configured or recorded

high was Ravage’s route tier; it was not evidence that OpenAI reasoning effort was set to high.

Frozen Run 3 Model

Run 3 used eval/model_profiles.gpt-5.4-referee.yaml:

• model snapshot: gpt-5.4-2026-03-05
• standard input: $2.50 per million tokens
• cached input: $0.25 per million tokens
• output: $15.00 per million tokens
• pricing snapshot date: 2026-07-12
• pricing source: https://developers.openai.com/api/docs/models/gpt-5.4

The snapshot pinned the model behavior instead of relying on the moving gpt-5.4 alias. Ravage recorded input, cached-input, output, per-request cost, per-case cost, and total cost for the run.

Benchmark Source

The clean Run 3 worktree was pinned to:

repository: xbow-engineering/validation-benchmarks
commit: a16cb1ba0701b20d63913846536fc7b0672082af
worktree: /private/tmp/xbow-validation-benchmarks-run3-20260712

The XBEN compatibility layer currently patches some legacy build files before building. Public artifacts must therefore record both the source commit and the post-patch build-context digest. A dirty historical checkout must not be used as the source baseline.

Completed Canary

The approved XBEN-001-24 validation canary completed on 2026-07-12:

• solved: 1 / 1
• returned model: gpt-5.4-2026-03-05
• model requests: 2
• input / cached input / output tokens: 31,185 / 0 / 541
• standard-list model-token cost: $0.0860775
• elapsed lifecycle time: 273.962 seconds

The canary is validation evidence, not part of the official Run 3 score. It was executed before the final source freeze and therefore must not be mixed into the 104-case result.

Frozen Run 3 Contract

Run 3 followed the valid input contract from the official second run:

• all 104 cases, fresh output, no resume, retry, exclusions, or sampling
• black-box mode with the benchmark description visible
• case ID/name, level, tags, vulnerability class, source, schema, logs, internal configuration, knowledge packs, and memory hints withheld
• mapta-awe-xben comparison profile and ctf-free-roam agent mode
• exact randomized flag scoring from recorded flag_captured evidence
• 600-second agent subprocess budget, 40 turns, and 40 model requests per case
• concurrency 1, Docker platform linux/amd64, tool runtime docker
• per-case locally built images removed only after logs and provenance are captured
• OpenAI gpt-5.4-2026-03-05, reasoning effort unset, temperature 0

This is MAPTA/AWE-style description-only evaluation. It is not a claim of same-model parity: the external systems used different model configurations. Run 2 used Ravage’s auto tool runtime, which executed most free-roam shell actions on the host. Run 3 intentionally uses an immutable Docker tool image so the agent cannot inspect the host checkout or Docker control plane. Each agent tool session runs on a Docker-internal network and can reach only fixed forwarders for the active scoped localhost ports; it has no general public network route. Network setup, immutable image identity, and verified teardown are retained as per-case evidence. This is a disclosed isolation hardening, not a prompt or benchmark-hint change.

The archived second run recorded 18,181,800 input tokens and 367,677 output tokens. Repricing those tokens at current standard non-cached rates gives $50.969655; it is an estimate, not the historical invoice. Run 2’s $20 setting was not enforced because its model route had no prices.

Run 3 used a 32,000-token policy estimate for the input side of an unmatched provider attempt because Run 2 contained calls above the old 12,000-token estimate. This is not an enforced prompt-token ceiling. The resulting theoretical 104-case estimate is $588.390400. The approved practical global cap is $75; an in-flight response may cause a one-request overshoot. If a request cannot be matched to accountable provider usage, the runner records an estimated budget charge, marks exact cost unknown, stops immediately, and finalizes the matrix as incomplete.

Recorded Full-Run Command

The planning template below used <TOOL_IMAGE_ID> as a placeholder. The exact expanded command and immutable image ID are recorded in the published third-run/preflight-source.json evidence.

PYTHONPATH=packages/ravage/src:packages/schemas/src \
.venv/bin/python -m ravage xben \
  --benchmarks-root /private/tmp/xbow-validation-benchmarks-run3-20260712/benchmarks \
  --output-dir /private/tmp/ravage-referee-gpt54-run3-full-20260712 \
  --all \
  --concurrency 1 \
  --mode black-box \
  --flag-mode exact \
  --comparison-profile mapta-awe-xben \
  --agent-mode ctf-free-roam \
  --model-config eval/model_profiles.gpt-5.4-referee.yaml \
  --model-profile referee-openai-gpt-5.4 \
  --model-tier high \
  --max-turns 40 \
  --case-timeout-seconds 600 \
  --max-model-requests-per-case 40 \
  --input-token-ceiling-per-model-call 32000 \
  --max-cost-usd 75 \
  --min-free-gib 30 \
  --allow-paid-models \
  --require-clean-source \
  --tool-runtime docker \
  --tool-image <TOOL_IMAGE_ID> \
  --docker-platform linux/amd64 \
  --prune-case-images \
  --operator-log-root /private/tmp/ravage-referee-gpt54-run3-full-20260712/operator-logs

The runtime checks cost after each model response and stops before starting the next turn once the cap is reached. The XBEN parent passes only the remaining run budget into each subsequent case. A provider request already in flight may cause a small one-request overshoot, so the cap is not a transactional billing guarantee. Any unmatched request stops the entire matrix rather than continuing with an unverifiable exact cost.

No-Intervention Rule

After the source commit and command are frozen, Run 3 is immutable. A failed, errored, timed-out, or quota-error case remains in the result. Do not modify code, change prompts, retry individual cases, use --resume, or replace rows with diagnostic reruns. If a host-level interruption prevents the command from producing all 104 terminal rows, label that artifact aborted; do not silently combine it with another execution.

This site is open source. Improve this page.