ravage

Benchmarking

Ravage has two current benchmark surfaces:

ravage xben: run Ravage against XBEN-style benchmark cases with exact flag scoring, per-case logs, preflight, paid-model controls, and audit artifacts.
ravage competitors: run Ravage or external agents through the isolated head-to-head harness and generate comparison artifacts from configured adapters.

The repository currently publishes only Ravage’s frozen 2026-07-12 XBEN Run 3. It does not publish a real external-agent matrix or leaderboard. The checked-in competitor configuration contains smoke adapters and is not a head-to-head result.

The old top-level ravage --benchmark ... examples were from an earlier local manifest path. Do not use them for public results. Use ravage xben for XBEN validation and ravage competitors for referee-style comparisons.

For the local lab matrix, flag counts, scoring policy, and recorded benchmark evidence, see Benchmarks And Local Test Boxes and XBEN Run Evidence.

Current Published Result

The current Ravage-only baseline solved 85 / 104 public XBEN cases (81.73%) in description-only black-box mode. The run records 16 failures, 1 error, 2 timeouts, $55.758722 in provider-usage list-price model cost, and $0.655985 per valid flag. Review the frozen evidence for the exact contract, per-case ledger, audit, provenance, and hashes.

XBOW now describes this public suite as outdated and saturated. Use the result as a public-benchmark regression and evidence-integrity baseline, not as proof of frontier performance, unseen-task generalization, or expected production pentest efficacy. The top-level Honest Limitations also covers repeatability, model dependence, cost scope, weaker coverage areas, runtime portability, and raw-evidence risk.

XBEN Flow

XBEN is an execution envelope around the public attack command. It provisions the case, writes a normal brief, runs ravage attack, exact-matches the hidden random proof, and tears the case down. The attacker used here is therefore the same attacker users run against local labs and their own authorized boxes.

List available cases before spending model calls:

ravage xben \
  --benchmarks-root /path/to/xbow-validation-benchmarks/benchmarks \
  --levels L1 L2 \
  --list

Run a no-spend preflight for a small selection:

OPENAI_API_KEY=... \
ravage xben \
  --benchmarks-root /path/to/xbow-validation-benchmarks/benchmarks \
  --output-dir runs/xben/preflight \
  --ids XBEN-001-24 XBEN-002-24 \
  --mode black-box \
  --comparison-profile mapta-awe-xben \
  --agent-mode ctf-free-roam \
  --model-profile hosted-openai \
  --model-tier low \
  --tool-runtime host \
  --max-turns 8 \
  --max-model-requests-per-case 8 \
  --max-cost-usd 5 \
  --preflight

Inspect runs/xben/preflight/preflight.json before running paid models. Hosted or paid-risk routes are blocked unless --allow-paid-models is present.

Run the same selected cases:

OPENAI_API_KEY=... \
ravage xben \
  --benchmarks-root /path/to/xbow-validation-benchmarks/benchmarks \
  --output-dir runs/xben/selected \
  --ids XBEN-001-24 XBEN-002-24 \
  --mode black-box \
  --comparison-profile mapta-awe-xben \
  --agent-mode ctf-free-roam \
  --model-profile hosted-openai \
  --model-tier low \
  --tool-runtime host \
  --max-turns 8 \
  --max-model-requests-per-case 8 \
  --max-cost-usd 5 \
  --allow-paid-models

Useful selection flags:

--ids XBEN-001-24 XBEN-002-24: run explicit IDs.
--range 1-10: run a numeric range.
--levels L1 L2: filter by benchmark level.
--sample 10 --sample-seed 1234: run a reproducible random sample.
--exclude-ids ...: remove known-bad or out-of-scope cases from a selection.
--resume: continue an interrupted run.
--retry-failed: rerun only failed cases from an existing output directory.

Useful execution and scoring flags:

--mode black-box|white-box|source-aware: select benchmark context.
--comparison-profile mapta-awe-xben: keep challenge context aligned with the MAPTA/AWE-style benchmark description flow.
--flag-mode exact|pattern: choose exact flag matching or pattern matching.
--case-timeout-seconds N: cap one case wall-clock runtime.
--max-model-requests-per-case N: cap model calls per case.
--max-cost-usd N: block runs when pricing metadata estimates exceed the cap.
--operator-log-root PATH: store operator logs outside the run directory when needed.
--tool-runtime host|docker|auto: choose host tools, Docker-backed tools, or auto mode.

Output Shape

An XBEN run writes one directory per run:

runs/xben/selected/
  preflight.json
  report.json
  artifacts.sha256
  operator-logs/
  XBEN-001-24/
    audit.db
    agent.stdout
    docker.log
    workspace/

Important report fields:

summary.total, summary.solved, summary.failed, summary.errored, and timeout counts.
per-case status, found_flag, elapsed_seconds, model_request_count, token usage, and cost_usd.
paths to stdout, workspace events, audit DB, transcripts, artifacts, and target evidence.
failure text for cases that did not produce a valid benchmark flag.

For public claims, keep the raw run directory, the report, and hash manifests. Do not edit logs after the run. If you need to rerun, write to a new directory and document the commit SHA used for that run.

Competitor Harness

Use the competitor harness when the claim is a head-to-head comparison rather than a single Ravage score:

The checked-in eval/competitor_harness.example.yaml is a harness smoke test. Replace its agents, expected flags, commits, models, and commands before treating any output as benchmark evidence.

ravage competitors preflight \
  --config eval/competitor_harness.example.yaml \
  --output-dir runs/competitors/preflight

Then run:

ravage competitors run \
  --config eval/competitor_harness.example.yaml \
  --output-dir runs/competitors/head-to-head

The harness creates an isolated Docker network per target box and scores each agent artifact using the same fields. A valid public leaderboard should include:

agent name, adapter version, target box, and model route;
valid flags, invalid/self-reported flags, false positives, and out-of-scope findings;
cost or cost_status: unknown;
artifact links for the adapter JSON, stdout, stderr, report, and hash file.

See Competitor Harness for the adapter contract.

Evidence Standard

Benchmark claims should be evidence-first:

A flag counts only when it is observed from the target and matches the expected flag policy.
A vulnerability finding needs replayable request/response proof or another target-origin evidence artifact.
Model text is never proof by itself.
Source-aware hints can guide attempts, but reported findings still need live confirmation.
Cost must be reported as measured, computed from token/pricing metadata, or explicitly marked unknown.

Public Artifact Handling

Public announcements should link to the reviewer-facing START_HERE.md, report, TSV, audit, and checksum manifests rather than directly to raw transcripts. The frozen 2026-07-12 bundle also commits its sealed raw archive publicly; that archive’s known contents include synthetic flags, disposable local-target authentication material, model completion identifiers, machine-local paths, complete exploit traces, and ephemeral session values. The recorded scan found no credential matching its selected provider-key patterns, but that narrow check is not a general safety guarantee.

The archive’s internal guide records a more conservative access-controlled handling standard. Do not rewrite the sealed bundle to resolve that historical policy mismatch because doing so would invalidate its published hashes. For future releases, publish redacted reviewer artifacts by default, keep raw security-test evidence access-controlled, and never include real customer or production credentials.

The outer archive checksum authenticates the exact compressed archive. The inner raw-artifact manifest verifies its named paths; like a normal shasum -c run, it does not prove that no additional paths exist. Future archives should also publish a complete inventory/extra-file check and suppress operating-system metadata during packaging.

Publishing randomized flags does not reveal the value used by a later run, but publishing complete solution traces can contaminate future evaluation on the same public cases through browsing or training exposure. Label later XBEN runs as public-regression results, keep agent tool networks isolated from the public internet, and use fresh or private variants for unseen-task claims.

Repeatability

For a release-quality run:

Record the git commit SHA before running.
Run preflight and keep the preflight output.
Run the benchmark into a fresh output directory.
Generate hash manifests for reports, logs, artifacts, and archives.
Commit only the evidence artifacts after the code commit if the goal is to prove no code changed between runs.

This is the proof story Ravage should lead with: same harness, same commit, exact-flag scoring, retained logs, and auditable cost fields.

This site is open source. Improve this page.