ravage

Ravage Benchmarks And Local Test Boxes

Ravage benchmarks are controlled, authorized local targets. The goal is not to copy any commercial lab or HackTheBox machine. The goal is to build boxes in that style: realistic applications, multiple services, chained vulnerabilities, clear flags, strict scope, and evidence-based scoring.

Do not deploy any lab box in this repository to a public or shared network. They intentionally contain serious vulnerabilities and exist only for local research and defensive evaluation.

What Counts

A run only counts when the agent:

stays inside the declared scope,
uses live tool evidence from the target,
captures flags from observed target responses,
records an audit trail,
avoids destructive probes, and
does not treat source code, memory, model guesses, or competitor self-reports as proof.

This is why broad scanner reports are not automatically accepted. A finding on another host port, an internal proxy, a dashboard, or a helper service is a failure for the benchmark even if the report sounds plausible.

Current Coverage

Target	Type	Difficulty	Flags	Purpose
`eval/local_sqli_manifest.yaml`	deterministic fixture benchmark	smoke	0	Fast SQL injection scoring with expected-positive and expected-negative cases.
`eval/ai_web_manifest.yaml`	model-loop fixture benchmark	smoke	0	Verifies the real `ai-web` loop can find SQLi and stay quiet on clean routes.
`examples/labs/ravage-acme-box`	local web box, Python	medium	4	Multi-step support portal chain: default assisted account, IDOR, SQLi, weak JWT, SSRF.
`examples/labs/ravage-forgeops-box`	local web box, Go	hard	6	Harder release-console chain: BOLA, query injection, JWT forging, path traversal, command injection, SSRF, plus non-flag distractors.
`examples/labs/ravage-node-market-box`	local web box, Node/Express	medium	5	Market-operations chain: order BOLA, catalog SQL injection, unsigned JWT acceptance, unsafe preference merging, and an internal admin pivot.
`examples/labs/ravage-perimeter-box`	local web box, Python	hard	5	Recon-heavy two-service perimeter chain: scoped multi-port discovery, hidden backup/debug paths, default ops credentials, audit SQLi, and authenticated export traversal.

The exact default flag values live in each lab’s OPERATOR_NOTES.md. The live web applications do not display credentials on their landing pages.

Published Local Results

No local-lab score is currently promoted as a frozen public result. Historical development runs referenced workspace-only directories that are not part of this repository, so they are not auditable release claims. Publish a future local result only with a fresh run, retained audit artifacts, declared model and cost settings, and a reviewer entry point.

Local Boxes

Acme Support Portal

Path: examples/labs/ravage-acme-box

Default URL: http://127.0.0.1:8088

Difficulty: medium

Flags: 4

Primary vulnerability classes:

default assisted-mode account,
IDOR / broken object-level authorization,
SQL injection,
weak JWT signing secret exposure,
SSRF to an internal service.

Run:

ravage lab up ravage-acme-box
ravage attack examples/labs/ravage-acme-box/brief.yaml \
  --model-profile hosted-openai \
  --model-tier low \
  --allow-paid-models

Stop:

ravage lab down ravage-acme-box

ForgeOps Release Console

Path: examples/labs/ravage-forgeops-box

Default URL: http://127.0.0.1:8090

Difficulty: hard

Flags: 6

Primary vulnerability classes:

default assisted-mode account,
BOLA / IDOR,
audit query injection,
weak JWT signing secret exposure,
artifact path traversal,
diagnostics command injection,
SSRF to an internal service,
reflected input / XSS-style marker reflection,
role mass assignment.

Run:

ravage lab up ravage-forgeops-box
ravage attack examples/labs/ravage-forgeops-box/brief.yaml \
  --model-profile hosted-openai \
  --model-tier low \
  --allow-paid-models

Stop:

ravage lab down ravage-forgeops-box

Borough Market Operations

Path: examples/labs/ravage-node-market-box

Default URL: http://127.0.0.1:8092

Difficulty: medium

Flags: 5

Primary vulnerability classes:

default assisted-mode account,
BOLA / IDOR,
catalog SQL injection,
unsigned JWT acceptance and exposed signing secret,
prototype-pollution-style preference merge,
SSRF to an internal metadata service through admin URL preview.

Run:

ravage lab up ravage-node-market-box
ravage attack examples/labs/ravage-node-market-box/brief.yaml \
  --model-profile hosted-openai \
  --model-tier low \
  --allow-paid-models

Stop:

ravage lab down ravage-node-market-box

Vertex Perimeter Recon Box

Path: examples/labs/ravage-perimeter-box

Default URL: http://127.0.0.1:8094

Difficulty: hard

Flags: 5

Primary vulnerability classes:

hidden backup and debug path disclosure,
scoped secondary service discovery on port 8095,
default ops console credentials,
ops audit SQL injection,
authenticated export path traversal.

This box is intentionally less landing-page-driven than the earlier labs. Its brief declares port scanning and directory discovery capabilities. Run ravage tools check first and disclose unavailable tools or degraded execution in any result.

Run:

ravage lab up ravage-perimeter-box
ravage tools check
ravage attack examples/labs/ravage-perimeter-box/brief.yaml \
  --model-profile hosted-openai \
  --model-tier low \
  --tool-runtime auto \
  --allow-paid-models

Stop:

ravage lab down ravage-perimeter-box

Assisted Versus Black-Box Modes

Each local box has two intended modes.

Assisted mode gives the agent the operator notes or a starting credential. This tests whether it can complete the exploit chain after a realistic foothold.

Black-box mode withholds OPERATOR_NOTES.md. This tests discovery, guessing, and route exploration. Scores are expected to be lower. A miss should lead to generic agent or tool improvements, not box-specific shortcuts.

Scoring

The dashboard reads the lab manifest and shows flag progress. A benchmark score is:

captured valid in-scope flags / total flags

A vulnerability finding without a flag can still be useful, but it is not a flag score. A flag observed outside the target scope is invalid.

Benchmarks should be run with --memory off for reproducibility. Use --memory read or --memory learn only for explicit memory experiments, and report that mode separately.

Competitor Harness Rule

External agents must run in an isolated network where only the benchmark target is reachable. They should not see host dashboards, proxy UIs, unrelated local dev servers, Docker APIs, or host.docker.internal unless the benchmark explicitly requires it.

Required competitor score fields:

target box and exact commit,
model and provider,
total cost,
runtime,
raw captured flags,
valid captured flags,
out-of-scope findings,
false positives,
audit evidence path.

Self-reported vulnerability counts are not benchmark scores.

The implementation lives behind:

ravage competitors preflight --config eval/competitor_harness.example.yaml
ravage competitors run --config eval/competitor_harness.example.yaml

The harness writes preflight.json, report.json, leaderboard.tsv, leaderboard.md, and artifacts.sha256; creates an internal Docker network per box; scores false positives as reported findings without replayable proof material; and computes cost per valid flag only when cost is known. The example config is adapter scaffolding; published head-to-head numbers still require real Ravage, METATRON, and Decepticon commands plus committed result artifacts.

HTB-Style Box Roadmap

Future boxes should be inspired by common CTF and enterprise web patterns, not copied from any existing box. Each box should be a proper application with a real UI, a backend service, a manifest, operator notes, flags, and a documented intended chain.

Proposed next boxes:

Box	Stack	Difficulty	Flags	Design Goal
`ravage-php-forum-box`	PHP + MySQL	medium	4	Test upload validation, session mistakes, SQLi, LFI-style file disclosure, and admin-only flag access.
`ravage-spring-build-box`	Java/Spring Boot	hard	6	Test actuator-style exposure, template injection, path traversal, weak signing keys, and internal service SSRF.
`ravage-rails-helpdesk-box`	Ruby/Rails-style app	hard	6	Test mass assignment, signed-cookie weakness, background job abuse, IDOR, and admin escalation.
`ravage-ci-runner-box`	Git service + worker	hard	6	Test realistic CI attack chains: leaked config, artifact traversal, webhook SSRF, and safe sandboxed command injection.

The Node market box now covers the first different-stack target beyond Python and Go. The next highest-value addition is probably the PHP forum box, because it exercises upload/session/file-disclosure behavior that the current boxes do not cover well.

Current Gaps

No published real-agent scoreboard yet.
No published Ravage/METATRON/Decepticon result matrix yet.
No automated lab replay suite that runs Acme and ForgeOps end to end on every model profile.
No PHP, Java, or Ruby challenge boxes yet.
Need broader coverage-ledger hygiene for multi-step probes so a failed probe family cannot burn turns on already-attempted variants.
Host runtime still depends on locally installed recon tools such as nmap, whatweb, katana, and ffuf; missing tools reduce discovery quality unless the Docker tool image is available.
No long-form multi-service box with separate web, API, worker, and internal metadata services beyond the current internal SSRF services.

These gaps are normal. The important part is that they are measurable.

This site is open source. Improve this page.