Run-plans (`docs/rig-eval/plans/`)¶

Declarative eval sweeps for ARCHER-Live (#1160). A run-plan is a version-controlled YAML file describing a matrix of backends × models × objective-sets × ablations × runs. The compiler (testenv/plan_compiler.py, #1162) turns it into whitelisted testenv/eval_harness.py invocations — the same commands a human would type by hand, but reproducible and schema-validated.

These replace the hand-rolled ablation shell scripts (scripts/quick_eval_8b_vs_32b.sh, hint_ablation_8b_vs_32b.sh, combined_ablation.sh).

Why a schema and not a script¶

The launch surface is Tailscale-reachable. Arbitrary browser-authored code would be a remote-code-execution (RCE) surface, so a plan is data, not code: parsed with yaml.safe_load only, and the compiler emits only parameters enumerated in plan.schema.json. Any flag not on the whitelist cannot reach argv.

Contract: plan.schema.json (JSON Schema, draft-07) is the canonical documented format.
Validation: testenv/plan_schema.py → validate_plan(dict) -> (ok, errors) enforces it in pure Python (no jsonschema dependency). A drift test keeps the Python whitelists equal to the schema enums.

Format¶

plan: <human-readable name>
run_sets:
  - name: <run-set name>
    backends: [rig]                 # rig | rog; one → sequential, two → --parallel only if parallel:true
    models: ["qwen3:8b", "qwen3:32b"]   # Ollama tags; QUOTE tags (the ':' and the YAML "Norway" trap)
    objectives: [PT-WEBEX-02, PT-EXPLOIT-01]   # explicit IDs …
    #   … OR a selector object:
    # objectives: { strategy: sparse, confidence-gate: 0.6, ambiguous: true, skip: [PT-XSS-01] }
    runs: 2                         # default 3
    parallel: false                 # opt-in to concurrent two-backend exec (gated on #1159)
    ablations: [hint-ablation]      # hint-ablation | phrasing-sensitivity | decoy-injection | with-playbook
    flags: [no-seed-playbook, no-compare]   # whitelisted passthrough only
    playbook_db: /tmp/exp.db        # isolated ARCHER_PLAYBOOK_DB (never production)

Field reference¶

Field	Required	Notes
`plan`	✓	plan name
`run_sets[].name`	✓	run-set name
`run_sets[].backends`	✓	list of `rig`/`rog`, ≥1
`run_sets[].models`	✓	list of Ollama tags, ≥1; matrix axis
`run_sets[].objectives`	✓	list of IDs or a `{strategy, confidence-gate, ambiguous, skip}` selector
`run_sets[].runs`		integer ≥1, default 3
`run_sets[].parallel`		bool, default false; two backends + `parallel: true` → one `--parallel --backends a,b`
`run_sets[].ablations`		fixed enum → harness ablation flags
`run_sets[].flags`		whitelisted passthrough flags only
`run_sets[].playbook_db`		isolated DB path → `ARCHER_PLAYBOOK_DB`

The YAML "Norway problem"¶

Unquoted no, yes, on, off coerce to booleans under YAML 1.1 semantics. Objective IDs and model tags are always strings — quote anything that could be misread as a bool. validate_plan rejects a coerced boolean where a string is required, so the trap fails closed rather than silently mis-running.

Worked example¶

quick-8b-vs-32b.yaml reproduces scripts/quick_eval_8b_vs_32b.sh — qwen3:8b and qwen3:32b over four inference-heavy objectives, 2 runs each, rig-routed, --no-seed-playbook --no-compare. It compiles to two eval_harness.py invocations (one per model), each with OLLAMA_HOST=http://127.0.0.1:11435 and ARCHER_BACKEND_FORCE=rig.

Behavioral-equivalence contract¶

The compiler is the equivalence contract: a compiled plan must produce byte-identical harness invocations (argv + env) to the hand-typed command. Golden tests (tests/core/test_plan_compiler_1162.py) assert the example plans compile to exactly the intent of the original .sh scripts.

Scope note (Phase 1)¶

The MVP compiler expresses the flat matrix (backends × models, with ablation/flag passthrough). The stateful playbook-drag protocol in combined_ablation.sh ARM 2 (seed an isolated DB with 8b, then compare 32b seeded-vs-clean) is a multi-phase orchestration, not a flat matrix — a plan can express a single with-playbook run against an isolated playbook_db, but the seed→restore→compare sequencing stays scripted until a later phase. See #1160.

Run-plans (docs/rig-eval/plans/)¶