Skip to content

Evidence-Grounded Tier-2 Verification

⚠️ PROVISIONAL — NOT CANONIZED. This is a design under validation, not a made decision. The core claims rest on small samples (n=4 evidence-forcing, n=13 sweep, n=14 gold). It is NOT written into DECISIONS.md and the method is NOT adopted until #1155 lands and the 120-row three-arm A/B confirms false-pass down (incl. the adversarial subset) AND false-negatives not up. Do not cite this as settled architecture until that gate passes (verify-then-canonize).

Status: design + Auditor tooling landed (2026-07-18); production rubric change and A/B validation pending #1155. Canonical-doc entries (DECISIONS/PROCESSES/NOMENCLATURE) are HELD until the A/B passes. Owner lane: Auditor (benchmark + validation definition) → Coder (rubric/scorer implementation). Refs: #849 (local T2 verifier), #1155 (outcome-gated aggregation), #1038 (output-only oracle),

1153 (V2 Phase-2 gate — MOP-4/MOP-6), DECISIONS.md "Output-only success oracle (#1037/#1038)".

This document captures the architecture and decisions behind making the Tier-2 (T2) LLM-as-judge score on verified evidence rather than inference, and the benchmark/validation system that selects the local T2 model. It is the reasoning trail for the #849 workstream.

1. The problem: every T2 judge failure is a judgment made on non-evidence

Tier-2 scoring (testenv/audit_review.py, _tier2_score) is an LLM-as-judge that scores a session 0–3 on a 7-dimension rubric; pass = overall ≥ 2, and a pass gates the training corpus — a false-pass poisons training (the asymmetric, load-bearing failure direction). The #849 sweep + adversarial audit found that every observed judge failure shares one root cause: the verdict was made on something other than verified real output.

Failure mode What it scored on instead of evidence Seen in
Aggregation leniency sound process (findings_grounding) averaged over the 2.0 line despite a failed objective qwen3:32b false-passed 6/28 (#849 sweep); fix = #1155
Echo-only false-OA (#1038) a success token present only in the command, not its output closed at the oracle layer by _real_output() (#1037/#1038)
Passthrough-tag misread the harness [Success: No output] tag read as task success documented cron-persistence human-label case
Human partial-credit a partial step (ligolo agent joined) credited as achieving the objective (reach the flag) single human rater over-scored 1/5 audited disagreements

The unifying fix is structural, not model-scale: force the judgment to cite a specific real-output line and mechanically verify that line exists in real output. This is the _real_output() oracle philosophy (DECISIONS.md, #1037/#1038) lifted from the code-oracle layer up to the LLM-judge layer.

2. Architecture: the evidence gate

Two channels are kept strictly separate for every session (scripts/t2_evidence_adjudicate.py):

  • REAL OUTPUT — concatenated command full_output (what the target actually returned). The only admissible evidence.
  • ECHO — the command text the agent typed. Claims, not evidence.

Pipeline:

session ──▶ evidence-citation prompt ──▶ judge (Opus gold | local model)
                                             │  returns {objective_achieved, score,
                                             │           deciding_evidence:[quoted OUTPUT lines]}
                                             ▼
                       verify_citation(quote)  ──▶  real_output | echo_only | not_found
                                             ▼
                       evidence_gate:  PASS accepted only if ≥1 citation is real_output-grounded,
                                       else UNGROUNDED_PASS  (suspect false-pass → quarantine)
                                             ▼
                       human adjudication worksheet  (verify the cited evidence, do not re-score)
  • verify_citation(quote, real, echo) locates a quoted line by normalized-substring match plus a fuzzy line-ratio (≥0.82) to tolerate truncation; real_output wins ties.
  • evidence_gate() returns GROUNDED_PASS / UNGROUNDED_PASS / FAIL_grounded / FAIL_uncited. UNGROUNDED_PASS is the safety net: a model that passes without a grounded citation is auto-flagged.
  • The human adjudication worksheet (Markdown per session) shows the verdict, each citation with a machine-verified ✅/⚠️/❌ channel mark, and the real output. The human's task is to verify the cited evidence (a fact-check — the human's strength), not score de novo (the weak-rater failure mode).

Key empirical result — forcing beats scale

The same qwen3:32b that false-passed 3 of 4 audited disagreement cases under the loose rubric produced correct, grounded verdicts on all 4 when evidence-forced (jsp_webshell/confirm_vsftpd/ pass_the_hash 2→0 FAIL-grounded; cron 1→3 GROUNDED_PASS citing the verified crontab line). The prompt structure, not the parameter count, drove correctness. This reframes #849: rubric design is the primary lever; model selection is secondary.

3. Proposed decisions (PROVISIONAL — pending the #1155 three-arm A/B; not yet in DECISIONS.md)

  1. Evidence-forced T2 rubric. The T2 judge must cite the exact real-output line grounding its verdict; a pass with no grounded citation is not a pass. (Lift _real_output() to the judge layer.)
  2. Code-side aggregation, evidence-gated pass (#1155). overall is computed in code from the dimension scores (the model no longer supplies it — removes the model-in-a-code-role anti-pattern), and a pass is allowed only if the cited deciding_evidence is grounded (present in real output) and relevant (contains a distinctive anchor from _extract_success_criterion). Unifies #1155 with evidence-grounding into one mechanism keyed on _real_output; gates on a fact, not the noisiest dimension. (Supersedes the earlier "cap overall by completion_validity", which would propagate that dimension's noise into false-negatives. Converged with Coder review 2026-07-18.)
  3. Select on false-pass + Cohen's kappa on the marginal band — never raw agreement. Raw agreement is asymmetric-blind and inflated by class balance; a lenient model can score high while poisoning the corpus.
  4. Gold (Opus) is the best available reference but stays UNFROZEN. It is validated only at 92% vs a 14-row human set (not 59), reads the same 20-line digest as the local models (no context advantage), and shares no measured poisoning bias — but is uncertified. It does not gate the corpus until re-validated on an expanded, multi-rater human set.
  5. Human = evidence-adjudicator + orthogonal bias check, not ground-truth oracle. A single self-identified-weak rater cannot be unquestioned truth; but the audit showed the human right in 4/5 model disagreements and catching the 32b's leniency (orthogonal value). Role: verify cited evidence, guard the two demonstrated human biases (partial-credit leniency; echo-vs-real-output).
  6. Benchmark = representative + adversarial, not the interrater hard-case set. The interrater sample is by-construction the least-agreed cases and does not generalize. Replace with a skill×outcome-stratified representative sample plus deliberately-seeded #1038 right-token/wrong-outcome sessions.

4. Tooling (Auditor-lane, no production-scorer change)

Tool Purpose
scripts/t2_benchmark_sampler.py Deterministic representative+adversarial benchmark sampler. Manifest: data/t2_benchmark_2026-07-18.csv (120 rows: 36 adversarial + 84 representative).
scripts/t2_evidence_adjudicate.py Evidence gate + adjudication. --backend gold\|ollama. Emits per-session human worksheets.

Corpus finding: 33.8% of scanned sessions carry a surface success token with a non-clean outcome — the ambiguity volume the T2 scorer must handle; classic echo-only false-OA is ~0 (post-#1038).

5. Way forward (sequenced — do not reorder)

  1. Land #1155 (outcome-gated aggregation) — removes the false-pass confound before any re-measure.
  2. Gold-label the benchmark (data/t2_benchmark_2026-07-18.csv) via the evidence-forced adjudicator (isolated copies; separate .tier2.gold.json).
  3. Human-adjudicate the worksheets — verify cited evidence; blind-label the adversarial + ambiguous subset; capture intra-rater re-labels to bound human noise.
  4. Re-run the model sweep deterministically (temp=0 + self-consistency) under the evidence-forced rubric, against the expanded gold+human anchor.
  5. Select on weighted false-pass + kappa on the marginal band; exclude genuinely-ambiguous cases.
  6. Governance: selected model never touches the training corpus until gold is re-validated; validate separately before any reuse as a mid-stream critic (#1121).

6. Connection to the V2 gate (#1153)

MOP-4 (false-positive rate, RED) and MOP-6 (residual-review backlog, RED hard gate) both block V2 Phase 2. The evidence-grounded T2 verifier is the mechanism that drains MOP-6 and tightens MOP-4 — but only once #1155 lands, or draining the backlog with a lenient scorer would worsen MOP-4. #1155 is therefore on the critical path to unblocking V2 Phase 2, not a side quest.