Skip to content

Evidence-Grounded Tier-2 Verification

⚠️ PROVISIONAL — NOT CANONIZED. This is a design under validation, not a made decision. The core claims rest on small samples (n=4 evidence-forcing, n=13 sweep, n=14 gold). It is NOT written into DECISIONS.md and the method is NOT adopted until #1155 lands and the 120-row three-arm A/B confirms false-pass down (incl. the adversarial subset) AND false-negatives not up. Do not cite this as settled architecture until that gate passes (verify-then-canonize).

Status: design + Auditor tooling landed (2026-07-18); both gold passes run (2026-07-19) — cloud Opus + a fully-local rig-32B cross-check (see §7); production rubric change and A/B validation still pending #1155. Canonical-doc entries (DECISIONS/PROCESSES/NOMENCLATURE) are HELD until the A/B passes. Owner lane: Auditor (benchmark + validation definition) → Coder (rubric/scorer implementation). Refs: #849 (local T2 verifier), #1155 (outcome-gated aggregation), #1038 (output-only oracle),

1153 (V2 Phase-2 gate — MOP-4/MOP-6), DECISIONS.md "Output-only success oracle (#1037/#1038)".

This document captures the architecture and decisions behind making the Tier-2 (T2) LLM-as-judge score on verified evidence rather than inference, and the benchmark/validation system that selects the local T2 model. It is the reasoning trail for the #849 workstream.

1. The problem: every T2 judge failure is a judgment made on non-evidence

Tier-2 scoring (testenv/audit_review.py, _tier2_score) is an LLM-as-judge that scores a session 0–3 on a 7-dimension rubric; pass = overall ≥ 2, and a pass gates the training corpus — a false-pass poisons training (the asymmetric, load-bearing failure direction). The #849 sweep + adversarial audit found that every observed judge failure shares one root cause: the verdict was made on something other than verified real output.

Failure mode What it scored on instead of evidence Seen in
Aggregation leniency sound process (findings_grounding) averaged over the 2.0 line despite a failed objective qwen3:32b false-passed 6/28 (#849 sweep); fix = #1155
Echo-only false-OA (#1038) a success token present only in the command, not its output closed at the oracle layer by _real_output() (#1037/#1038)
Passthrough-tag misread the harness [Success: No output] tag read as task success documented cron-persistence human-label case
Human partial-credit a partial step (ligolo agent joined) credited as achieving the objective (reach the flag) single human rater over-scored 1/5 audited disagreements

The unifying fix is structural, not model-scale: force the judgment to cite a specific real-output line and mechanically verify that line exists in real output. This is the _real_output() oracle philosophy (DECISIONS.md, #1037/#1038) lifted from the code-oracle layer up to the LLM-judge layer.

2. Architecture: the evidence gate

Two channels are kept strictly separate for every session (scripts/t2_evidence_adjudicate.py):

  • REAL OUTPUT — concatenated command full_output (what the target actually returned). The only admissible evidence.
  • ECHO — the command text the agent typed. Claims, not evidence.

Pipeline:

session ──▶ evidence-citation prompt ──▶ judge (Opus gold | local model)
                                             │  returns {objective_achieved, score,
                                             │           deciding_evidence:[quoted OUTPUT lines]}
                                             ▼
                       verify_citation(quote)  ──▶  real_output | echo_only | not_found
                                             ▼
                       evidence_gate:  PASS accepted only if ≥1 citation is real_output-grounded,
                                       else UNGROUNDED_PASS  (suspect false-pass → quarantine)
                                             ▼
                       human adjudication worksheet  (verify the cited evidence, do not re-score)
  • verify_citation(quote, real, echo) locates a quoted line by normalized-substring match plus a fuzzy line-ratio (≥0.82) to tolerate truncation; real_output wins ties.
  • evidence_gate() returns GROUNDED_PASS / UNGROUNDED_PASS / FAIL_grounded / FAIL_uncited. UNGROUNDED_PASS is the safety net: a model that passes without a grounded citation is auto-flagged.
  • The human adjudication worksheet (Markdown per session) shows the verdict, each citation with a machine-verified ✅/⚠️/❌ channel mark, and the real output. The human's task is to verify the cited evidence (a fact-check — the human's strength), not score de novo (the weak-rater failure mode).

Key empirical result — forcing beats scale

The same qwen3:32b that false-passed 3 of 4 audited disagreement cases under the loose rubric produced correct, grounded verdicts on all 4 when evidence-forced (jsp_webshell/confirm_vsftpd/ pass_the_hash 2→0 FAIL-grounded; cron 1→3 GROUNDED_PASS citing the verified crontab line). The prompt structure, not the parameter count, drove correctness. This reframes #849: rubric design is the primary lever; model selection is secondary.

3. Proposed decisions (PROVISIONAL — pending the #1155 three-arm A/B; not yet in DECISIONS.md)

  1. Evidence-forced T2 rubric. The T2 judge must cite the exact real-output line grounding its verdict; a pass with no grounded citation is not a pass. (Lift _real_output() to the judge layer.)
  2. Code-side aggregation, evidence-gated pass (#1155). overall is computed in code from the dimension scores (the model no longer supplies it — removes the model-in-a-code-role anti-pattern), and a pass is allowed only if the cited deciding_evidence is grounded (present in real output) and relevant (contains a distinctive anchor from _extract_success_criterion). Unifies #1155 with evidence-grounding into one mechanism keyed on _real_output; gates on a fact, not the noisiest dimension. (Supersedes the earlier "cap overall by completion_validity", which would propagate that dimension's noise into false-negatives. Converged with Coder review 2026-07-18.)
  3. Select on false-pass + Cohen's kappa on the marginal band — never raw agreement. Raw agreement is asymmetric-blind and inflated by class balance; a lenient model can score high while poisoning the corpus.
  4. Gold (Opus) is the best available reference but stays UNFROZEN. It is validated only at 92% vs a 14-row human set (not 59), reads the same 20-line digest as the local models (no context advantage), and shares no measured poisoning bias — but is uncertified. It does not gate the corpus until re-validated on an expanded, multi-rater human set.
  5. Human = evidence-adjudicator + orthogonal bias check, not ground-truth oracle. A single self-identified-weak rater cannot be unquestioned truth; but the audit showed the human right in 4/5 model disagreements and catching the 32b's leniency (orthogonal value). Role: verify cited evidence, guard the two demonstrated human biases (partial-credit leniency; echo-vs-real-output).
  6. Benchmark = representative + adversarial, not the interrater hard-case set. The interrater sample is by-construction the least-agreed cases and does not generalize. Replace with a skill×outcome-stratified representative sample plus deliberately-seeded #1038 right-token/wrong-outcome sessions.

4. Tooling (Auditor-lane, no production-scorer change)

Tool Purpose
scripts/t2_benchmark_sampler.py Deterministic representative+adversarial benchmark sampler. Manifest: data/t2_benchmark_2026-07-18.csv (120 rows: 36 adversarial + 84 representative).
scripts/t2_evidence_adjudicate.py Evidence gate + adjudication. --backend gold\|ollama. Emits per-session human worksheets.
data/t2_benchmark_gold_2026-07-19.csv + adjudication/2026-07-19/ Cloud gold — 120 sessions, claude-opus-4-8, evidence-forced. Gate dist: 40 GROUNDED_PASS / 2 OFFCRITERION / 1 UNGROUNDED / 72 FAIL_grounded / 5 FAIL_uncited.
data/t2_benchmark_gold_local_rig32b_2026-07-19.csv + adjudication/2026-07-19-local-rig32b/ Local gold cross-check — 120 sessions, qwen3:32b on rig, evidence-forced, fully local (§7).

Corpus finding: 33.8% of scanned sessions carry a surface success token with a non-clean outcome — the ambiguity volume the T2 scorer must handle; classic echo-only false-OA is ~0 (post-#1038).

5. Way forward (sequenced — do not reorder)

  1. Land #1155 (outcome-gated aggregation) — removes the false-pass confound before any re-measure.
  2. Gold-label the benchmark (data/t2_benchmark_2026-07-18.csv) via the evidence-forced adjudicator (isolated copies; separate .tier2.gold.json).
  3. Human-adjudicate the worksheets — verify cited evidence; blind-label the adversarial + ambiguous subset; capture intra-rater re-labels to bound human noise.
  4. Re-run the model sweep deterministically (temp=0 + self-consistency) under the evidence-forced rubric, against the expanded gold+human anchor.
  5. Select on weighted false-pass + kappa on the marginal band; exclude genuinely-ambiguous cases.
  6. Governance: selected model never touches the training corpus until gold is re-validated; validate separately before any reuse as a mid-stream critic (#1121).

6. Connection to the V2 gate (#1153)

MOP-4 (false-positive rate, RED) and MOP-6 (residual-review backlog, RED hard gate) both block V2 Phase 2. The evidence-grounded T2 verifier is the mechanism that drains MOP-6 and tightens MOP-4 — but only once #1155 lands, or draining the backlog with a lenient scorer would worsen MOP-4. #1155 is therefore on the critical path to unblocking V2 Phase 2, not a side quest.

7. Validation — two gold passes, and the forcing thesis tested (2026-07-19)

The central claim of §2's "forcing beats scale" result — rubric design is the primary lever; model scale is secondary — is directly testable: run the same evidence-forced adjudicator with two different judges over the full 120-row benchmark and compare, holding the rubric constant so any difference is purely the judge model.

  • Cloud reference — claude-opus-4-8 (data/t2_benchmark_gold_2026-07-19.csv).
  • Local candidate — qwen3:32b on rig, fully local, nothing left the host (data/t2_benchmark_gold_local_rig32b_2026-07-19.csv). Motivation: the local-first principle and the Ollama/RIG direction — if a local judge tracks Opus, the cloud judge can be retired for this step.

Agreement (local 32B vs cloud Opus):

Measure Result
Pass/fail (objective_achieved) 110/120 = 91.7%, Cohen's kappa 0.826
Exact evidence-gate category 106/120 = 88.3%
Score 0–3 exact 79%, within-1 92%, mean |Δ| 0.32
Adversarial subset (36 rows) 32/36 = 89%

Reading it. The forcing works directionally: a local 32B under the evidence-forced rubric reaches kappa 0.83 against Opus — far tighter than the loose-rubric leniency #849 first found. But the 10 disagreements are asymmetric and in the dangerous direction: 9/10 are local-PASS / cloud-FAIL (local more lenient — the corpus-poison direction), and 3 of those are adversarial right-token/ wrong-outcome rows the local judge fell for and Opus caught. So under the current forcing the local 32B is close but still measurably more lenient on exactly the cases that gate the corpus.

Consequences (all PROVISIONAL — the human adjudication resolves them): 1. The relevance-mode evidence gate auto-caps the 2 UNGROUNDED_PASS local passes; the other 7 are GROUNDED_PASS-but-Opus-disagrees and need the criterion-relevance anchor or a human. 2. Gold anchor stays cloud Opus until the 10 divergences are human-adjudicated. Per the #849 pattern (human sided with the stricter read 4/5), most are expected to confirm Opus — which would disqualify local 32B as the gold judge while leaving it viable as a cheap pre-filter. 3. This does not yet retire the cloud judge; it quantifies how close local gets and where it fails, and it hands the decisive 10 rows to human adjudication.

This §7 result is a validation data point, not a canonization: the method is not adopted until the

1155 A/B passes its gate (false-pass down incl. the adversarial subset, false-neg not up).