Skip to content

Program: Eval-Integrity Rebaseline & False-Positive Re-Evaluation

For: ARCHER-Coder + Auditor (execution); operator coordinates. From: sag/Coder lane · 2026-06-28 Trigger: systemic success_fn false positives confirmed (75/81 success_fns key on output/command keywords + tool-ran, not real achievement). Proven: 404→SUCCESS, no-payload-XSS→SUCCESS, failed-msf→SUCCESS. Detail + per-class spec: AUDIT-success-fns-false-positives.md.

Why a full program (not a patch)

The false-pass is in the eval's core success methodology, so everything downstream inherited it:

Contamination chain 1. success_fns (75/81) → pervasive false positives 2. → baseline.csv (2026-06-14) generated under this → the pinned reference is corrupt 3. → playbook cache seeded from "successful" runs → poisoned with false-success entries 4. → ft corpus + Tier-2 scores → fine-tune data contaminated (Tier-2 judge reads the same transcripts → also suspect) 5. → all recent eval results (ablation, quick-eval, model comparisons) → unreliable

Phased plan

  • Phase 0 — Freeze (now). Don't promote any baseline; don't seed the playbook; quarantine recent eval results; tag ft/Tier-2 data from the buggy regime as suspect. Add the known-false regression set (404-XSS, no-payload-XSS, failed-msf, OptionValidateError-msf) — every fix must score these FAIL.
  • Phase 1 — Redefine "success." Adopt one rule: success = independently-verified achievement, evidence from output only, tool-ran necessary-not-sufficient. Canonical gate = a per-objective verify_fn (state probe); keyword match demoted to secondary. (Per-class criteria in the audit.)
  • Phase 2 — Rebuild success_fns to that rule. Order by impact: (a) Class C → _real_output() (~20, mechanical); (b) Class B web-injection re-fetch verify (~5–8); (c) Class A exploit/auth/privesc/persistence (~30, author criteria); (d) recon/scan last.
  • Phase 3 — Clean rebaseline. Full sweep with fixed success_fns → new baseline.csv; Auditor certifies against the regression set.
  • Phase 4 — Decontaminate downstream. Re-validate/purge playbook entries whose source runs are now false-fails (#1029 revalidate tooling); re-score or quarantine the Tier-2/ft corpus; decide retrain-vs-keep.

Lanes

  • ARCHER-Coder: Phase 2 success_fn rebuild + Phase 1 verify_fn framework.
  • Auditor: Phase 1 standard sign-off; Phase 3 rebaseline certification; owns the known-false regression set.
  • Operator: sequences phases; Phase 4 retrain-vs-keep call.
  • sag/Coder (me): produced this audit/spec; can build the known-false regression harness (Phase 0) and the per-function table if useful — but the rebuild is the ARCHER-Coder's.

Immediate next 3

  1. Freeze baseline-promotion + playbook-seeding (Phase 0).
  2. Coder + Auditor agree the Phase-1 success standard (the audit's per-class table is the draft).
  3. Start Phase 2 with Class C (_real_output() switch — cheapest, immediate trust gain) while the regression set is built.

Open decisions for the operator

  • Retrain-vs-keep on the contaminated ft corpus (Phase 4) — likely retrain after rebaseline, but cost/benefit is yours.
  • Whether to nuke + rebuild the production playbook vs. revalidate-in-place (#1029 tooling supports the latter).
  • Scope of the rebaseline sweep (all 29 objectives vs. a prioritized core first).