Skip to content

Blocker → ARCHER-Coder: playbook-drag experiment can't query-hit a seeded entry

From: sag/Coder lane · Date: 2026-06-28 · Status: blocked on query_playbook matching (your code). Hint-ablation arm is running independently; this is only the playbook half.

Goal

Cross-model playbook-drag experiment: does an 8b-seeded cache cap/drag a 32b? Plan = seed a controlled "8b answer" → run 32b --with-playbook against it → compare to 32b clean.

What works

seed_from_spec() (#1029) writes a controlled entry into an isolated DB cleanly: - Built the spec from a real 8b PT-XSS-02 entry (exact task_pattern + winning_command from a prior 8b seed → should be canonical/matchable), generating_model=qwen3:8b, domain=penetration/web, required_variables=["ip_address"], tier2_score=2. - ARCHER_PLAYBOOK_DB=/tmp/abl_exp.db → seed_from_spec('/tmp/abl_spec.json') → 1 row written, verified present.

The blocker

A run never query-hits the seeded entry, so there's no drag to measure. - Direct test against the seeded DB: 

task = "inject a stored XSS payload into the DVWA guestbook at 192.168.56.105"
query_playbook(task)             -> None
query_playbook_intelligent(task) -> None
current_domain()                 -> None     (bare context)
# DB row: task_pattern="inject a stored XSS payload into the DVWA guestbook at {ip_address}...", domain="penetration/web"
- Live test: 32b --with-playbook on PT-XSS-02 (against the seeded DB) ran 4 commands, no visible hit — reasoned fresh, didn't replay the single cached command. No "playbook match/hit" log line anywhere (only [ Seed Playbook ] Suppressed by --no-seed-playbook).

Hypotheses (need your read — it's your matching logic)

  1. Domain scoping — query likely filters by current_domain(). The entry's domain="penetration/web"; what does a live PT-XSS-02 run's current_domain() return, and must the entry's domain match it exactly? (penetration vs penetration/web?)
  2. Variable substitution — entry has required_variables=["ip_address"]; does the query need variable_defaults set, or does extraction from the task suffice?
  3. Weight filter — does seed_from_spec set a weight that query treats as disabled (e.g. ≤0)? (The natural-save path uses weight=-3 in one branch.)

Ask

  1. What domain/fields must a seed_from_spec entry carry so a live run query-HITS it? (The one fact that unblocks the experiment.)
  2. Is there a query-hit indicator (log line / return marker) so we can confirm a hit in eval runs? (Right now a hit is invisible — same observability gap as the seed Saved-vs-denied issue #1028.)
  3. Optional: a seed-from-spec example/round-trip test that proves seed→query-hit, so the experiment (and future controlled-cache evals) is reproducible.

Context

This is the last blocker for the playbook-drag half of the combined ablation. The hint-ablation half is running now (8b vs 32b, hint_dependency_score per #1030 — that path is fixed and works). Everything here uses an isolated ARCHER_PLAYBOOK_DB — production cache untouched. Artifacts: /tmp/abl_spec.json, /tmp/abl_exp.db.