Crafting Achievable Eval Objectives¶

Scope: how to design eval objectives that are achievable by design — i.e. that a capable agent can genuinely pass in one session against the live range, and whose success_fn can only be satisfied by real evidence. Companion to the objective rules in docs/roles/coder.md (success_fn-before-hint, health-checks-in-setup, pipeline stop conditions) and the failure taxonomy in the eval-integrity epic (#1037).

The failure class: "impossible by design"¶

An impossible-by-design objective is one that cannot be passed by any agent, regardless of capability — because the objective, its setup_fn, or the range precludes success. These are objective-design failures, distinct from genuine-model failures: no hint, no model improvement, and no oracle change will ever make them pass. The only fix is to redesign the objective or the range.

They are insidious because a false-passing oracle hides them — it scores them as wins on the model's command text or narration, so they look achievable and inflate the pass rate. You cannot see an impossible objective until the oracle scores on real evidence (the #1037 work is what surfaced the cluster below). Achievability auditing requires a trustworthy oracle first.

Confirmed instances (the seed set)¶

Objective	Why impossible	Class
PT-EXPLOIT-02 (#1041)	`setup_fn` pre-triggers a single-connection vsftpd backdoor → consumes the one shot before the agent acts	setup consumes the objective
PT-EXFIL-01 (#1043)	nc-listener exfil needs two concurrent processes (listener + sender); ARCHER is a one-shot, blocking, sequential agent	execution-model mismatch
PT-EXFIL-02 (#1044)	objective mandates sftp, which Metasploitable's SSH denies; only scp/ssh work	mandated method refused by target
original T2b (`b314266`)	required a root shell from an nmap NSE script — NSE detects, it cannot open a shell	success criterion the method can't yield

Root cause¶

Every one embeds a hidden assumption the system violates — about concurrency, resource lifecycle, target configuration, or what a tool can produce. The job of objective design is to make those assumptions explicit and check them against four hard constraints: the agent's execution model, the target's real configuration, the lifecycle of the resource under test, and the harness's lack of cross-objective sequencing.

The seven principles¶

One objective = one self-contained agent session. No cross-objective sequencing — the harness does not enforce "run X first" (the T2a→T2b dependency that started the vsftpd saga). A precondition must be established by this objective's own setup_fn or by the agent itself — never by assuming a prior objective ran.
setup_fn establishes preconditions but must not consume or complete the objective. If the precondition is a single-use or stateful resource (a single-connection backdoor, a one-time token), setup_fn can only reset/clean it — the agent must arm it. Pre-arming a single-use resource is self-defeating (PT-EXPLOIT-02).
Match the objective to the execution model: one-shot, sequential, blocking. A command runs to completion before the next starts. Patterns requiring concurrency — a listener held open while a sender runs — are impossible for a serial agent. Either setup_fn provides the persistent half (e.g. background the listener on Kali, verify the loot file afterward) or rescope the objective to a sequential method (PT-EXFIL-01).
Mandate the outcome, not a method — and validate the method against the live target first. Prefer "exfiltrate /etc/passwd" over "exfiltrate via sftp". If a specific method is required, confirm the target accepts it (docker exec -i archer-kali bash -c '<cmd>', 10 seconds) before locking the objective to it (PT-EXFIL-02 mandated a denied subsystem).
The achievability litmus: can you capture one genuine success transcript? If no expert (or the model) can produce a single real pass against the live range in one session, the objective is impossible by design. An open "capture a genuine success transcript" item is the tell. Write the success_fn first (existing rule) and prove one real pass exists before the objective enters the eval set.
Match the success criterion to what the chosen tool can actually produce. NSE scripts detect; they don't open shells. Requiring uid=0 from nmap --script can never fire. For each accepted tool path, confirm it yields output the success_fn accepts (this is the success_fn-validates-outcome rule, applied at design time).
You can't see impossibility until the oracle is honest. A false-passing success_fn reports an impossible objective as a win. Achievability auditing is meaningless on a broken oracle — fix and validate the oracle first (#1037 → #1038/#1039), then audit achievability.

Case study: the vsftpd saga (two rounds)¶

A clean example of how reasonable-looking fixes compound into impossibility:

Round 1 — a good fix. The original T2b asked an nmap NSE script to open a root shell (_t2_nmap_nse required uid=0 from nmap --script) — impossible (principle 6). Fixed by splitting exploit from detect: PT-EXPLOIT-01 = exploit→root via msfconsole (which triggers and shells in one connection); PT-EXPLOIT-02 = detect/confirm the backdoor (nc trigger + verify port 6200), no shell required. The split was correct — two legitimate, distinct skills, each matched to what its tool can do.
Round 2 — a bad fix (today's #1041). The detect objective needs the backdoor open. That created a cross-objective dependency on the exploit running first (principle 1) — which the harness can't enforce, so the detect objective produced 0 commands when the exploit failed (#177). The "fix" was to pre-trigger the backdoor in setup_fn to make detect self-contained. But the backdoor is single-connection, so the pre-trigger consumes it (principle 2). Impossible again.

The correct fix: keep the exploit/detect split; drop the setup_fn pre-trigger. The detect task already says "trigger it with nc and verify port 6200 opens" — the agent arms it (nc to :21) and verifies it (nc -z :6200) in one sequential session. setup_fn only resets any stale port-6200 state.

Pre-flight achievability checklist¶

Run this before adding or keeping any objective. A "no" on any item means redesign:

[ ] Self-contained: achievable in one agent session with no dependency on another objective.
[ ] Setup is non-consuming: setup_fn establishes state without completing or spending a single-use resource the agent must use.
[ ] Execution-model compatible: no concurrency required of the serial agent (or setup_fn owns the concurrent half).
[ ] Method works on the target: the required tool/method is accepted by the live target (docker exec verified), or only the outcome is mandated.
[ ] Tool can yield the criterion: every accepted tool path produces output the success_fn accepts.
[ ] Genuine transcript exists: one real success has been captured against the live range (the real gold anchor).

Relationship to existing rules¶

This guideline extends the objective rules already in docs/roles/coder.md — write the success_fn before the hint, health checks in setup_fn not hints, pipeline hints need a stop condition, zero-commands-is-context-not-logic — with an achievability / execution-model / target-compatibility lens. In the #1037 failure taxonomy (infra / oracle-strictness / objective-design / genuine-model), everything here is the objective-design quadrant: the fix is the objective or the range, never the hint or the model.