Rig OLLAMA_NUM_PARALLEL sweep + 24GB VRAM ceiling (#1023)¶
Date: 2026-06-24 · Harness: scripts/stress_dual_system.py + run_1023_* runners · rig-local (no link confound).
Raw: raw/stress-numparallel-8b.jsonl, raw/stress-numparallel-14b.jsonl. (Regenerated 2026-06-28 after the original CSL-OPS copy was lost to an uncommitted-repo reset.)
⚠ Corrected 2026-07-17 — see the update at the bottom. The "32B = NP=1 only" guidance in the tables below is superseded: a direct NP=2 measurement shows 32B does batch at NP=2 (23.1 GB at
num_ctx=8192, fits, 1.7×). The 06-24 32B rows were extrapolated from the single-slot footprint, not measured. Original text kept intact below per the append-only-record principle.
qwen3:8b — aggregate throughput (tok/s) by NUM_PARALLEL × concurrency¶
| NP | C=1 | C=2 | C=4 | C=8 |
|---|---|---|---|---|
| 1 | 125 | 129 | 130 | 130 |
| 2 | 124 | 213 | 217 | 218 |
| 4 | 73 | 134 | 189 | 191 |
NP=1 plateaus at single-stream (~130); NP=2 ~1.7× aggregate with best tail latency; NP=4 regresses (solo collapses to 73 t/s, aggregate below NP=2).
qwen3:14b (prod model) — NP sweep + GPU memory¶
| NP | agg C=2 | agg C=4 | solo t/s | GPU used |
|---|---|---|---|---|
| 1 | 78 | 78 | 75 | 14.2 GB |
| 2 | 131 | 133 | 75 | 19.1 GB |
| 3 | 71 | 68 | 38 ⚠ | 23.0 GB |
| 4 | 34 | 50 | 18 ⚠⚠ | 22.9 GB |
NP=2 wins on 14b too (~1.7×, comfortable 19GB). NP=3/4 hit the 24GB wall → memory pressure collapses throughput.
The 24GB context/parallelism wall by model size¶
| model | usable ceiling on 24GB |
|---|---|
| 8b | NP up to 4+, ctx to 32K (fits easily) |
| 14b | NP=2 comfortable (19GB); ctx to 32K fits |
| 32b | NP=1 / ctx ~8K only — 22GB at 8K, spills to CPU at 16K (24GB) / 32K (29GB) |
Recommendation¶
OLLAMA_NUM_PARALLEL=2on the rig for eval/offload — optimal for both 8b and 14b; do not exceed 2 on this card.- Pin 32b at
num_ctx=8192, NP=1 — never trigger a CPU spill. 32b alone = 22 of 24GB, so no headroom for more context or a second resident model. OLLAMA_MAX_LOADED_MODELS: model switch costs ~3.3s reload; keep 1 for 32B, 2 only for mixed 8B+14B work. Route same-model work together.
2026-07-17 update — NP=2 re-confirmed on current stack; 32B correction; ROG contention (2a)¶
Re-ran with
scripts/stress_dual_system.py(now pinsnum_ctx=8192). Full analysis + raw JSONL: CSL-OPSresearch-scratch/dev-rig/stress-1023-32b-and-14b-2026-07-17.md.
NP=2 holds on both models (num_ctx=8192, all 100% GPU, 0 failures):
| Model | NP=1 agg (C=2) | NP=2 agg (C=2) | NP=2 GPU |
|---|---|---|---|
| qwen3:14b | 77.5 | 131.7 | 11.6 GB |
| deepseek-r1:32b | 36.7 | 61.9 | 23.1 GB |
32B correction (supersedes the "NP=1 only" rows above): 32B does batch at NP=2 — the 8K KV slot
is ~1 GB, so 2 slots + weights = 23.1 GB (fits under 24 GB), 1.7× throughput, no spill. The 06-24 rows
extrapolated from the single-slot footprint. Do NOT exceed NP=2 or raise ctx (94% VRAM → NP=3 / >8K
OOMs). OLLAMA_NUM_PARALLEL=2 is optimal for 8b, 14b, AND 32b — set it as the standing rig config.
2a — ROG-local vs rig, prod model: ROG-local qwen3:14b @ :11434 under range load (7 VMs) =
5.1 t/s (128-tok gen, 24.9 s), vs the rig's ~75 — a ~15× gap. ROG runs 14b/8b fine (it is the
baseline ARCHER config); it is just slow at 8 GB (the 9.3 GB model partial-offloads to CPU, and the
range VMs contend for it). So-what: offload throughput-bound work (full sweeps, data collection,
parallel eval arms) to the rig, where the 15× + NP=2 batching pay off; a single interactive ROG session
is fine as before. This measures the offload payoff, not a claim that ROG can't inference.
(Correction 2026-07-17: removed an earlier "crippled / never serve on ROG" overstatement — see the
interpretation-correction note in the dev-rig doc. Clean range-down baseline not taken.)
Eval-objective throughput — rig vs rog (2026-07-17, #1036 verification)¶
Clean per-objective wall-clock on real inference-bound (curl-based) objectives — warm model,
ARCHER_BACKEND_FORCE=rog for ROG (the #1098 guard blocks single-arm ROG evals otherwise):
| Objective | cmds | rig | rog | rog/rig |
|---|---|---|---|---|
| PT-WEBENUM-02 | 2 | 11.2s | 16.8s | 1.5× |
| PT-XSS-01 | 2 | 9.5s | 50.5s | 5.3× |
| PT-PIVOT-03 | 6 | 25.1s | 170.9s | 6.8× |
Key finding: the 15× raw inference ratio is muted to 1.5–7× at the objective level, and capped there.
Eval-objective wall-clock is dominated by fixed overhead (setup, curl/chisel/ssh, oracle scoring), not
inference. More steps → higher ratio (2-cmd 1.5–5.3×; 6-cmd pivot 6.8×) but it plateaus ~7×, not 15×,
because per-step tool time is backend-independent. Where workloads land: eval/data-gen ~2–7×;
training/fine-tuning gets the full 15×+ and is the rig's maximal use (pure GPU, zero tool overhead,
and ROG can't do it at all); serving 32B is a capability ROG lacks. So offload inference-heavy data-gen,
put training on the rig, leave short/tool-bound eval + interactive work on ROG. Full analysis: CSL-OPS
dev-rig/stress-1023-32b-and-14b-2026-07-17.md.