A placement compiler for quantized MoE models

PLACEMENT,COMPILED.

I wanted the Q6 model—not the smaller quant my GPU was forcing me into. ExpertFlow finds the hidden CPU expert bottleneck and spends VRAM where it removes the most work.

RUN THE PROOFVIEW SOURCE ↗
28.13DECODE TPS+22.48% over strongest stock
Strongest stock22.967 TPS
Matched protocol10 × 512 tokens
Process-owned peak10,966.801 MiB
Target16 GB GPU · Q6
01 / BOUNDARY
Hidden CPU expert bottleneck

The layer said GPU.
The expert work said CPU.

Whole-layer placement was too coarse. ExpertFlow identifies the routed operands separately and keeps the selected complete expert banks close to CUDA.

Stock whole-layer placement

The costly crossing

ROUTER
GPU
EXPERTS
CPU
OUTPUT
GPU

Router on GPU → expert matmuls on CPU → copy the result back.

→
ExpertFlow expert-bank placement

The compiled route

ROUTER
GPU
EXPERTS
GPU
OUTPUT
GPU

Selected complete packed banks remain CUDA-resident for execution.

02 / PROOF
Replay → Live → Rebuild

Do not take the number on faith.

Choose the proof path that matches the machine and the time available.

Any computer · no model

Replay the evidence

Hash-check the committed artifacts and reconstruct the measured release result.

uv sync --frozen
uv run expertflow demo --replay
Compatible NVIDIA system

Run the live matched test

Verify identities, launch fresh matched processes, and preserve raw evidence.

.\scripts\live-tps-demo.ps1 -Mode Demo
03 / OUTPUT
Hardware-specific compiler output

Twelve complete banks.
One measured plan.

The selected layers are not a universal hand-picked recipe. They are the emitted Q6 placement for this model, memory budget, and GPU.

12

full 128-expert CUDA banks

Static shadow7,849.881 MiB
Placementbefore graph build
Per-token transfernone
CUDA graphsenabled
30-layer Gemma routed stack
04 / DECISION
The evidence changed the product

The clever cache lost.
The placement compiler shipped.

Simulation

NO CACHE OPPORTUNITY

Measured Q6 routing plus measured cache costs projected lower throughput.

→
Measured recorded evidence

Full static residency

No eviction, reactive loading, prediction, repacking, or per-token expert transfer.

05 / BUILD
Codex + GPT-5.6 workflow

Every failed hypothesis became the next decision.

GPT-5.6 guided ideation and experiment framing. Codex with GPT-5.6-sol managed the implementation, testing, measurement, failure isolation, packaging, and release loop.

IDEATEframe the bottleneck
INSTRUMENTobserve the real path
VERIFYparity, memory, TPS
DECIDEfollow measured gates
SHIPCLI, replay, runtime
OPEN THE EVIDENCE LEDGER ↓
Quality evidence

49/100 → 50/100 MMLU

PPL point: -2.92%; 95% upper bound: +2.25%. The strict +1% confidence requirement was not met.

Live benchmark

35.6699 aggregate TPS

Four slots, 20/20 requests; stock was 24.5231 TPS. Concurrent outputs were not fully deterministic.

Context profile

262,144 allocated tokens

417 tokens were processed in the bounded run. Peak was 15,635.582 MiB with 675.418 MiB reserve.