4 coding agents on one RTX 3090: ~290 tok/s total from a custom CUDA engine for Qwen3.8-27B (3-4x llama-server --parallel)

I've been building OpenJet, a local coding agent with its own CUDA megakernel engine for Qwen3.8-27B Q4_K_M. Until now it served one conversation at a time. The new release serves…

I've been building OpenJet, a local coding agent with its own CUDA megakernel engine for Qwen3.8-27B Q4_K_M.Until now it served one conversation at a time.The new release serves several agents at once on a single GPU.

https://reddit.com/link/1xgs8hp/video/p0zd2rxse0yh1/player Numbers (RTX 3090, Qwen3.8-27B Q4_K_M ): per agent llama-server --parallel 4 OpenJet --agents 4 decode, short prompts 18-20 tok/s 64-72 tok/s decode, 23K context each 15-16 tok/s 52-55 tok/s first token of the next request on those contexts 13.4 s 0.2-0.5 s another agent decoding while a 20K prompt is processed 0.6 tok/s 9.5 tok/s That's 256-288 tok/s total across the 4 agents, vs 72-80 tok/s total for llama-server.

One agent alone still gets 117 tok/s.Four real coding agents, which pause to run tools and tests, total 150-200 tok/s.How the parallel decoding works: Generating tokens on a GPU is limited by memory bandwidth.

Every decode step reads all 16.7 GB of weights from VRAM, which on a 3090 (936 GB/s) takes about 18 ms on its own.The maths done on those weights is small by comparison.So the engine doesn't run the agents one after another.

Each step, it stacks every active agent's work into one batch of rows: each agent's next token plus its 2 MTP / prompt-lookup draft tokens, so 12 rows for 4 agents.It streams the weights through once, and each weight it loads is multiplied against all 12 rows with tensor cores while it's still on chip.

Four agents cost about one weight read per step, not four.That's why each agent drops from 117 to 64-72 tok/s instead of to ~30: the shared weight read is most of a step's cost.What does grow with more agents is attention (each agent reads its own KV cache) and the extra rows' compute.

The whole step runs as one persistent CUDA kernel with no per-layer launches, and the drafts are verified in the same pass, so every accepted draft is an extra token from the same weight read.What keeps it fast with real agents: New prompts don't stall everyone else.

When an agent sends a new prompt, each step carries the next chunk of it alongside the other agents' rows, so they keep generating while it's processed.The result is bit-identical to processing the prompt on its own.

One shared 132K-token KV pool (q8_0).It's paged, so one agent can use most of it while the others stay short.There's no fixed 4 x 32K split.Shared system prompts are stored once.A system prompt plus tool list of 1024 tokens or more that the agents have in common is processed and stored once for all of them.

Conversations stay in their slot.A follow-up request goes back to the slot already holding its conversation, so only the new tokens get processed.Also new: Sampled MTP drafts (thanks to iiLaurens): at temperature > 0, drafts are now sampled from the draft head and verified by rejection sampling.

The output distribution is unchanged and more drafts get accepted.mk_top: a live terminal view of each agent's tok/s, context use, current task and latest output, plus GPU stats.mk_top --run 4 starts the server and 4 agents on eval tasks so you can watch them.

Accuracy: the output is not byte-identical to llama.cpp, because the arithmetic is ordered differently.Against llama.cpp's own next-token distributions, KL divergence is in the same range as llama.cpp batched vs.

token-by-token.The README has the full correctness table.Try it (Linux, RTX 30/40/50 with 24 GB, CUDA 12+): python3 megakernel/server/mk_server.py -m Qwen3.8-27B-Q4_K_M.gguf --agents 4 python3 megakernel/tools/mk_top.py Then point up to 4 OpenAI-compatible agents at http://127.0.0.1:18080/v1 .

Limits: the Q4_K_M engine only (the mixed-quant build serves one conversation at a time), at most 8 agents, and presence/repetition penalties are ignored.Repo: https://github.com/l-forster/open-jet I am also planning on adding support for TurboQuant next, to get the max context window to 256K on a single card with minimal output degradation.

submitted by /u/Adorable_Weakness_39 [link] [comments]