DeepSeek-V4.1-Flash-1017 at 41 tok/s on one 3090
No, this is not a lobotomized 1-bit squeeze, its full Q2 nazeshinjite's DeepSeek-V4.1-Flash-1017 Q2 GGUF on a single 3090, natively on Windows no WSL. On an RTX 3090 with 192 GB of…
No, this is not a lobotomized 1-bit squeeze, its full Q2 nazeshinjite's DeepSeek-V4.1-Flash-1017 Q2 GGUF on a single 3090, natively on Windows no WSL.On an RTX 3090 with 192 GB of RAM I'm writing at 41 tok/s (average across multiple prompts) and the output agrees with DeepSeek's own API on the next token 90.2% of the time on general text and 92.4% on code/math, aka within a point of exact routing.
Fits full 1M context.My scuffed interface engine runs the three most-downloaded quants of 1017, their files as they are: nazeshinjite's ds4 Q2 GGUF (the default) ISTA-DASLab's GSQ-RCO (IQ2_XS / IQ2_XXS /IQ1_M) Unsloth's (UD-IQ1_S to UD-Q8_K_XL).
Repo: https://github.com/musicurgy/delta-inference-engine Hardware/model: RTX 3090 24 GB, i5-13600K, 192 GB DDR5 (4x48 GB at 5600), a PCIe 4.0 NVMe.Windows 10/11, no WSL.Quant Download size 192 GB RAM 128 GB RAM Top-1 vs API (general text, exact routing) ds4 Q2 (default) 366 GB 41 tok/s 33 tok/s 91.5% GSQ IQ2_XS 159 GB + the default 38 tok/s 26 tok/s 90.0% GSQ IQ2_XXS 142 GB + the default 41 tok/s 32 tok/s 87.8% GSQ IQ1_M 125 GB + the default 42 tok/s 37.5 tok/s 85.5% Unsloth UD-IQ1_S 233 GB 40 tok/s 37.5 tok/s 89.3% Setup: Download the ZIP (or clone), double-click Start DeepSeek.bat .
The first run installs Python if needed, downloads the prebuilt engine for your CPU and the model (every file SHA-256-checked, the space checked first), and starts the server; after that the same file just starts it.
Choose Model.bat switches quants.If anything goes wrong, Start DeepSeek (debug).bat packs everything needed into one zip for an issue (no prompts, user name removed).Privacy: localhost only by default, and nothing from your prompts is written to disk unless you turn a log on.
Needs: An NVIDIA RTX card with 12 GB+ (24 GB: 41 tok/s, 16 GB: 31 tok/s, 12 GB: 17 tok/s, measured on my 3090 being memory limited) .GTX 10 series and older can't run it 128 GB+ RAM.A fast NVMe SSD with 385 GB free Any x86-64 CPU (AVX2 is the fast path; older CPUs get a slower build automatically).
Credits: The engine is a Windows port of Xolotl , itself built on Strata ; several of Xolotl's latest fixes are merged back in.DeepSeek for the model, nazeshinjite for the GGUF, ISTA-DASLab for GSQ-RCO, Unsloth for their quants.
MIT licensed.Special thanks to Claude for doing most of the lifting.If there is interest I'll add some support for muti GPU or other garbage.This project is very new and needs a lot of validation on other hardware, these numbers are from my machine, feel free to post yours.
submitted by /u/musicurgy [link] [comments]