I built a local GPU benchmark server for 4B to 35B models because I was tired of synthetic cloud leaderboards and tiny sample tests

Hi everyone, whenever I looked at existing LLM leaderboards I was missing practical numbers for running quantized open weight models on affordable local hardware. Most tables only…

Hi everyone, whenever I looked at existing LLM leaderboards I was missing practical numbers for running quantized open weight models on affordable local hardware.Most tables only copy vendor cloud scores or run tiny subsets of a benchmark, and none of them tell you how long a model actually sits there calculating before it finishes.

Because nothing like that existed, I set up a dedicated Linux test server with two RTX 5060 Ti cards giving 32 GB of VRAM in total, running vLLM in Docker.Over the past weeks I ran 12 current local models from 4B up to 35B through complete benchmark datasets without any subsampling, including IFEval, IFBench, ARC AGI 1, SimpleQA, SimpleQA Verified, SciCode and vLLM bench serve.

The lineup includes Qwen 3.5, Qwen 3.6 and Qwen 3.8 in 4B, 9B, 27B and 35B A3B, Google Gemma 4 in E4B, 26B A4B and 31B, plus GLM 4.7 Flash in 4 bit quantizations like NVFP4 and AWQ.Every single model is tested twice across all benchmarks, once with thinking turned off and once with thinking turned on, using the official sampling parameters.

While the tests run, my system tracks the exact wall clock duration and throughput in tokens per second to calculate a time efficiency score from 0 to 100 for every single cell.That turned out to be the most interesting part of the whole project.

You can immediately see which models deliver strong scores in 20 or 30 minutes and which ones burn GPU power for 10 hours straight because they get stuck reasoning in circles on simple questions.Especially the MoE models like Qwen 3.6 35B A3B and Gemma 4 26B A4B surprised me by finishing in a fraction of the time while beating or matching the dense 27B and 31B models.

I put all results into a clean, sortable comparison table on a new subpage of my site at https://llmbenchmarks.io/local-gpu-benchmarks/ so anyone with a 16 GB or 32 GB setup can check the numbers before downloading 20 GB of weights.

The site is a personal project with no ads, no tracking and no affiliate links, and new benchmarks like Chess Puzzles are already running on the server right now.Feel free to take a look and let me know which local models you would like to see tested next.

submitted by /u/Careful-Report6526 [link] [comments]