Comparing ISTA-DASLab's DeepSeek-V4.1-Flash-1017-GSQ-RCO-GGUF 2.25bpw against the official API

I've been running ISTA-DASLab's 2.25bpw quant of DSv4.1-1017 for a few days, and it's holding up well. But the fact that it's only 2.25bpw average did make me wonder how good the q…

I've been running ISTA-DASLab's 2.25bpw quant of DSv4.1-1017 for a few days, and it's holding up well.But the fact that it's only 2.25bpw average did make me wonder how good the quant actually is against the full model.

Local setup: Project-maya v1.1.27.ISTA-DASLab's 2.25bpw quant.Quality-relevant settings: "DS4_INDEX_TC": "1"."MAYA_DS4_PROMPT_SKIP": "2".Test: 100 question text-only subset of Humanities Last Exam.Benchmarking models isn't easy, and these results are nothing but a smoke test.

There are lots of issues with these results: 1) I ran the questions with a 32768 token limit per question, because anything larger becomes prohibitively time consuming.It already took my local instance around 12 hours to complete the test, and that's at an average of 40.4 toks/s.

2) When approaching the token limit, Deepseek's API and project-maya work differently.Project-maya will reduce thinking_budget to around 3/4 of the token limit so it always gives an answer, but as a result it never used more than around 24768 tokens on a question.

The Deepseek API will instead just use all 32768 tokens thinking if it needs and then output nothing.This really fucks with the results on a lot of questions unfortunately.There were some questions local answered correctly because it was forced to give a result.

There were others that the API answered correctly while using between 24k-32k tokens, where the local model was wrong while being capped at 24k.3) I ran on thinking "high", not "max" for both, because Deepseek's API was thinking too much and failing to output anything for most questions on "max".

Even on thinking "high", the API failed to answer 26 questions and thinking was cut short to 24k on a similar number of local questions.4) This is only a 100 question subset, and I saw significant run-to-run variance on both the official API and my local model too.

There are some questions that both sides will sometimes pass, sometimes fail using the settings as tested.To get proper statistically relevant comparisons you'd need to run this test 10 times or probably more.

The results themselves: Local mode: 13 correct "strict" answers, 20 semantically correct answers.API: 13 correct "strict" answers, 21 semantically correct answers.On the 74 questions the official API actually completed: Local semantic: 19/74 Official semantic: 21/74 Both correct: 17 Local only: 2 Official only: 4 Neither: 51 It's also worth noting that when both were wrong, the wrong answer was identical 23 times, and semantically the same in 11 more.

7 more questions were wrong "simiarly" IE both picked fungi but chose different species.Conclusion: Given the flaws in my testing, it's impossible to say specifically how close the quant is to the full fat model.

You'd need to burn hundreds of millions of tokens to really know with certainty and that's just on Humanities Last Exam.However what I can say fairly confidently now is that the 2.25bpw quant isn't completely lobotomized.

Anything the API could answer, the local model was there or there abouts.For example I'm pretty confident that if I took any question the API could answer, and asked my local model with a 256k token limit it'd probably answer the same.

Both sides are capable of a much better score with unlimited token budget and max reasoning.submitted by /u/PhysicalIncrease3 [link] [comments]