Invariant · Quantisation Compiler

Qwen3-8B at 2 bits — results

Weights-only 2-bit post-training quantisation (W2), measured end to end against the FP16 reference and against off-the-shelf 2-bit baselines. Source: bundle/report.md and bundle/tables/*.csv, measured 2026-10-09 to 2026-10-11.

Accuracy

12-task accuracy

Math reasoning (GSM8K, two prompt styles), code generation (HumanEval, MBPP), instruction following (IFEval), factual recall (TriviaQA), and tool calling (BFCL — simple, multiple, parallel, parallel-multiple, irrelevance, multi-turn), averaged into one score per model.

12-task average by method

percent correct · bits/weight noted below each bar

fp16 reference
Invariant — 2bit
imatrix g256 — 2bit, off-the-shelf
kusakana — 2bit, third-party
02040608010080.6fp1616 bpw72.0Invariant2.19 bpw40.5kusakana~2.25 bpw17.8imatrix g2562.19 bpw
View as table
MethodBits/weight12-task avg
fp161680.6
Invariant2.1972.0
kusakana~2.2540.5
imatrix g2562.1917.8

The closest honest comparison isn't same-method-different-bits — it's letting the standard method spend one more bit: imatrix needs 3.25 bits/weight (74.1) to land ahead of Invariant's 2.19 bits/weight (72.0). At Invariant's own 2.19-bit budget, the same off-the-shelf method (imatrix g256) scores 17.8.

Throughput

Problems solved, against the clock

One GPU, one session, at 64 concurrent users and at a single user. Each line is cumulative problems solved as time passes — the slope is a solved/min number, made visible.

GSM8K — cumulative solved

1319 problems

fp16
Invariant
02004006008000m15m30m45mfp16 — 157Invariant — 852
View as table (sampled points)
Minutesfp16 solvedInvariant solved
0 min16
10 min37172
20 min78392
30 min111577
40 min141763
45 min157852

HumanEval — cumulative solved

164 problems

fp16
Invariant
040801200m2m4mfp16 — 136Invariant — 109
View as table (sampled points)
Minutesfp16 solvedInvariant solved
0.0 min11
1.0 min46109
2.0 min80109
3.0 min106109
4.0 min121109
4.5 min136109
Attempts

Three attempts, against the clock

GSM8K at batch 64. Three sampled attempts per problem (temperature 0.6); a problem counts when its slowest attempt finishes. Majority vote is the served result; any-of-3 is an upper bound.

GSM8K — cumulative solved, 1 attempt vs 3 attempts

1319 problems

fp16 — 1 attempt
Invariant — 1 attempt
Invariant — majority of 3
Invariant — any of 3 (oracle)
0250500750100012500m5m10m15m20m25mfp16 — 12531 attempt — 1184majority of 3 — 1204any of 3 — 1251
View as table (sampled points)
Minutesfp16 — 1 attemptInvariant — 1 attemptmajority of 3any of 3 (oracle)
0 min1111
5 min263804297307
10 min5071184597621
15 min7661184893928
20 min990118411591204
25 min1249118412041251
Appendix

Supporting results

Per-request throughput and the kernel sweep behind the headline numbers above.

What the speed is worth, in problems solved

Full GSM8K (1319 problems, think, 8192 cap) and HumanEval (164 problems), timed end to end — generation only, graded after. fp16 vs. Invariant, solved per minute, log scale.

GSM8K — solved per minute

1319 problems, 45-min budget per run · log y-axis

fp16
Invariant
1310301003003.518.95.4×INVARIANT / FP161 user48.0123.32.6×INVARIANT / FP1664 users
View as table
Concurrencyfp16 (solved/min)Invariant (solved/min)Invariant ÷ fp16
1 user3.518.95.4×
64 users48.0123.32.6×

HumanEval (coding) — solved per minute

164 problems · log y-axis

fp16
Invariant
1030100300100030.3112.83.7×INVARIANT / FP161 user371.0540.01.5×INVARIANT / FP1664 users
View as table
Concurrencyfp16 (solved/min)Invariant (solved/min)Invariant ÷ fp16
1 user30.3112.83.7×
64 users3715401.5×

The kernel underneath: bandwidth-bound → compute-bound

Published kernel numbers (RTX 4090, vLLM, 64-token decode) for the packed 2-bit artifact against the same weights served dense in fp16 — only the matmul differs. The 2-bit advantage is a bandwidth story: it decays as batching amortizes the weight read, reaching parity at 256 concurrent sequences.

Decode throughput vs. concurrency

tokens / second, log concurrency axis

packed 2-bit
dense fp16
0200040006000800014163264128256
View as table
ConcurrencyPacked 2-bit (tok/s)Dense fp16 (tok/s)Speedup
1322595.46×
412462265.51×
1634528903.88×
32532717033.13×
64676131042.18×
128769553531.44×
256813081101.00×