Weights-only 2-bit post-training quantisation (W2), measured end to end against the FP16 reference and against off-the-shelf 2-bit baselines. Source: bundle/report.md and bundle/tables/*.csv, measured 2026-10-09 to 2026-10-11.
Math reasoning (GSM8K, two prompt styles), code generation (HumanEval, MBPP), instruction following (IFEval), factual recall (TriviaQA), and tool calling (BFCL — simple, multiple, parallel, parallel-multiple, irrelevance, multi-turn), averaged into one score per model.
12-task average by method
percent correct · bits/weight noted below each bar
| Method | Bits/weight | 12-task avg |
|---|---|---|
| fp16 | 16 | 80.6 |
| Invariant | 2.19 | 72.0 |
| kusakana | ~2.25 | 40.5 |
| imatrix g256 | 2.19 | 17.8 |
| Task | fp16 | Invariant | imatrix g256 (2bit) | kusakana |
|---|---|---|---|---|
| GSM8K nothink | 91.7 | 81.4 | 18.6 | 71.1 |
| GSM8K think | 83.0 | 84.9 | 11.1 | 62.5 |
| HumanEval | 82.9 | 66.5 | 20.1 | 51.2 |
| MBPP | 66.6 | 50.8 | 16.4 | 44.2 |
| IFEval | 81.7 | 74.9 | 32.2 | 44.5 |
| TriviaQA | 56.7 | 32.4 | 14.4 | 25.8 |
| BFCL simple | 94.5 | 94.0 | 0.3 | 37.5 |
| BFCL multiple | 96.5 | 91.5 | 1.0 | 48.0 |
| BFCL parallel | 94.0 | 90.0 | 0.0 | 2.0 |
| BFCL par-mult | 91.0 | 89.5 | 0.5 | 3.0 |
| BFCL irrelev. | 83.8 | 82.1 | 99.2 | 94.2 |
| BFCL multi-turn | 45.0 | 26.0 | 0.0 | 1.5 |
The closest honest comparison isn't same-method-different-bits — it's letting the standard method spend one more bit: imatrix needs 3.25 bits/weight (74.1) to land ahead of Invariant's 2.19 bits/weight (72.0). At Invariant's own 2.19-bit budget, the same off-the-shelf method (imatrix g256) scores 17.8.
One GPU, one session, at 64 concurrent users and at a single user. Each line is cumulative problems solved as time passes — the slope is a solved/min number, made visible.
GSM8K — cumulative solved
1319 problems
| Minutes | fp16 solved | Invariant solved |
|---|---|---|
| 0 min | 1 | 1 |
| 5 min | 263 | 804 |
| 10 min | 507 | 1184 |
| 15 min | 766 | 1184 |
| 20 min | 990 | 1184 |
| 25 min | 1249 | 1184 |
| Minutes | fp16 solved | Invariant solved |
|---|---|---|
| 0 min | 1 | 6 |
| 10 min | 37 | 172 |
| 20 min | 78 | 392 |
| 30 min | 111 | 577 |
| 40 min | 141 | 763 |
| 45 min | 157 | 852 |
HumanEval — cumulative solved
164 problems
| Seconds | fp16 solved | Invariant solved |
|---|---|---|
| 0s | 1 | 1 |
| 5s | 42 | 46 |
| 10s | 103 | 108 |
| 15s | 136 | 108 |
| 20s | 136 | 108 |
| Minutes | fp16 solved | Invariant solved |
|---|---|---|
| 0.0 min | 1 | 1 |
| 1.0 min | 46 | 109 |
| 2.0 min | 80 | 109 |
| 3.0 min | 106 | 109 |
| 4.0 min | 121 | 109 |
| 4.5 min | 136 | 109 |
GSM8K at batch 64. Three sampled attempts per problem (temperature 0.6); a problem counts when its slowest attempt finishes. Majority vote is the served result; any-of-3 is an upper bound.
GSM8K — cumulative solved, 1 attempt vs 3 attempts
1319 problems
| Minutes | fp16 — 1 attempt | Invariant — 1 attempt | majority of 3 | any of 3 (oracle) |
|---|---|---|---|---|
| 0 min | 1 | 1 | 1 | 1 |
| 5 min | 263 | 804 | 297 | 307 |
| 10 min | 507 | 1184 | 597 | 621 |
| 15 min | 766 | 1184 | 893 | 928 |
| 20 min | 990 | 1184 | 1159 | 1204 |
| 25 min | 1249 | 1184 | 1204 | 1251 |
Per-request throughput and the kernel sweep behind the headline numbers above.
Full GSM8K (1319 problems, think, 8192 cap) and HumanEval (164 problems), timed end to end — generation only, graded after. fp16 vs. Invariant, solved per minute, log scale.
GSM8K — solved per minute
1319 problems, 45-min budget per run · log y-axis
| Concurrency | fp16 (solved/min) | Invariant (solved/min) | Invariant ÷ fp16 |
|---|---|---|---|
| 1 user | 3.5 | 18.9 | 5.4× |
| 64 users | 48.0 | 123.3 | 2.6× |
HumanEval (coding) — solved per minute
164 problems · log y-axis
| Concurrency | fp16 (solved/min) | Invariant (solved/min) | Invariant ÷ fp16 |
|---|---|---|---|
| 1 user | 30.3 | 112.8 | 3.7× |
| 64 users | 371 | 540 | 1.5× |
Published kernel numbers (RTX 4090, vLLM, 64-token decode) for the packed 2-bit artifact against the same weights served dense in fp16 — only the matmul differs. The 2-bit advantage is a bandwidth story: it decays as batching amortizes the weight read, reaching parity at 256 concurrent sequences.
Decode throughput vs. concurrency
tokens / second, log concurrency axis
| Concurrency | Packed 2-bit (tok/s) | Dense fp16 (tok/s) | Speedup |
|---|---|---|---|
| 1 | 322 | 59 | 5.46× |
| 4 | 1246 | 226 | 5.51× |
| 16 | 3452 | 890 | 3.88× |
| 32 | 5327 | 1703 | 3.13× |
| 64 | 6761 | 3104 | 2.18× |
| 128 | 7695 | 5353 | 1.44× |
| 256 | 8130 | 8110 | 1.00× |