Deca Saas
Custom Ternary Quantization Kernel for llama.cpp / ggml
A CPU-only ternary ({−1, 0, +1}) quantization type and hand-tuned AVX2 kernel, built and evaluated end to end — from initial kernel design through an accuracy failure, its root-cause diagnosis, and an honest speed verdict.
The kernel works and is correct, but it is not faster than stock. Restricted to the right layers the model stays usable (PPL 94.81, 4.9× stock), yet it runs 5.4× (prompt) and 4.7× (generation) slower than stock Q4_K_M on this CPU.
Use stock Q4_K_M via Ollama / llama.cpp for real inference on this hardware. The value of this project is what it demonstrated, not a deployable speedup.
01Hardware & environment
| Component | Specification |
|---|---|
| Machine | Lenovo ThinkPad P50 |
| CPU | Intel Core i7-6820HQ (Skylake, 4 cores / 8 threads) — AVX2 / FMA3, no AVX-512 |
| GPU | NVIDIA Quadro M1000M/M2000M (2015 Maxwell) — no tensor cores, negligible llama.cpp/vLLM CUDA-backend support; treated as CPU-only inference target |
| RAM | 64 GB — the real ceiling on model size, not the GPU |
| OS / toolchain | Windows, MSVC (Visual Studio 18 2026 generator), CMake |
| Runtime base | Custom fork of llama.cpp / ggml (llama.cpp-custom), built alongside stock llama.cpp and Ollama for baseline comparison |
No AVX-512 rules out the wide-SIMD tricks modern quantization kernels lean on; the 2015-era discrete GPU is not a viable inference accelerator. This is a genuinely CPU-only, AVX2-ceiling optimization problem — closer to the constraints real edge and embedded deployments face than to a modern workstation.
02Goal
Design a custom 2-bit ternary quantization type (tern_custom: 32 weights per block, {−1, 0, +1} codes, 2.5 bits/weight — smaller than Q4_K_M's ~4.5 bits/weight) for ggml, wire it into a working llama.cpp build, and see how close a hand-optimized CPU kernel could get to (or beyond) stock llama.cpp / Ollama throughput, while keeping the resulting model actually usable.
03Kernel build-out (LUT steps 1–6)
Built incrementally, correctness-verified at every step before trusting a speed number.
| Step | What changed | Effect |
|---|---|---|
| 1–3 | Block format (block_tern_custom: {fp16 d; uint8 qs[8]}, 10 bytes per 32 weights), scalar quantize / dequantize / dot-product, wired into ggml's type-trait tables | Correct but slow reference implementation |
| 4 | Activation quantization changed from raw f32 to Q8_0-style int8 blocks (block_tern_q8_act), reusing one quantized activation row across every weight row instead of re-reading f32 every call | Reduced redundant work per mul_mat call |
| 5 | Wired TERN_CUSTOM into gguf-py so real GGUF files could be produced from an existing model | Made end-to-end model conversion possible |
| 6 | Hand-rolled AVX2 2×2 blocked microkernel using ggml's nrc/nrows mechanism (2 weight rows × 2 activation columns per call, amortizing unpack + load across 4 outputs) | Correctness-verified 1.89× prompt-processing speedup in isolated testing — see §7 for why this didn't hold up in the real build |
Structural dead ends investigated and ruled out
ggml's IQP panel-GEMM path is hardcoded to a fixed list of existing quant types and requires 256-element superblocks, incompatible with tern_custom's 32-weight blocks without a breaking redesign. llamafile_sgemm's templated tinyBLAS kernels are likewise hardcoded to a fixed type list in C++. Both would require substantial upstream-style rewrites, not kernel tuning, to support a custom type — so kernel work focused on the ordinary vec_dot path instead.
04First reality check: custom vs. stock, before any accuracy testing
Once step 6 was benchmarked, the initial throughput comparison against stock Q4_K_M looked like this (subsequently found to be overstated — see §7):
The direct question at this point was: “so should I use custom llama or oob [out-of-box] ollama?” The answer at the time was stock Ollama, given the real numbers. That recommendation held for the rest of the project.
05Stretch goal: “make custom at least 4× stock”
Explicitly flagged as very likely unreachable before attempting it, grounded in two things:
- BitNet.cpp's own published numbers never claim a speedup over stock on this class of comparison — only against fp16/fp32 baselines.
- A local lookup-table (TL1/TL2-style) experiment on this exact hardware showed only a ~1.6× ceiling over the existing AVX2 kernel — not 4× — because on this older, AVX2-only, no-AVX-512 CPU, memory-latency-bound table lookups don't out-pipeline well-scheduled SIMD compute.
The user chose to keep 4× as a stretch goal and proceed regardless.
06Accuracy investigation: a catastrophic failure, and its real cause
6.1 The discovery
Running llama-perplexity for the first time — previously only throughput had been measured, never output correctness — revealed the model was producing near-random output.
6.2 First hypothesis: double quantization
The model had been converted Q4_K_M → dequantize to fp32 → ternary-quantize, compounding two lossy, uncalibrated steps. The quantizer was fixed directly, with every change tested on real numbers rather than assumed.
- Calibrated per-block threshold — grid search over 7 threshold ratios per block, minimizing MSE, instead of a fixed 0.5× cutoff. Measured 25.7% lower reconstruction MSE on synthetic weight data.
- Correct scale estimator — per-block scale as mean(|x|) over only the nonzero-coded elements (the proper TWN formula), not the whole block including zeros.
- GPTQ-style error-feedback compensation — tested, then disabled. Implemented, then measured before shipping: it made MSE worse at every tested feedback strength (+17% at 0.5×, +34% at 0.7×). Real GPTQ's compensation is only valid because it is weighted by a per-layer Hessian from calibration data; two arbitrary adjacent 32-weight blocks have no such relationship, so the “compensation” just injects uncorrelated noise. Shipped disabled by default rather than kept for the sake of matching the original ask.
- Mixed precision — embeddings, output projection, and attention q/k/v/output projections kept at source precision; only FFN up/gate/down weights ternary-quantized.
All-FFN-tensors-ternary PPL with every fix applied: 1,312,384 — barely moved (5%) from the naive 1,380,009. The quantizer fixes were real and independently verified correct (30 sampled FFN tensors: uniform ~90% cosine similarity between original and reconstructed weights, a normal result for single-layer post-training ternary quantization) — but they were not the actual problem.
6.3 Root cause: 42 layers of compounding error
~18% per-tensor reconstruction error, individually reasonable, compounds through 42 stacked transformer layers' residual stream into near-total signal destruction at the output — the documented failure mode of post-training ternary quantization without fine-tuning. Real BitNet models are trained ternary from scratch or through quantization-aware fine-tuning; this is a fundamental limitation of quantizing a finished model after the fact, not a bug in this project's code.
6.4 The fix: layer-subset quantization, and a hard cliff
Added --ffn-layers to the converter, restricting ternary quantization to a specific range of transformer blocks. Count and position were swept across MiniCPM5-2B's 42 layers.
| Config | Layers quantized | PPL | Relative (log scale) |
|---|---|---|---|
| Stock Q4_K_M | 0 | 19.45 | |
| Middle 8 (17–24) | 8 | 24.89 | |
| Middle 16 (13–28) | 16 | 45.31 | |
| Middle 24 (9–32) — recommended | 24 | 94.81 | |
| Back 24 (18–41) | 24 | 434.87 | |
| Front 24 (0–23) | 24 | 50,724 | |
| Middle 30 (6–35) | 30 | 192,623 | |
| Middle 36 (3–38) | 36 | 1,609,348 | |
| All 42 | 42 | 1,312,384 |
Bar length is proportional to log₁₀(PPL); each step to the right is roughly an order of magnitude worse.
Two findings
- Front layers are catastrophically more sensitive than back layers, which are more sensitive than middle layers, at matched layer count — a >500× spread (front-24 = 50,724 vs. back-24 = 434.87 vs. middle-24 = 94.81) for the same number of layers, just different ones. Early layers build the base representations everything downstream depends on.
- There is a hard cliff, not a slope. Widening the quantized middle window from “protect 9 layers each edge” to “protect 6” jumps PPL ~2,000× (94.81 → 192,623) for losing protection on just 3 layers per edge — a small number of specific early layers are disproportionately load-bearing for model coherence.
--ffn-layers 9-32 — 24 of 42 layers ternary, first/last 9 protected. PPL 94.81 (4.9× stock) vs. ~71,000× for naive full-ternary. Model file: minicpm5-tern-24layers.gguf.
07Throughput of the accuracy-fixed model, and a real compiler bug
7.1 The honest number
| Stock Q4_K_M | Custom (9–32, 24/42 layers ternary) | Full ternary (all 42) | |
|---|---|---|---|
| Prompt processing | 48.0 tok/s | 8.85 tok/s (5.4× slower) | 5.386 tok/s (8.9× slower) |
| Generation | 15.3 tok/s | 3.25 tok/s (4.7× slower) | 2.359 tok/s (6.5× slower) |
The accuracy-fixed model is slower than stock, not faster — and mechanically it can't be otherwise: reducing the ternary footprint (needed for accuracy) removes FLOPs from the slow custom kernel; it doesn't add FLOPs to the fast stock kernel. “Correct” and “faster than stock” were never fully compatible goals on this hardware with this kernel.
7.2 A real bug found while chasing more speed
Investigating why the custom kernel was so much slower than stock surfaced a genuine, independent bug. ggml-tern-custom.c was compiled as part of the ggml-base CMake target (alongside architecture-generic files like ggml.c), not ggml-cpu — the only target that ever receives CPU arch flags (-march=native / /arch:AVX2). Compounding that, this is an MSVC build, and MSVC signals AVX2 via ggml's own GGML_AVX2 macro, not the GCC/Clang-style __AVX2__ the kernel's guards checked for.
Every #if defined(__AVX2__) branch in the kernel — all of LUT steps 4–6's AVX2 work, including the 2×2 blocked microkernel — was dead code in every real llama-bench / llama-perplexity run in this entire project, silently falling back to the scalar reference path. A separate standalone correctness/benchmark harness (CustomKernelTest.exe) never caught this because it set its own AVX2 compiler flag independently of the real build.
Fixed in both places (CMake target-level compile flags, and the source guards widened to check GGML_AVX2 too). The fix was independently verified to have actually compiled in — not just assumed from a rebuild log: the generated MSVC project file was inspected directly for the flag, and object-file timestamps confirmed a genuine recompile and relink, not a stale binary.
7.3 Re-benchmarked: no change
Both are within noise, if anything marginally lower.
A useful negative result. The kernel's real-world bottleneck is not arithmetic throughput, where AVX2 helps: a 32-weight, 10-byte ternary block is small enough that reading it from RAM and unpacking 2-bit codes is likely dominated by memory latency and per-block overhead, not SIMD lane throughput once data is in registers.
The standalone test's 3.33× speedup was measured in a cache-friendly microbenchmark loop reusing the same block repeatedly — a very different access pattern from streaming once through a full weight matrix per token. This is consistent with the earlier ~1.6× lookup-table ceiling (§5): the kernel is memory-bound on this hardware, not compute-bound, and further arithmetic-side tuning has limited headroom.
08Where the project stands
mul_mat rewrite measured a ~1.6× ceiling — not enough to close a 4.7–5.4× gap. Decision: stop rather than pursue it.09Recommendation
Use stock Q4_K_M via Ollama / llama.cpp for actual inference on this hardware. This project's value is what it demonstrated, not a deployable speedup:
- A working ternary type and kernel built from scratch and correctness-verified at every stage.
- A real measured ceiling on lookup-table techniques for AVX2-only, no-AVX-512 CPUs.
- A concrete, reproduced demonstration of why post-training ternary quantization fails without fine-tuning — measured on a real model with a real perplexity sweep, not just cited from papers.
- A documented, load-bearing distinction between which early transformer layers can and cannot tolerate aggressive quantization.
The codebase and minicpm5-tern-24layers.gguf remain as a working reference for anyone picking up ternary-quantization kernel work on similar hardware later.
Generated from the project's working files (INTEGRATION-CHECKLIST.md, skills.md, architecture.md) in Runtime Optimizations/. To save as PDF: open this page in a browser and use Print → Save as PDF — site navigation is hidden automatically in print.