Engineering · Research
Building a Custom Ternary Quantization Kernel for llama.cpp — What Worked, What Didn't, and Why
A from-scratch 2-bit ternary quantization type and hand-tuned AVX2 kernel for CPU-only inference,
built and evaluated end to end on a Skylake ThinkPad: the kernel work, a catastrophic accuracy failure
and its root-cause diagnosis, a hard cliff in per-layer quantization sensitivity, a real compiler bug
that had been silently disabling the AVX2 path all along, and the honest verdict on whether any of it
beat stock llama.cpp.
llama.cpp
ggml
AVX2
Quantization
Read the report →
Deca Saas