AI model quantisation

Quantisation is a way of compressing an AI model by storing its numbers with less precision, so it takes up less memory and runs on more modest machines. It's what lets you run models on a home computer that, uncompressed, would need a much more expensive machine. In exchange, the model loses some quality, and it loses more the more it's compressed.

A model is, at heart, an enormous list of numbers. Stored at full precision they take up a lot of space; if they're rounded, they take up much less and the model keeps working almost the same. That's why, when you download a model, you'll see labels like Q4, Q5 or Q8: the number indicates, roughly, how many bits are used for each value, and fewer bits means more compression. What's confusing is thinking the most compressed is always the worse option: very often a large, somewhat compressed model performs better than a small uncompressed one.

Quantisation is one of the first things you run into as soon as you start downloading local models, and with more than 50 tested I've had no choice but to understand it. What it is, explained without jargon, and why it matters for your graphics card I explain in "More than fifty models tested at home". If you're starting out, try a middle level first, like Q4 or Q5, and only change if you notice the model getting things wrong more than it should.