More than fifty models tested at home

I've tested more than fifty AI models on my own machines. Some have moved in with me and others lasted as long as it takes to realise they make things up more than a snake-oil salesman. Here I explain what a local model is, what those odd labels with numbers and letters mean, and the rule I follow today so I never fill the disk with junk again.

You've surely seen the headline: "This free model beats GPT", "The model you can run on your laptop that humiliates the paid ones". Every few weeks a new one comes out, and every few weeks someone swears that this time it's for real.

I believed it. For a while I downloaded everything that came out, like someone collecting football stickers. At one point I had more than 400 GB of models on a single server's disk, until one morning a backup failed because there was no space left and I had to sit down and look at what I had there. The answer: lots of models I'd opened once, asked something, said "oh, not bad", and never touched again.

The iron, the side of local AI you can touch, I covered in What hardware you need to run your own AI. Here I talk about what goes inside: the models.

A model is, at heart, a very big file full of numbers. Those numbers are what, when you write something to it, calculate the answer word by word. There's no magic and no database of stored sentences: there's a file weighing several gigabytes and a heap of arithmetic.

The important thing for beginners is that not all models can be downloaded. The ones behind ChatGPT or Claude are closed: you can only use them over the internet, on the company's servers. The ones you can have at home are "open-weight" models, which their creators publish so anyone can download and run them. The models for local AI you'll see most belong to a few families: Llama, Qwen, DeepSeek, Gemma and Mistral, each with versions in different sizes.

And one detail that took me a while to understand: the model, on its own, does nothing. It's like a record without a record player. It needs a program to load and run it, and that's what Ollama, LM Studio, llama.cpp and Unsloth, which I use for what is about.

When you look for a model, it almost always carries a number with a B after it: 7B, 14B, 32B, 70B. The B stands for billion. It tells you how many parameters the model has, that is, how many of those numbers I mentioned. In plain terms: its size.

Bigger usually means more capable, but it also needs more memory, above all on the graphics card. And that's where the dream crashes into the reality of your hardware.

My clearest example was with two sizes of Qwen, the 7B and the 32B. The small one took about 5 GB and answered almost instantly; the big one needed close to 20 GB and took far longer. The difference showed as soon as I asked for something a bit tangled: summarising a twelve-page contract without leaving out clauses. The small one swallowed half; the big one didn't. For sorting emails, on the other hand, I barely noticed anything. That was my first lesson: size matters depending on what for.

Here comes the word that scares people most and should scare them least: quantisation.

If a model is a huge list of numbers, each number can be stored with a lot of precision or a little. At full precision it takes up an enormous amount. If you round them, the file slims down a lot and the model keeps working almost the same. That's quantising: compressing by storing the numbers with less precision. It takes less memory in exchange for losing some quality, and it loses more the harder you squeeze.

That's why you'll see labels like Q4, Q5 or Q8 when downloading. The number indicates, roughly, how many bits are used for each value: fewer bits, more compression and less memory. One caveat: these labels belong to the GGUF format, the one used by llama.cpp, Ollama or LM Studio, and within it there are variants (Q4_K_M, Q4_0...) that don't compress in the same way. Other formats and programs use other names and other methods, so a "Q4" from one place isn't necessarily exactly the same as "4 bits" from another.

The surprise, the one that changed my habits: a big, somewhat compressed model often performs better than a small uncompressed one. I did exactly the opposite, because "compressed" sounded to me like "broken".

I checked it with Qwen 3 8B in Q4_K_M and in Q8, on the main server (the 12 GB RTX 4070, where both versions fit whole), with Ollama and the same 8,192-token context. The Q4 version was roughly 30% faster: about 35 tokens per second against 27 and, on my task of summarising meeting minutes in Spanish, out of twenty sets of minutes, I noticed something worse in only two: the odd decision over-summarised. It's my own test, with few samples and my judgement as the yardstick, not a laboratory. The Q8 was somewhat finer, but not enough to justify the extra memory.

This also explains why the same model shows up in dozens of variants with almost identical names. It's not that there are dozens of models: it's just one, in several sizes and several compression levels, sometimes uploaded by different people. At first it looks like a maze; once you understand the two variables, size and compression, it sorts itself out.

The first one I got running was a 7B Llama, in spring 2024, and it seemed a miracle that a home computer could answer me with something that made sense. Then I went through Mistral, Qwen, Gemma and DeepSeek, more or less in that order, with the odd strange excursion into specialised models I won't mention again out of embarrassment.

I soon realised that trying a model "to see how it goes" is useless. You ask it two friendly questions, it answers well and you're left with an impression. So I made myself a small fixed exam. Every new model goes through the same tasks: summarising a technical document of about ten pages in Spanish, sorting thirty real emails into five categories, extracting data from an invoice in a structured format and writing a simple Python function. Always the same examples, so I can compare. And the same conditions: the instructions saved in a text file that I copy as they are, a low temperature (0.2) so it doesn't improvise, an 8,192-token context and, whenever it fits, the same machine and the same program.

About the exam data: the emails and the invoice are mine or from a client who gave me permission, I strip out names, addresses and account numbers before using them, and everything stays on my machines.

Some releases changed things. The arrival of Qwen 2.5 was the first time a local model wrote to me in Spanish without sounding like machine translation. And DeepSeek's distilled reasoning models taught me that a small model that "thinks before it speaks" can get more right, even if it takes twice as long.

I note the results in a simple spreadsheet: model, exact version and download date, size, quantisation, machine, program, a score from zero to five on each task and a comments column. Next to the score I keep the raw figure, because a 4 on its own says nothing: for the emails, how many of the thirty it sorted correctly; for the invoice, how many of the ten fields it got right; for the summary and the function, my assessment from reading the result. Nothing sophisticated, and with only one person marking, but it has stopped me re-downloading models I'd already ruled out.

Before leaving one installed I also read its model card, the sheet published by whoever made it, and above all the licence. "Open weights" doesn't mean you can do anything you like with it: some use permissive licences and others come with their own conditions.

This table is a snapshot from September 2026 and will go out of date soon; I'll keep updating it.

The licences are the ones listed on each model card when I checked them; if you're going to use one for a business, read it yourself at the source, as they can change.

The 14B Qwen has stayed because it gives the best balance between speed and quality in Spanish for repetitive work. Gemma stayed for how naturally it writes; if I have to review a text, I prefer it. The small coding one I use as a quick helper, not to program anything serious: for that I still use Claude. The distilled DeepSeek is slow, because it doesn't fit entirely on the card and leans on RAM, but when the task has a catch it's worth the wait. And the 3B Llama is there because it fits on almost anything and helps me check something works before moving a big model.

The list of deleted models is longer than the list of survivors. There was a model that, on the invoice task, invented a complete IBAN number with all the confidence in the world. There was another that answered well in Spanish for the first two sentences and then switched to English without warning, like a guest who gets bored of the conversation. A 70B one took a couple of minutes per answer on my machine, and the quality didn't justify even half that wait. And a couple that didn't even start because they didn't fit in memory, which I downloaded anyway, because reading the requirements is for the weak.

What disillusioned me most, though, were the internet rankings. Looking for the best model for local AI in a leaderboard doesn't help much: they're measured with tests that look nothing like your work and they change every few weeks. The model at the top of one of those tables was among the worst in my email exam.

After so much downloading and deleting, my rule is the opposite of the one I followed at first:

1. First, the task. What do I want it to do: summarise, sort, write, code?
2. Then, the available memory, above all on the graphics card. That decides which sizes are realistic.
3. With those two facts, I pick two or three candidates from well-known families and run them through my usual examples.

I start with Q4 or Q5 and only go up a level if the model makes too many mistakes. And, above all, I don't download twenty at once. I did that and learnt nothing, except how to buy disks.

What's true in the promise of the free model that beats the paid ones? There are very good local models for specific tasks, and that's already a lot. But the one that does everything, on your computer and for free, I haven't found. It has taken me about 60 hours spread over several months to arrive at a table of five models.

Which programs I run these models with, I tell you in Ollama, LM Studio, llama.cpp and Unsloth, which I use for what. And the verdict on whether all this is any use is in What good is having AI at home?.