Ollama, LM Studio, llama.cpp and Unsloth, which I use for what

I started with one program to run models locally and I've ended up with four. Not out of collecting, but because each one solved a problem the previous one didn't. Here I explain what I use Ollama, LM Studio, llama.cpp and Unsloth for, the real problems each one has given me and which one I'd install depending on what you want to do.

The videos make it look easy: "install this, download a model and you've got your own private ChatGPT". One program, two clicks, done. I started like that too, with Ollama, in spring 2024, on my work computer at the time (Windows and a 4 GB graphics card), because it was the one in every tutorial. And for a couple of weeks I was happy.

Then I wanted to try models without typing commands, and the second program appeared. Then I wanted to squeeze more speed out of an underpowered graphics card, and the third appeared. Then it occurred to me to fine-tune a model, and along came the fourth. At this rate, within a year I'll have eleven.

In More than fifty models tested at home I talked about the models: the files that answer. Here I talk about the programs that run them, which is the piece almost everyone confuses with the model.

A model is a big file full of numbers. On its own it does nothing: it doesn't open with a double click or say hello. It needs a program to load it into memory, run it and let you write to it. That program is called a runtime, or inference engine, and when it also serves requests from other programs, a model server.

It shouldn't be confused with a harness, a word I keep on this blog for agents. Harnesses, like Claude Code, OpenClaw or DeepSeek Harness, don't just use the model: they give it tools, memory and permissions to read files, run commands or browse the internet. It's usually summed up like this: an agent is a model plus a harness.

Three of the programs in this article are runtimes: they load the model and let you use it, some with a nice window and others from the terminal (that black window where you type commands). The fourth, Unsloth, is above all a fine-tuning tool. What you do with the model afterwards is up to you, and that's what What surrounds the model so it's actually useful is about.

LM Studio is a desktop app for Windows, Mac and Linux that lets you search for, download and use models without touching the terminal. It has a search tool that tells you whether a model fits on your machine, a chat similar to ChatGPT's, the option to work with documents and a server mode so other programs can connect to it.

I open it on my work laptop, an HP Omen with Windows and an 8 GB card, when I want to try a new model before deciding whether it deserves a place on the main server. It's my testing room: I download, ask, compare two models in a short while and, if one is any good, move it to where it's going to do real work.

The real problem it gave me: I downloaded several versions of the same model from its search tool and ended up with more than 100 GB of duplicates in a folder I didn't even know existed. It took me an afternoon to work out why the system disk was full. Now I have the models folder moved to a separate disk and I check it every so often.

And a reminder I give myself: a comfortable window doesn't make the model any smarter. If the model is weak or doesn't fit in your memory, LM Studio will show you that with a beautiful interface, but weak and slow all the same.

Ollama was born for the terminal: you type "ollama run" followed by the model's name and start chatting. Today it also has a desktop app, but its strong point is still something else: it opens a door on your computer, what techies call an API, through which other programs ask the model for answers as if it were a cloud service.

That's why it's the one I use underneath. On the main server, Ollama serves the models to a web interface, to some scripts that sort and summarise documents and to a couple of client tools that run inside their network. Nobody sees it, but it carries the daily workload.

The real problem: after an update, a model started answering much worse without me touching anything. It turned out the default context memory setting (how much text the model remembers in a conversation) had changed and my long documents were being cut in half. It took me two days to find it. Since then I set those parameters by hand.

And a security warning, because that door is a real door. By default Ollama only listens on the computer itself, but as soon as you open it to the network so other machines can use it, it asks for no username or password: anyone who reaches it can use your models. On my servers it's only reachable from the internal network, the firewall blocks everything else, the web interface has user accounts and passwords, and I update Ollama and the interface once a month, reading what changes first. The same goes for the server mode of LM Studio or llama.cpp: none of it is left open to the internet.

It's worth saying clearly because it's the most common confusion: Ollama is not an AI. It's the program that loads and runs the models. If someone tells you "I use Ollama's AI", they mean they use some model, like Qwen, Gemma or Llama, through Ollama.

llama.cpp is the engine underneath several of the comfortable tools. It works with files in GGUF format, the most common for downloading compressed models, and is driven mainly from the command line, though it includes a small server. What it gives you in exchange for its unfriendly face is control: you can decide how the model is split between the graphics card and the computer's normal memory, and fine-tune performance in detail.

I prefer it when a model doesn't fit entirely on the graphics card and has to be split carefully. On the main server, the one with the 12 GB RTX 4070 and 64 GB of RAM, with a 32B model in Q4_K_M and a 4,096-token context, which the comfortable tools loaded at a little over 1 token per second (a token being the piece of a word the model writes with), by adjusting by hand how many layers went to the card I got to about 2, with the same model, the same context and the same question. Change any of those and the figure changes, so take it as a rough guide. It's no miracle: it's still slow, but it's twice as fast, and for tasks that aren't in a hurry that shows.

The real problem: the first time I tried to compile it with graphics card support I spent an entire afternoon fighting with driver versions. In the end I used a precompiled version, which is what I should have done from the start. And the parameters: there are so many that it's easy to touch one you don't understand and make things worse. I write down what I change and what result it gives each time.

Unsloth is mainly for fine-tuning models, that is, training them a bit more with your own examples so they adapt to a task, using less memory and less time than the usual methods. With it you can fine-tune a small or medium model on a single home graphics card. Its recent versions also include an interface for running models, and it publishes compressed versions of many well-known models, which is how many people discover it.

An important clarification: fine-tuning isn't teaching it your documents. If you want the model to consult your information, there are simpler techniques, and I cover them in What surrounds the model so it's actually useful. Fine-tuning makes sense when you want it to always answer with a specific style, format or criterion.

What I fine-tuned, and I mark it as experimental: a small 3B model to sort a client's support emails into seven categories and always return the same format. I labelled about 600 examples by hand, trained on 480 and set aside 120 the model never saw, to measure with. I used LoRA, which adjusts only a small part of the model, with three passes over the data. On those 120 held-out emails, the result was an accuracy of around 85% (102 out of 120), against 70% for the same model without fine-tuning and with good instructions. The mistakes were concentrated in two categories that even I struggle to tell apart: technical incidents and billing queries. It improved, yes. But a bigger model without fine-tuning reached 82% on the same set without all that work, so I haven't put it into production yet. The client's emails stayed on my network, without names or contact details.

The order was this: Ollama first, in spring 2024; LM Studio a few months later, on that same computer, when I got tired of trying models from the terminal; llama.cpp when I wanted to run big models on small cards; and Unsloth in early 2026, out of curiosity about fine-tuning. Each jump came from a problem the previous one didn't solve, not from a new video.

If you're starting out and what you want is to try things, install LM Studio. It's the friendliest door and in one afternoon you'll have a model answering. If the terminal doesn't scare you, Ollama also works as a first step.

If you want to connect the model to other programs, so your scripts or tools can ask it for things, Ollama. That's what it's built for and it's very easy to leave running on a server.

If you want performance, squeezing a tight graphics card or carefully splitting a big model, llama.cpp. But bring patience.

If you want to fine-tune, Unsloth. Though first check whether a good model with good instructions already solves the task. In my case it almost did, and it would have saved me an entire weekend.

What's true in the promise of one program for everything? To get started, yes: one is enough. To work, each program does one thing well, and the sensible thing is to use the right one and not fight with the wrong one. In What surrounds the model so it's actually useful I tell you what I've built around it so that all this is good for more than chatting.