llama.cpp

llama.cpp is a free, open-source program for running AI models on your own computer very efficiently, even without a powerful graphics card. It's the engine underneath several more user-friendly local AI tools, and it works with files in GGUF format, the most common one for downloading compressed models. For beginners, it's more a piece worth knowing about than an everyday app.

It was born to run the Llama models, hence the name, but today it works with a huge number of open-weight families. What throws beginners is that it doesn't have the friendly face of other apps: it's mostly operated from the command line (typing instructions in a text window), although it includes a small server so you can use it from the browser or from other programs. In exchange, it gives you more control: you choose how the model is split between the graphics card and regular memory, and you can fine-tune performance down to the last detail.

llama.cpp is one of the four programs I use to run my local models, along with Ollama, LM Studio and Unsloth. What I use each one for, how I ended up using all four instead of one and the real problems each has given me I explain in "Ollama, LM Studio, llama.cpp and Unsloth: which one I use for what". If you're just starting, a user-friendly tool will probably suit you better; llama.cpp makes sense when you want to squeeze every drop out of your machine.