What surrounds the model so it's actually useful

Having a model running at home is like having an engine sitting on the table: impressive, noisy and it takes you nowhere. To make it useful you have to build things around it. Here I describe the four pieces I've gradually added to my local models, what problem each one solved and what went wrong along the way.

In Ollama, LM Studio, llama.cpp and Unsloth, which I use for what I left the models running. Loaded, answering, fast even. And then what?

My first days with an installed model were a bit sad. I had it running on a server, asked it things from the terminal a couple of times a day and, after a week, realised that for everything I really needed I was still opening Claude. I had a model at home and used it for nothing. Like the exercise bike that ends up as a clothes rack.

Almost every video ends right there: the model replies "Hello! How can I help you?", the presenter smiles and the subscribe button appears. Nobody tells you about the next day, which is when you discover that a model that only says hello doesn't save you a single minute. This article is about the next day.

Local AI has a very limited starting point: a model produces text, full stop. It doesn't remember what you talked about yesterday, doesn't see your documents, doesn't touch your tools and doesn't get to work unless you write to it. It's a brain with no memory, no eyes and no hands.

Having AI running locally means, above all, that your data doesn't leave home and you don't depend on someone else's service. Mind you, that doesn't make it private or secure by magic: if the machine is poorly protected, open to the internet or shares folders with the wrong people, the data is at home, but exposed. But for that to be any use you need pieces around it. I've built four, and I'll describe them in the order they arrived:

1. An interface to use it comfortably.
2. Search across my documents.
3. Connection to my tools via MCP.
4. Scripts that launch tasks without me being there.

None is essential on its own. Together, they turn a toy into something I use.

The first piece was a web interface: Open WebUI, an open-source application that gives the local model a ChatGPT-like window in the browser. I installed it on the same server where Ollama runs.

The problem it solved was very specific: I wanted to use the model from any computer on the internal network, including the laptop, without opening a terminal or installing anything on each machine. And I wanted someone else at home to be able to use it with their own user, without seeing my conversations. With the interface, that was sorted in one afternoon.

What failed: after an update the connection to Ollama was lost and the interface showed an empty model list. I spent a good while convinced I'd deleted the models, until I saw it was just a misconfigured address. I also left it reachable from outside the internal network for a while, without thinking much about it, and when I noticed I closed it again. Now you can only get in from inside or through a secure connection to my network.

The second piece is the one that has changed my usage most. Searching your documents with a model works like this, in plain words: when you ask something, a system first searches your documents for the chunks that look relevant and passes them to the model along with the question, so it answers with them in front of it. This is called RAG, short for "retrieval-augmented generation". The name is scarier than the idea.

Don't confuse it with fine-tuning, which I covered when talking about Unsloth. Fine-tuning changes the model so it answers with a certain style or criterion. RAG doesn't touch the model: it puts the right information in front of it every time you ask.

I set up the document search built into the web interface itself with about 900 documents: technical manuals for my servers, project notes and documentation for programs I use often.

The results, feet on the ground: in a test with 40 questions whose answers I knew, it got about 28 right, 7 half right and failed 5. The failures were almost always of the same kind: the search brought back the wrong chunks, for example from a manual for an old version, and the model answered with total confidence based on them. Also, tables in the PDFs got chopped up badly and answers involving numbers were the least reliable. Time per answer: between 5 and 15 seconds, acceptable for lookups.

Lesson: RAG works as well as the documents you give it. Clearing out old versions and duplicates improved the results more than changing the model.

And it has limits worth knowing beforehand. The index, the chopped-up copy the search looks through, doesn't notice changes on its own: if you update a manual and don't index it again, the model keeps answering from the old version. I re-index by hand every time I add or remove a batch of documents. Nor does it understand permissions: whatever goes into the index can be seen by anyone who asks in that collection. That's why client documents aren't in the shared household collection, but in a separate one that only my user can see.

The third piece is the one that gives the model hands. MCP (Model Context Protocol) is an open standard introduced by Anthropic, the company behind Claude, at the end of 2024, for connecting an AI to tools and data: files, email, databases, the browser. The most useful comparison is a universal plug: instead of building a custom connection for each tool, both sides just need to speak the same language. The MCP server is the small program that acts as a bridge to each specific tool and tells the model what it can do with it.

With the local models I set up three connections: one to read a folder of working documents, one to query an inventory database of my machines and one to read the calendar. All three read-only. This is where local models struggle more than the big ones: the small ones sometimes didn't quite understand when to use each tool or called it with badly formed data; with a 14B model or larger, things improved quite a bit.

And the risk, which I don't want to gloss over: installing an MCP means granting permissions. That small program can read, and sometimes write, whatever you connect it to. You need to know where it comes from, who maintains it and what it can touch. My boundaries, in short: I only install MCP servers from known, reviewed sources; each one sees one folder or one table, never everything; passwords and access keys go in a separate file the model doesn't see; any action that writes or deletes asks for confirmation; and every call is noted in a log I check when something doesn't add up. The ones I use daily and how I install them I explain more calmly in How I connected AI to my tools.

The fourth piece is the least flashy and the one that saves me the most hours: scripts, small programs that launch tasks on their own at a set time or when something happens. One sorts the emails in an alerts account every night and leaves me a summary in the morning; another summarises the server logs and flags what looks odd; another tags new documents that land in a shared folder. They run on two of my internal servers, the same ones where Ollama lives.

Before, I'd have built several of these in n8n, the block-based automation tool. Now I use it less because agents are replacing it: I had the log summary in n8n and rebuilt it with an agent that wrote the whole script in one afternoon. I go into detail in Why I no longer need n8n.

It's worth separating three things that look alike. A script always does the same thing, step by step, and the model just fills a gap, such as summarising or sorting. A tool, like an MCP, is something the model can decide to use while it's talking to you. An agent decides on its own which steps to take and can run commands on the machine, and that's the one that needs the most care. My scripts run under their own user without administrator rights and can only write to their own folder; an agent allowed to run whatever it likes is never left alone on a server.

What failed: the email script got stuck one night on a huge message, with attachments of several megabytes, and hung until the next day without warning. Now it has a maximum run time and sends me an alert if something doesn't finish.

Local is enough for me in three kinds of work. When there's client data I don't want to take out of the house: one client's document classification is done entirely locally. When the task is repetitive and high-volume: summarising and sorting about 150 emails and documents a day without paying for each one. And when searching my own documents is all I need.

I go back to Claude when careful reasoning is needed, when there's code involved that matters or when the task has many chained steps. There the difference shows straight away, and not by a little.

My numbers, approximately: around 30% of the automated tasks run locally; the rest is still in the cloud. I estimate the savings on paid model usage at about €20 a month, which doesn't come close to paying for the electricity and the hours invested. The full assessment, with numbers, is in What good is having AI at home?.

If you want to build something usable in a weekend, this is the list I'd have liked to have:

• A simple program to run the model: LM Studio or Ollama.
• A small model that fits comfortably on your machine.
• A web interface to use it comfortably.
• A folder of documents with search, few and well chosen.
• One read-only MCP connection, from a source you know.
• A simple script that does a single task a day.

What I wouldn't do the first weekend: open anything to the internet, grant write permissions, fine-tune models or connect client data.

What's true in the promise of the model that already does it all? That the model is the most famous piece and the least sufficient. The useful stuff is around it. Whether all this pays off, I tell in What good is having AI at home?.