Running a local model on modest hardware

The first thing to accept about running a language model on a Raspberry Pi is that it will be slow. The second is that slow is not the same as useless.

This is not a benchmark post. I do not have a rig that produces meaningful tokens-per-second comparisons, and the numbers would depend on the model, the quantization, and the day. What I can describe is what the experiment is good for.

Why bother#

There are three honest reasons to run a model locally:

• Privacy. The data never leaves the network. For notes, documents, and anything that resembles personal information, that matters.
• Cost predictability. No per-token bill and no surprise usage.
• Understanding. Running it yourself teaches you where the latency actually comes from.

There is also a less impressive reason: it is interesting. That is allowed.

What the setup looks like#

The Pi 4 runs a containerized inference server, with the model stored on a local disk. Access is over the private Tailscale network, so nothing is exposed publicly.

The important design choices are not about the model:

• Keep the model on fast storage. An SD card is the bottleneck before the CPU is.
• Limit the context window deliberately. A huge context on small hardware is a good way to spend a minute waiting for a response.
• Run inference as a separate service with its own resource limits, so a runaway request does not take down everything else.
• Put a timeout on the client. A model that never responds is worse than one that fails.

Where small models are useful#

For narrow, well-specified tasks, a small local model is genuinely helpful:

• Rewriting a paragraph for clarity.
• Summarizing a short document.
• Extracting structured fields from messy text.
• Generating a first draft of something routine.

The common thread is that the task has a clear input and a clear success condition. That makes it easy to notice when the model is wrong.

Where they fall short#

Long documents, multi-step reasoning, and anything that requires current knowledge are all painful. The context window is a hard limit, and a small model will confidently invent details rather than admit uncertainty.

That last behavior is the one to design around. If the output is going to be used without review, the model needs to be constrained to a task where a wrong answer is obvious. Otherwise it is not an assistant; it is a source of plausible errors.

The useful habit#

Running a local model changes how I think about the hosted ones. The local version is slow enough that you can feel the cost of each request. It also makes the failure modes visible: truncated context, repeated tokens, a confident answer that ignores half the input.

Those are not unique to small models. They are just easier to see when the model is running on a desk next to you.

Would I recommend it#

If the goal is to replace a hosted model, no. If the goal is to understand what these systems are doing, and to have a private place to experiment, absolutely.

The constraint is the value. A model that answers in two seconds teaches you less than one that takes forty.