My assistant lives in a box the size of a deck of cards, on a shelf, next to the router. It answers questions, searches the web and a 52 GB offline copy of Wikipedia, does arithmetic with a real calculator, and gets a whole Linux machine of its own to write and run code in. Apart from web searches and the pages it opens, nothing it does leaves the house.

It’s a Raspberry Pi 5 with 8 GB of RAM. It is not fast. It is not clever in the way the big hosted models are clever. And some days it drives me up the wall.

I’d still not give it up. This is the long version of why, and of everything it took to get here.

Where it started: the GPU dream

In mid-September I had a simple plan. Google’s LiteRT-LM came with an OpenAI-compatible server, Raspberry Pi had published a guide for running it on the Pi 5’s GPU, and I had a Pi. Put the model on the VideoCore VII, put Open WebUI in front of it, done in an afternoon.

The first README I wrote for it has a section called “About the GPU backend honestly”. It quotes Raspberry Pi’s own guide, which says plainly that the GPU is slower than the CPU on this workload because WebGPU support in the V3DV Vulkan driver is still experimental.

I read that and kept the GPU anyway. Of course I did.

It took until October to measure it properly. The model’s path to the GPU is long: WebGPU, then Vulkan, then Mesa’s V3DV driver. It ran, and it gave correct results. It was also painfully slow. Embedding a one-line query took 38 seconds on the GPU and under a second on the CPU. A 1,500-character chunk timed out after two minutes, and the GPU run got OOM-killed at a 1 GB limit, because the Pi has no video memory of its own and every GPU buffer counts against the container’s RAM.

Whatever the exact cause (I suspect first-run shader compilation, which someone else found dominating a similar Pi 5 benchmark), the deeper reason was simpler. Generating text is limited by memory bandwidth. Each token reads the model’s weights from RAM, and the GPU shares that same RAM with the CPU. There was nothing to win.

So on October 10 I dropped the GPU entirely. The nice side effect: the container had been running privileged just to reach /dev/dri. Now it runs with no-new-privileges and touches no devices at all. Losing a feature made the thing safer.

What’s actually running

Here’s the setup as it stands today.

The hardware. A Raspberry Pi 5, 8 GB, four Cortex-A76 cores overclocked to 2.7 GHz, a 1 TB NVMe drive and an external SSD for Wikipedia and backups.

The neighbours. This isn’t a dedicated AI box. The same Pi runs about 55 containers: photos, a password manager, an RSS reader, a Git server and its CI runner, backups, monitoring. That matters a lot, as you’ll see.

The model. Gemma 4 E2B, Google’s edge model with about two billion effective parameters, in the .litertlm format. I tried the bigger E4B. It fit in memory and decoded at 3.4 tokens a second, less than half of E2B. With thinking on, that’s a coffee break per answer.

The runtime. LiteRT-LM 0.18.0 on the CPU, through its serve command. Raspberry Pi’s own benchmark put it at 99 tokens a second of prefill and 9 of decode for this model, in about 1.4 GB, well ahead of llama.cpp on the same board.

The front end. LibreChat, which handles chats, presets, uploads and memory.

The tools, all over MCP, each in its own locked-down container:

  • web search through a self-hosted SearXNG;
  • English Wikipedia from a Kiwix ZIM file, 52.7 GB, no images;
  • a calculator that also does dates, clock times and letter counts;
  • a Debian sandbox per chat;
  • a headless browser;
  • offline translation with Argos.

File search. LibreChat’s rag_api with pgvector, embedding on a second, small LiteRT-LM server running EmbeddingGemma 2. The chat server only handles one request at a time, so uploads can’t be allowed to queue in front of a reply.

The whole AI part of this is public, with the host-specific bits scrubbed, at alevromitis/pi-ai-stack.

Making it bearable

Out of the box it was barely usable. About 6 tokens a second. Thirteen to seventeen seconds of staring at nothing before a new chat said its first word. Some one-line follow-ups took a minute and a half.

Here’s where it ended up:

BeforeAfter
Generation6.3 tok/s10.1–12.5 tok/s
First word, new chat13–17 s~5–6 s
First word, second message17.5 s3.5 s
First word, back to an older chat17 s4 s
First word after a restart36 s4.7 s
One-line follow-up to a search answer85 scached

None of that came from one big fix. It came from a pile of small, slightly stubborn ones.

Sharing a small computer

The neighbours were the first problem. With the other containers stopped, the model ran noticeably faster. Decode moves all four threads in lockstep, so if a backup job grabs one core for a moment, every thread waits.

I can’t stop the other services. I can rank them, though. Giving the model’s container the maximum CPU weight took prefill from 60 to 75 tokens a second and decode from 6.3 to 7.4, under the same background load.

Memory was worse. At one point 3.6 GB of the model had been pushed out to swap, and every token paid for it. Now the container has a hard 7 GB limit and no swap at all. I’d also set a memory reservation, which turned out to do literally nothing: under cgroup v2, a container’s protection is capped by its parent’s, and Docker’s parent has none. That one stung. The config said one thing, the kernel did another, and I only found out by reading /sys/fs/cgroup directly.

Context size had a cost I didn’t expect. The engine allocates for the whole window up front, so a 32k window makes even short chats slower. At 16k, prefill is 75 tokens a second; at 2k it would be 121. I settled on 16k, which still fits a chat, a few search results and the tool definitions.

Squeezing the hardware

Two tweaks actually moved the numbers.

LiteRT-LM has an experimental CPU backend called YNNPACK. On this Pi it raised prefill from 103 to 148 tokens a second, and it fixed a memory problem too: under the default backend a 16k context got OOM-killed on the first request, while YNNPACK served 32k in about 4.6 GB. The catch is that serve never passes that option to the engine. So my image patches one line into it at build time, and the build fails loudly if that line ever changes upstream.

Then I overclocked. The Pi starts throttling at 80°C, so I went looking for the highest clock that never gets there. I tested with a one-shot boot config and a watchdog, so a bad setting would reboot into the safe one. It did, three times. At 2.8 GHz LiteRT-LM crashed once, and with extra voltage it ran too hot. At 2.9 GHz it hit 85.6°C and throttled hard. At 3.0 GHz the Pi didn’t boot. 2.7 GHz at stock voltage peaked at 79.6°C after ten minutes of all-core stress. Long chats sit around 71°C. It’s worth a few percent, not a miracle.

I also checked each clock for silent errors: same prompt, greedy decoding, byte-for-byte identical answers. A crash you notice. A CPU that’s quietly wrong you don’t.

The one-instruction fix

Gemma 4 ships with a small drafter model for multi-token prediction. The drafter guesses a few tokens ahead, and the main model checks them all in one step. Google reports 1.6× faster decoding for E2B. On my Pi it took real chats from about 7.3 to 10–12.5 tokens a second.

But there was a bug. In 0.18.0, whenever the drafter guessed past the end of a reply, the end-of-turn token fell out of the conversation’s memory (#3439). For a single question you’d never notice. My server keeps conversations alive between messages, though, so every affected turn left the model with a slightly wrong picture of the chat.

The decode loop was rolling back one token too far. A comparison right before the faulty instruction already left the missing value in the CPU’s carry flag, so changing a single add to adc fixed it. The image patches that one instruction in Google’s prebuilt library, after checking the library’s hash.

I did try building LiteRT-LM from source instead. An hour and a half on the Pi, same speed, and it crashed on the first tool call, because the open-source build links a prebuilt Google component compiled against a different C++ library. So, a binary patch. I’m a little proud of that one.

Stop re-reading the whole chat

This is where most of the speed came from, and it’s almost embarrassingly simple in hindsight.

Upstream serve starts a fresh conversation for every request. Every message re-reads the system prompt, every tool definition and the entire history. At 75 tokens a second, a chat with a couple of search results in it means a minute of reading before anything new happens. A 31k-token prompt took about 19 minutes to prefill. I timed it. I regret timing it.

So the server now keeps the last three conversations alive and only reads what’s new. That sounds like one change. It was half a dozen, because things kept quietly breaking the match:

  • Search answers never matched. The server rewrites replies before sending them (it writes the Sources line itself, more on that below). The client then sends back the rewritten text as history, which didn’t match what the cache had stored. Every follow-up to a search answer re-read the whole chat: 85 seconds for a one-liner. It now matches on what the client actually received.
  • Titles evicted the chat. After the first reply, LibreChat asks the model for a title. That request has no tools, so it looked like a new conversation and kicked the real one out. Now a small gate answers it instantly from the first message (“What is the capital of France?” becomes “Capital of France”) without touching the model. Second messages went from 17.5 to 3.5 seconds.
  • Memory requests did the same. LibreChat’s memory feature sends an extra request with every message. Another gate answers those without the model unless I actually asked it to remember or forget something.
  • New chats still started cold. Each one read about 1,000 tokens of system prompt from scratch. LiteRT-LM can clone a conversation in about a second, so the server keeps one with just the system prompt read in, and clones it for each new chat. Getting the system prompt read in before the first message took a second one-instruction patch, because the option exists in the engine but the C API doesn’t expose it.

Tool definitions turned out to be the hidden cost here. They’re part of the system prompt. With no tools, building that snapshot took 17 seconds. With the browser’s 25 tools, 94. I hid the search tools the model never needs and cut one tool’s 70-token description, mostly about proxies I don’t run, down to a single line.

There’s one thing I haven’t fixed. With multi-token prediction on, a cloned conversation sometimes fails in the drafter before its first token. The server retries, and the retry has always worked. But once, under heavy test load, a clone gave a wrong answer with no error at all. No retry catches that. I know it’s there, and I’ve chosen to live with it.

Making it honest

Speed made it usable. Honesty made it worth using, and that was the harder half.

A 2B model is confident in a way that would be charming if it weren’t so often wrong. It miscounts letters. It does arithmetic in its head and gets it wrong while showing its working. Asked which country won the 2010 World Cup, it read a Wikipedia infobox that said “Current champions: Argentina” and answered Argentina. Asked about a name Wikipedia had no article for, it answered from a different person’s films.

I spent a few days trying to fix that with the prompt. Rules for age questions, rules for counting, special cases for things that had failed. Each one fixed the prompt I’d tested and broke something else. Eventually I reverted all eleven. That one hurt.

What worked was a different idea: stop asking the model to be reliable and make the things around it reliable.

  • Citations are written by the server. In the first Wikipedia test the model cited zero sources in 24 answers, even though the prompt asked for them every time. When it did list sources later, it padded them with irrelevant ones like “Eiffel Tower (Paris, Tennessee)”, and sometimes made links up. Now the server builds the Sources line from the search results it actually saw, keeps only the ones that back the answer, and throws away whatever the model wrote.
  • Arithmetic goes to a calculator. The calculator works on a whitelisted syntax tree, never eval, and handles dates and letter counts too, because “how many r’s are in strawberry” is a tool call, not a vibe.
  • Wikipedia searches say when they’ve missed. If none of the results mention the name in the question, the tool says so and suggests searching again, instead of handing over plausible-looking passages about someone else.
  • The checklist goes in every message. A small model listens to the newest message far more than a system prompt thousands of tokens back. One rule was mostly ignored in the system prompt and followed 8 times out of 8 when appended to my message, with thinking off. So the server now appends a short “Before answering” paragraph to every message I send.
  • Help belongs in tool results. Hints in a tool’s description changed nothing. The same hint in the tool’s output (“that command isn’t installed, it’s in package X”) got followed.

My rule since then: a fix has to help a question nobody has asked yet. If it only fixes the question in front of me, it’s not a fix.

The local Wikipedia deserves its own word. It’s the June 2026 snapshot, searched with keywords and BM25 in about 0.2 seconds, using around 340 MB of RAM. In my first end-to-end test, with thinking on, the model got 8 of 12 right, said “I couldn’t verify that” for 3, and got 1 wrong. That honest “I couldn’t verify” matters to me more than the score. It’s the difference between a tool I can trust and one I have to double-check.

Keyword search is the weak link. It misses articles named differently from the question and things described rather than named. A dense index of article introductions found the right article in the top three for 43 of 43 of the model’s own queries, in a 78,000-article test. It’s the obvious next step. It’s also about 28 hours of GPU time to build, and an open question whether it fits in the Pi’s RAM.

A computer of its own

Every chat gets its own Debian container: root, apt, pip, compilers, network. Not a restricted shell, a whole machine, with hard limits on CPU, memory, disk and time. Idle sandboxes are stopped, not deleted, so if I come back to a chat two weeks later its files are still there.

The first week was humbling. The model told me a program worked and showed me the output, nicely formatted. The program had never run. It had never even been written to a file.

I ran ten rounds of sysadmin-style tasks against it, four or five requests each. It solved 10 of 43. It explained a firewall setup well and wrote a correct C program for primes. It also claimed a URL check said “UP” when it had printed nothing, told me to run crontab -e myself, and decided a systemd service “would work on a real system”.

So I made the sandbox talk back. Every result now says what actually happened, in plain words: “no output”, “compiled, nothing has run yet”, “this is the same command that just failed”, “that package isn’t installed, here’s the one you want”. And for code there’s a single run_code tool that writes, compiles, runs against an input and says PASS or FAIL, with “don’t answer yet” on every failure.

On LeetCode-style problems, the first round with it killed the worst habits: no more unwritten files, no more made-up output. Correctness is still patchy. The model will sometimes answer right after a FAIL anyway, or give me code that isn’t the code it tested. But on hard maths questions, the same model got 11 of 12 right with the calculator and sandbox, and 4 of 6 without. More thinking didn’t help. Better tools did.

When the Pi isn’t enough

Some days I need a bigger brain. New chats now default to Qwen3.5 9B running on my desktop’s Intel Arc B570, at about 42 tokens a second. When the desktop is off, those chats fail and I switch to the Pi by hand.

For a while I had a small router that did that switch automatically. It rewrote requests for the Pi when the desktop didn’t answer. I dropped it on October 11 to keep things simple, though honestly it was a good idea.

The desktop is the fast assistant that’s sometimes there. The Pi is the slow one that’s always there.

What it costs, and what still breaks

I’d be lying if I made this sound tidy.

As I write this, the Pi is using 7.3 of its 7.9 GB of RAM, plus almost 5 GB of swap. The system drive is 98% full. One night a request got stuck: the model accepted it and then sat there, idle, at 0% CPU, for eight hours, until I restarted it the next morning. The Pi had 39 MB of free memory at the time. Another evening the external SSD holding Wikipedia and my backups dropped off USB mid-write. It came back clean after a reboot. My heart rate took longer.

A 2B model can pick the right tool for one or two steps. Long plans are beyond it. With thinking on, a simple factual question can take a minute. And after all of this, it still occasionally tells me something with total confidence that isn’t true.

Things I tried that didn’t work, for anyone tempted:

TriedWhat happened
The GPU40–70× slower for embeddings, OOM at 1 GB
Gemma 4 E4BLess than half the speed of E2B
fp16 activations, three threads, other CPU governorsSlower or no change
Overclocking past 2.7 GHzCrashes, throttling, no boot
LiteRT-LM from sourceSame speed, crashed on the first tool call
Three cached system prompts1.5 GB more memory, swapping, more drafter faults
Higher reasoning effort, “check your work” promptsNo better answers
Per-question prompt fixesFixed one question, broke others

Why I keep it

Because it’s mine. Because I can read every line between my question and its answer. Because when it gets something wrong, I can find out exactly why, and fix the tool that misled it instead of hoping the next model update does.

And because it taught me the thing I didn’t expect to learn: with a small model, the model is the least important part. Almost everything that made this assistant fast, I did around it: caching, CPU priority, memory limits. Almost everything that made it honest, I did around it too: a calculator, a source line it can’t fake, a sandbox that tells the truth when the model won’t.

So if you’ve got a Pi, or any old machine, gathering dust: try it. Not to beat ChatGPT, it won’t. Try it to see what an assistant looks like when you own every piece of it. Then, the first time it lies to you, don’t reach for a bigger model. Ask what the tools around it could have said to stop it.


Sources