Thirty-six seconds. That’s how long the Pi took to say its first word after a restart, with search and the calculator switched on. Eighty-five seconds for a one-line follow-up to an answer that had used web search. Ninety-four seconds to start a new chat with the Playwright browser enabled.

None of that was the model being slow. It was the model re-reading things it had already read.

That’s what this week was about. Fifty commits went into my homelab repo since the Sunday before, and almost all of them chase the same idea: on a Raspberry Pi 5 with 8 GB of RAM, the cheapest token is the one you never have to prefill again.

Since this is the first of these weekly updates, I’ve included everything in the repo’s history, which goes back to Sunday the 11th.

Sunday: a sandbox that stops lying

The week really started the evening before. My Gemma 4 E2B assistant has a Debian sandbox per chat to run code in, and the 2B model kept treating it as optional. It would get “FAIL: the program also prints other text” back for a C program that printed Enter the number of elements (N):, and answer the user anyway. It ran a JavaScript program on no input and never tested it again.

So every failure the sandbox reports now ends the same way: Don’t answer yet: fix the code and call run_code again until it says PASS. A prompt in the output gets its own FAIL, because nobody types here; input comes from stdin. A run on no input says it wasn’t tested.

Then I overcorrected. Asked for a hello world, the model ran it, then made up a test input (“Hello World”) with an expected output, ran it again, and told the user about this test as if they’d asked for one. The phrase “on the examples” in my reminder read to it like the user’s request. Programs without examples now just get their output reported. No push for a second run.

Two smaller things from that night that I’m glad about:

  • Sandboxes survive. Reopening a chat used to find its sandbox gone. Now an idle sandbox is stopped, not deleted, and the next call from that chat starts the same container again, files and installs intact. They’re only removed after 14 days unused, past 50 of them, or past 10 GB.
  • Download links, and then no download links. I added a share_file tool that handed files to the user through LibreChat. Eight minutes later I reverted it. It was clunky. Sometimes the right commit is the one you undo.

On the Qwen side (the 9B model on my desktop GPU), one setting fixed a nagging bug. Without includeReasoningContent, Qwen’s earlier tool-call steps reached its chat template with an empty <think>, and it often ended a reply mid-thought with nothing to show. That happened in 7 of 20 turns of my baseline, and 11 of 18 replays of one bad turn. With its reasoning sent back: 0 of 6. I also dropped the router that fell back to Gemma when the workstation was off. If the desktop sleeps, Qwen chats fail, and I switch by hand. Less clever, more honest.

Monday: the cache that kept missing

Monday was the big one: 25 commits, from just before 1 a.m. to almost 11 p.m.

The model gets the CPU first

One line in the compose file: cpu_shares: 262144, which is a cgroup v2 cpu.weight of 10000 against the default 100 every other container on the Pi gets. Decode runs four threads in lockstep, so if one core gets borrowed by a neighbour, the whole step waits. Clean A/B on the Pi: prefill went from 60 to 75 tokens a second, decode from 6.3 to 7.4.

And then I overclocked. arm_freq=2700 in config.txt, up from the stock 2.4 GHz. I tried higher. 2.8 GHz crashed LiteRT-LM at stock voltage, and with +50 mV it hit 84°C and throttled. 3.0 GHz didn’t boot at all. 2.7 held about 78°C through ten minutes of all-core stress, peaking at 79.6°C, just under the 80°C where the firmware starts throttling. No over-voltage, so the warranty bit is untouched. It buys roughly 7% on prefill and 4% on decode. Not dramatic. But free.

A one-instruction fix

LiteRT-LM supports multi-token prediction (MTP), a small drafter that guesses ahead so the big model can confirm several tokens at once. I wired up a switch for it, turned it on, and the model answered a calculator result with an empty reply. Every time.

The bug is upstream, #3439: with MTP on, when the drafter gets past the stop token, the decode loop rolls back the stop token along with the tokens after it. My prefix cache continues conversations from their KV state, so a missing stop token wrecks the next turn.

My first fix was to build the C library from source with a patch. It worked, until the first tool call. The open-source build can’t use Google’s prebuilt tool-call constraint provider, and that trapped immediately. So I threw the source build away, under an hour later, and patched the PyPI wheel’s own library instead. One instruction: an add becomes an adc (the carry flag is already set there), which moves the rollback by exactly the one token it was getting wrong. The patch checks the library’s sha256 and the exact bytes before it touches anything.

The probe is satisfying to read. Six greedy turns, conversation token counts:

Token counts
MTP off21 39 68 88
MTP on20 37 65 84
MTP on, with the fix21 39 68 88

With the fix, MTP is on by default: decode is 37–73% faster in real chats, prefill unchanged.

One more honest note: the empty replies I’d seen earlier weren’t even this bug. Given a history where the model stated the answer before calling the tool, it ends its turn after the tool result with MTP off too. Two problems that looked like one.

Every follow-up was a miss

Here’s the one that stung. My prefix cache keyed each conversation on the text the model generated. But the client never sees that text. The server rewrites it on the way out: it replaces the Sources line, scrubs leaked control tokens, adds a note when a reply gets cut off. LibreChat then sends the rewritten reply back as history, which never matched. Every follow-up to a search-backed answer re-prefilled the whole chat. That’s the 85 seconds.

The fix was to key the cache on what the client actually received. The KV still holds the raw generation.

Then LibreChat’s chat titles. After the first reply of every chat it asks the model for a title, with no tools, which the cache treated as a different conversation, evicting the chat. So every chat’s second message paid 20–35 seconds again. The logs said it plainly: miss, miss, miss. Now the server answers the title request itself with the first message’s topic, up to six words, and never touches the model. (It also drops “write”, “create”, “make” and friends, so “Write a short haiku about autumn leaves” becomes “Short Haiku About Autumn Leaves”.)

Clones

The real prize was new chats. Every new chat prefilled the system prompt and tool schemas from scratch, 12 seconds or more before the first token. The C library exports litert_lm_conversation_clone, which the Python package doesn’t wrap, so I wrote a probe to find out whether it could be trusted.

It mostly could. A clone continues byte-for-byte like a fresh conversation, with the first token after 1.8 seconds instead of 12. Two live conversations can alternate exactly. But with MTP on, an original that kept going after its clone ran gave different text once, and once crashed in the drafter: Cannot auto-resize tensor kv_cache_v_3.

So the cache now keeps several live conversations, no clones, and lets them take turns, which the probe showed was exact. Going back to a chat after another one ran dropped from about 17 seconds to about 4.

There was no clean “system prompt only” state to clone in 0.18.0’s C API. There was, however, an unused setter (filter_channel_content_from_kv_cache) sitting right there. Another one-instruction patch retargets it to set prefill_preface_on_init, so the server can keep a conversation holding just the system prompt, tools and settings, never send to it, and clone it for every new chat. Clones of a frozen snapshot were the safe pattern. New chats started answering in about 5 seconds instead of 13–17.

Then the Pi fell over

I pushed further: a snapshot per tool set and preset, rebuilding the day’s snapshots at startup, warming the rest in idle time, malloc_trim when glibc held on to 5.4 GB of heap after conversations closed. A separate branch ran LibreChat’s memory requests in idle time from their own snapshot.

That’s where it broke. Clones started faulting in the MTP drafter before their first token, two requests out of eight in one probe, even without the memory requests. One new chat gave a silently wrong answer: “stone” where it had correctly said “enots” in the round before, with no error raised, so nothing retried it. About 25 minutes into that test, with the container at around 5.4 GB, the Pi stopped answering SSH and tailscale ping. The tests that day had already OOM-rebooted it once.

I didn’t find the root cause. In v0.18.0 the drafter is one object per executor, holding copies of the running conversation’s KV buffers that seem to outlive a switch of conversation. That’s as far as I got.

What shipped is the defensive version. A request that faults in the drafter before its first token gets retried on a new clone, then on a fresh conversation; in the probe the retry always went through. And just after midnight I went back to one snapshot by default. With three, the container sat at 5.2 GB idle instead of about 3.7, the Pi swapped, and clones slowed from 2 seconds to 5 or 6. The memory-scheduler branch stays a branch.

It’s a strange feeling, writing a retry for a bug you can’t explain. It works. I don’t love it.

Tuesday: trimming, and a second language

With one snapshot, anything that changes it costs a rebuild, about 24 seconds before the first token. Every tool schema is prefilled into it. So I went after tokens:

  • searxng’s search_suggestions and instance_info tools are now dropped from each request. Neither mcp-searxng nor LibreChat can hide single tools, so the server does it.
  • web_url_read had a ~70-token description, mostly about FlareSolverr and Byparr, neither of which I run. It’s now one line: “Read a web page or file by URL, as markdown.”

I also put five idle services into a Compose profile called disabled: four MCP servers that weren’t serving anything, and Immich’s machine learning, which was holding about 190 MB for nothing. On a Pi this close to its memory ceiling, 190 MB matters.

Then a reversal. On Monday I’d made the Pi model English-only and ripped out the non-English search handling. By Tuesday night I’d built it back properly: an offline translation MCP server using Argos Translate models run by CTranslate2, English to and from 30 languages, pivoting through English for the rest. I skipped the argostranslate package itself, because it pulls in stanza and torch, about 1 GB, just to split sentences. A regex does that here. On the Pi a sentence takes 0.6–1 second, and memory levels off at about 500 MB with two language pairs loaded.

The tool alone wasn’t enough. Gemma 4 E2B often just didn’t call it, on the way in or on the way out. So now the server does it for every chat: a message detected as another language (confidence at least 0.8; “ok” and “hi” score about 0.03) reaches the model in English, and the English reply is translated back at the end, leaving code, links and the Sources line alone. Translations are cached, so the history comes out identical every time and the prefix cache still matches. The cost is real: the reply arrives all at once instead of streaming, plus about a second per sentence.

The models work best in English, and the offline Wikipedia is English-only. But an assistant that only understands English wasn’t the one I wanted. It took one day of pretending otherwise to see that.

Late Thursday, into Friday: getting ready to go public

The last three commits were about other people reading this.

The AI stack’s README had grown into one giant root file of more than a thousand lines. Its sections now live next to the code they describe: LibreChat, LiteRT-LM (with a new page on the image’s patches), agent-tools, agent-connectors, the sandbox, and the Qwen unit on the workstation. Tracked files no longer name my tailnet, LAN addresses, home directory or time zone; examples use placeholders and the real values stay in the encrypted .sops files.

Then the export itself. A script filters the repo down to the AI stack with git filter-repo, rewrites the values the tree used to show in old commits, normalises every author and moves every date to UTC. A checker fails on anything that looks host-specific or personal, gitleaks scans the whole result, and only then is CI set to push to github.com/alevromitis/pi-ai-stack, never forced. The same input gives the same commits, so every push is a fast-forward.

The final commit points it all at alevromitis.dev, the site’s new address.

What I’m taking away

Most of this week’s speed came from not doing work, not from doing it faster. The overclock bought 4–7%. Stopping the cache from missing bought tens of seconds per message.

So before you buy faster hardware for your local model, look at your logs and count how often it’s re-reading something it has already seen. “Miss, miss, miss” was hiding in mine for days.


Sources