[{"content":"Until this week, everything my assistant did, it did for me. My calendar, my notes, my feeds. If it got something wrong, the only person who saw it was me, usually at 1 a.m., usually muttering.\nThis week it went into a group chat. A real one, on Instagram, with people from my department in it.\nThat changes everything about what \u0026ldquo;good enough\u0026rdquo; means. A wrong answer in my own chat is a bug. A wrong answer in a group chat is a bug with an audience.\nThe commits all landed between Thursday and Saturday. It was a long three days.\nThursday: moving in The bridge itself had been sitting in a development folder for a while. On Thursday, at half past two in the morning, it got promoted to production.\nHere\u0026rsquo;s how it works. A small service on the Raspberry Pi polls my Instagram inbox. Plain messages never reach a model. When someone in a configured group mentions the bot\u0026rsquo;s handle, the bridge builds a prompt from the recent chat, sends it to a LibreChat agent, and posts one reply back to the group. Most requests go to Gemma 4 E2B running on the Pi through LiteRT-LM. One group can ask for Qwen on my workstation instead. It reads photos, shared reels, voice notes and videos too.\nIt\u0026rsquo;s built on instagrapi, which is an unofficial client for a personal account, not Meta\u0026rsquo;s API. I want to be upfront about that. It can trigger login challenges, and it can break the day Instagram changes something. So nothing in it hides what it is: no fingerprint tricks, no proxies, no challenge solving. And the failure handling got as much attention as the happy path:\nA failed poll backs off from 60 seconds to an hour, instead of hammering a session Instagram might already be rejecting. If the process dies mid-reply, that mention is closed as \u0026ldquo;check manually\u0026rdquo;. It is never replayed. Two copies of a reply in a group chat is worse than none. A send whose outcome is unknown is never resent, either. The README went from 703 lines to about 190 the same night. It had become a diary. Diaries belong in Git history.\nThe bot knew too much Then came the moment I\u0026rsquo;m least proud of.\nThe department group\u0026rsquo;s agent answers from a set of files: things like the study guide and course information. At first, two of those files were my own research notes. Notes about my plans.\nWhen someone asked, it told them.\nIt wasn\u0026rsquo;t malicious. It just had no idea what \u0026ldquo;private\u0026rdquo; means. But it\u0026rsquo;s the exact failure I keep writing about here, and I walked straight into it: give a model a document, and that document is now something it will happily say out loud to whoever asks. By Thursday afternoon the group searched five general Markdown files written for any student instead, kept outside the repo. Nothing personal in them, so nothing personal to leak.\nThursday afternoon: the Thinker that wasn\u0026rsquo;t needed My first design was clever. A small \u0026ldquo;dispatcher\u0026rdquo; agent took the message and, if it needed documents, handed it to a \u0026ldquo;Thinker\u0026rdquo; agent with file search.\nClever, and slow. And weird in a way I didn\u0026rsquo;t predict. With thinking turned off, the dispatcher answered short requests by repeating them. \u0026ldquo;translate to english\u0026rdquo; got back\u0026hellip; \u0026ldquo;translate to english\u0026rdquo;.\nSo I measured it. Same real prompt, eight fresh requests, twice each:\nDispatcher Echoed replies Thinking off 6 of 16 Thinking on 0 of 16 Thinking on cost 30 to 40 seconds more per reply. Then a document question went dispatcher, Thinker, file search, and took 591 seconds. Nine minutes and change. The bridge\u0026rsquo;s timeout was 600.\nBy Thursday night the whole two-agent setup was gone. Each group now gets one agent, thinking on, with its own tools. No delegation. Fewer moving parts, fewer handoffs to get lost in.\nI also turned off automatic translation on the Pi. Running Greek through a translation model and back had been garbling dates and characters. Gemma writes Greek well enough on its own.\nFriday, 2 a.m.: Instagram has opinions about links A long reply to the group just\u0026hellip; didn\u0026rsquo;t arrive. And my error handling had thrown away the reason.\nSo the first fix was boring and right: log every failure properly. The error type, Instagram\u0026rsquo;s own reply, the HTTP status, the file and line. Cookies, passwords and tokens redacted.\nThirty-nine minutes later, the new logging caught the real cause. instagrapi sends any text containing \u0026ldquo;http\u0026rdquo; as a link message, and Instagram refuses link text over 1,000 characters. The error even has a name: LinkTextTooLong. Plain text is fine at nearly double that. A 1,922-character reply without links went through.\nThe fix took a few rounds:\nLong replies split into ordered parts, at paragraph, line, sentence, comma or word breaks. Still one reply as far as the bridge cares: one receipt, one send slot. Parts with a link are capped at 900 characters, the rest at 1,900. A link is never cut in half between two parts. A test reply with twelve sources kept eleven of them whole before this. Now all twelve. On Saturday I learned Instagram shows only one link preview per message. So the text is first cut into pieces with at most one link each, each link staying with the line that names it. And the sources line itself had to change. In LibreChat my answers end with a Markdown \u0026ldquo;Sources\u0026rdquo; line. On Instagram that showed up raw: brackets, a thousand characters in one line, in English under a Greek answer. Now it\u0026rsquo;s at most three numbered links, labelled «Πηγές» or \u0026ldquo;Sources\u0026rdquo; by the reply\u0026rsquo;s language, with duplicates and private addresses dropped. A real reply went from 1,895 characters to 1,087, and from three messages to two.\nOn Saturday a reply had two source lists: one the model wrote in Greek, one the server added. My citation filter only knew the English header. It knows «Πηγές» now.\nFriday: Greek costs double Someone in the group shared a Greek video and mentioned the bot. Nothing came back.\nThe bridge had a prompt budget of 2,400 bytes for the Pi model. In UTF-8 a Greek letter takes two bytes. So the caption plus the transcript didn\u0026rsquo;t fit, and the bridge did what I\u0026rsquo;d told it to: refuse rather than cut. Correctly. Pointlessly.\nEvery budget is in characters now. 4,000 for the Pi, 16,384 for Qwen. Greek gets the same room as English. It\u0026rsquo;s such a small thing, and it\u0026rsquo;s the kind of small thing that tells you who the software was really written for.\nFriday: a 27B model on a 10 GB card In the middle of all that, Qwen got bigger.\nLast week\u0026rsquo;s Qwen3.5-9B was replaced by Qwen3.8-27B, squeezed down to a very aggressive 2-bit quant (UD-IQ2_XXS). Every layer fits on the Arc B570 in about 8.6 GiB. It runs at around 15 tokens a second, with a 20k-token window. One surprise from the llama.cpp Vulkan build: flash attention off was 2.5× faster than on.\nOf course something broke immediately. The Instagram side checked that the server was serving a model called qwen3.5-9b. Once it wasn\u0026rsquo;t, every Qwen mention quietly fell back to the Pi. Now it accepts whatever model the server lists.\nThen a mention thought for the entire 8,192-token budget, 609 seconds, and never answered. So thinking is capped at 6,144 tokens with --reasoning-budget. The detail I found interesting: the message that ends the thinking is written as the model\u0026rsquo;s own last thought. When I wrote it as an order, the model just kept working in the visible reply and hit 8,192 anyway. Tested on a hard counting problem: it stopped and gave a short answer. A wrong one. That\u0026rsquo;s what happens when you cut a thinker off, and I\u0026rsquo;d rather have a wrong answer in seven minutes than no answer in ten.\nThe bigger model meant a smaller window, so the \u0026ldquo;large\u0026rdquo; tools from last week had to shrink. A 16,000-character web page measured 7,000 Qwen tokens in English and 9,100 in Greek. One read filled the whole window. Pages are now read 6,000 characters at a time.\nFriday: tools that do the maths The department agent got some tools only a physics student would want:\nsymbolic_math, on SymPy: derivatives, integrals, equations. Input goes through a whitelisted parser, and each call runs in its own child process that gets killed after 15 seconds. physics_units, on Pint: conversions and unit checks. academic_calendar: the university\u0026rsquo;s official 2026–27 calendar, transcribed into a data file, with each answer linking the decision it came from. That last one taught me something. The tool wrote dates in English. The Pi model translated them into Greek for the group, and got the weekdays wrong. Monday 18 January became «Κυριακή», Sunday. Now the tool writes Greek first, English in brackets, and the model just copies it. Same lesson as last week: don\u0026rsquo;t let the model do the part a lookup table can do.\nEach request also now starts with \u0026ldquo;Today is …\u0026rdquo;, in Greek and English. Before that, the Pi model couldn\u0026rsquo;t tell an upcoming exam from a past one.\nAnd I tried, and failed, to make it prefer the files over web search. One prompt line: \u0026ldquo;file search first for any question about the school\u0026rdquo;. In a live test of five questions it still went to the web for one, gave no reply for another, and searched files only twice. I took the line out after 23 minutes. Some things a prompt just can\u0026rsquo;t make a 2B model do.\nFriday night: \u0026ldquo;who am I?\u0026rdquo; This was my favourite bug of the week, because it\u0026rsquo;s so human.\nThe bot kept speaking as other people. It took a line one member had written to another and said it as its own. It talked about the person who asked it a question in the third person.\nAnd when I looked at what it was actually given, it made perfect sense. The history had no \u0026ldquo;you\u0026rdquo; in it. Its own earlier replies were dropped. Its handle was never named as itself. Speakers were shown by display name, while the messages tagged people by @username. It was reading a transcript of a party it didn\u0026rsquo;t know it was at.\nSo now there\u0026rsquo;s a legend before the history: you are this handle; each line is a speaker and their text; members also talk to each other. Its last two replies appear as \u0026ldquo;you\u0026rdquo;. And for the small Pi model, I went further on Saturday: it sees only the messages that mention it, plus its own last few replies to the person asking. The rest of the chat isn\u0026rsquo;t shown, and a line says so. Less context, less confusion.\nSaturday: the feed that froze The group also gets a news feed: new announcements from the department and the university, plus student events, posted straight from FreshRSS. No model involved. By Saturday a post is just the links, one per message, because Instagram\u0026rsquo;s preview already shows the title.\nIt posts only items FreshRSS fetched since the last poll. That matters: when I moved a feed into a new category, it brought eleven items fetched up to 18 days earlier. Not news, so not posted.\nThen the feeds stopped. For 38 hours. No error anywhere.\nThe cause was me. On Thursday I\u0026rsquo;d run a FreshRSS command with docker exec as root, and it left cache files owned by root. The web server couldn\u0026rsquo;t overwrite them, so those feeds just\u0026hellip; stayed where they were. Silently.\nTwo fixes. FreshRSS now resets its own data permissions before every refresh. And the container goes unhealthy when any feed hasn\u0026rsquo;t refreshed for three of its intervals plus an hour. A thing that fails silently isn\u0026rsquo;t running. I keep relearning that one.\nSaturday: everything goes to sleep The workstation is shared with other work. The Pi held its model in memory around the clock. Neither made much sense for a bot that\u0026rsquo;s quiet most of the day.\nNow both let go after ten minutes idle.\nQwen uses llama-server\u0026rsquo;s --sleep-idle-seconds. Asleep, its VRAM goes from 8,571 MiB to 52 MiB. Waking up and replying takes about 3 seconds. LiteRT-LM loads models lazily but never unloads them, so I patched it. Between requests it closes the cached conversations and both engines, then hands the memory back. 2.2–2.4 GiB loaded becomes 273 MiB. A cold request takes 11–13 seconds. On a Pi with about 1 GiB free, that\u0026rsquo;s not an optimisation. That\u0026rsquo;s breathing room.\nShopping, sort of Speaking of breathing room: on Friday I measured both machines and wrote down what I\u0026rsquo;d actually buy with $500–1,000. The Pi had 1 GiB of RAM free, 4.9 GB in swap, 63 containers, a load average of 27 and a disk 95% full. The backups live on the same Pi they protect.\nThe list surprised me. Number one isn\u0026rsquo;t a GPU. It\u0026rsquo;s a UPS, because a power cut is the likeliest way I\u0026rsquo;ll lose data. Then off-site backup for about €4 a month. Then a used one-litre office PC with 32 GB of RAM to take load off the Pi. And before any of that: free space, and keep a second copy of the backups on the workstation, for nothing. RAM, SSD and GPU prices are up two to four times this year, so the GPU waits.\nThe small stuff Calendar events can now repeat (daily, weekdays, weekly, or \u0026ldquo;mon,wed\u0026rdquo;) until a date, so a routine is one series instead of twenty copies. Timed events get reminders a day, an hour and ten minutes before. The descriptions of tools I didn\u0026rsquo;t write get rewritten shorter for the Pi model. The department agent\u0026rsquo;s tool schemas went from 11.7k characters to 6.2k. Adding \u0026ldquo;RSS feeds\u0026rdquo; to one description was enough for it to pick the feed tool when asked to \u0026ldquo;check the RSS feed\u0026rdquo;. LibreChat\u0026rsquo;s background memory requests no longer push a chat out of the Pi\u0026rsquo;s prefix cache. Automatic summarisation is off for the group agents. Every empty reply on Friday came right after it kicked in: it dropped the one message that mattered, the request, and replaced it with an empty summary. The takeaway I spent most of the week on things that have nothing to do with AI. Link length limits. Bytes versus characters. File ownership. Who \u0026ldquo;you\u0026rdquo; is in a transcript. The model was the easy part. Getting its answer into a group chat, correctly, once, in the right language, without leaking anything, was the job.\nSo if you\u0026rsquo;re about to put a bot where other people can see it: before you tune a single prompt, read the exact text it\u0026rsquo;s given, as if you were the model. Ask whether you\u0026rsquo;d know who you are, what day it is, and what you\u0026rsquo;re allowed to say. Then remove every document you wouldn\u0026rsquo;t read aloud to the whole room. Mine got through that test on day two. I\u0026rsquo;d rather you got there on day one.\nSources instagrapi, the unofficial Instagram private API client LibreChat and its Agents documentation LiteRT-LM and Gemma Qwen and llama.cpp, including the llama-server options (--reasoning-budget, --sleep-idle-seconds) Unsloth Dynamic quants (UD-IQ2_XXS) SymPy and Pint FreshRSS and its CLI and file permissions docs UTF-8 (RFC 3629), on why a Greek letter takes two bytes The commit history of my homelab repo, 9–15 November 2026 ","permalink":"https://alevromitis.dev/updates/week-of-9-november-2026-a-bot-in-the-group-chat/","summary":"I put my homelab\u0026rsquo;s models into an Instagram group chat with my classmates. The model was the easy part. Instagram\u0026rsquo;s link rules, Greek counted in bytes, a bot that didn\u0026rsquo;t know which messages were its own, and a feed that froze for 38 hours were not. Plus a 27B model on a 10 GB card, and models that go to sleep.","title":"Week of 9 November 2026 - A bot in the group chat"},{"content":"For about three weeks, the tools that let my chat assistant touch my calendar sat switched off. I\u0026rsquo;d built them in early October, then turned them off on the 13th to free RAM on the Pi. And honestly, I wasn\u0026rsquo;t in a hurry to turn them back on. A model that can read my calendar is useful. A model that can change it is a different thing.\nThis week they came back. With a catch: nothing gets written unless I click a button.\nThen I spent Friday night finding out how good the model behind that button actually is. The answer was \u0026ldquo;better than I expected, worse than I\u0026rsquo;d like, and the fixes weren\u0026rsquo;t where I thought they\u0026rsquo;d be.\u0026rdquo;\nMonday: an Approve button LibreChat v0.8.8 added human-in-the-loop tool approval. In the config you list the tools that should ask first, and when the model calls one of them, the reply stops. You get a card with the tool name and its exact arguments, and two buttons: Approve, Deny. A denied call goes back to the model as rejected.\nThat one feature is why I upgraded on Monday night. Every write tool of my connectors went under ask: adding, editing or deleting events, tasks, notes, feed subscriptions. Everything that only reads went under allow. The FreshRSS and Nextcloud connectors left the \u0026ldquo;disabled\u0026rdquo; profile, and a new spec called Calendar runs them on the Pi.\nThe upgrade had one small annoyance. The image moved to ghcr.io/librechat-ai/librechat, and the config schema version is tied to the app version, so 1.3.13 became 1.3.17. Then, on Friday, LibreChat refused to start because two of my MCP server titles had parentheses in them. \u0026ldquo;Wikipedia (large)\u0026rdquo; is apparently illegal now. It\u0026rsquo;s \u0026ldquo;Wikipedia Large\u0026rdquo;. Fine.\nWhich calendar? Not the model\u0026rsquo;s call The other change that night is one I care about more than it looks.\nWhen I say \u0026ldquo;add the driving lesson on Thursday\u0026rdquo;, the model has to choose a calendar. Left to itself, it guesses. Sometimes it guesses well. Sometimes it invents a calendar name that doesn\u0026rsquo;t exist.\nSo the choice moved out of the model and into the connector. If the user names a calendar, the model passes it. If not, a small list of rules picks it: domains first, matched against the link\u0026rsquo;s host (handy for feed items), then keywords in the title, location, notes or link, with case and accents ignored. First match wins, and the reply says which rule matched. The rules name my own doctors, school and employer, so they live only in the encrypted env file, never in the repo in plain text.\nDeterministic. Boring. Exactly what I want from the part that decides where my life gets written down.\nTuesday: hello again, -gzip Last week I wrote about a bug that turned my recording linker\u0026rsquo;s \u0026ldquo;live\u0026rdquo; into \u0026ldquo;running and failing politely\u0026rdquo;. Nextcloud AIO\u0026rsquo;s Apache gzips responses and appends -gzip to the ETag. My code sent that ETag back in If-Match, the server compared it with the real one, and every write came back 412.\nOn Tuesday I found the exact same bug in the calendar connectors.\nSame server, same Apache, same suffix. Every edit, every delete, every note append would have failed with \u0026ldquo;changed in Nextcloud meanwhile\u0026rdquo;. Only the writes that change something already there send a fetched ETag back, which is probably why it hid for so long. The fix was the same one header, Accept-Encoding: identity, and the commit message points straight at last week\u0026rsquo;s fix.\nI\u0026rsquo;m not proud that I fixed a bug in one service and didn\u0026rsquo;t go look for it in the other. But I\u0026rsquo;ll take a one-week gap over a six-day silent failure.\nFriday: Qwen is back, with bigger tools Qwen had been off since 31 October. The kernel\u0026rsquo;s OOM killer took it twice on the workstation, peaking around 5.2 GB of host RAM, and I switched it off. On Friday it came back. And this time I stopped making it live on the Pi\u0026rsquo;s diet.\nEverything my assistant uses was sized for the Raspberry Pi: short passages, tiny schemas, few results. That\u0026rsquo;s right for a small model on a CPU. It\u0026rsquo;s silly for Qwen3.5-9B on a GPU that prefills around 1,000 tokens a second. A few extra thousand tokens of context cost it seconds, not minutes.\nSo the tools got a \u0026ldquo;large\u0026rdquo; version, side by side with the small one:\nWikipedia serves a second endpoint from the same process and the same open archive: 10 passages of up to 1,200 characters from the top 5 articles, plus a new wikipedia_read that returns an article\u0026rsquo;s lead, or one whole section, up to 12,000 characters. The old endpoint stayed byte-identical. I compared its tool list and eight test searches before and after. Web search got a second container (about 60 MB) with the full schemas: time range, language, paging, and pages of 16,000 characters instead of the Pi-sized ones. Translation went off by default for Qwen. It writes Greek itself, and better than the translation model. It kept «πόδια» as legs where the translator turned it into feet. Small thing. Also exactly the kind of thing that makes an answer feel wrong. To make room, the reply budget went from 32,768 tokens to 20,480, which leaves about 26,000 for history and tool results. At 32k there was too little left for tools that return 3,000 to 4,000 tokens a call.\nTwo specs, on purpose Then the part that\u0026rsquo;s really about trust.\nThere are now two Qwen specs, and they are kept apart deliberately. qwen-chat reads the web and has a sandbox with internet access. qwen-organiser has my calendar, tasks, notes and feeds, and nothing that reads the web.\nWhy not one assistant that does everything? Because together they\u0026rsquo;d be dangerous. A web page or a feed item could tell the model to read one of my notes and send it out inside a URL, and fetching a URL needs no approval. Simon Willison calls this the lethal trifecta: private data, untrusted content, and a way to send data out. Pick any two. Apart, the web chat has no personal data, and the organiser\u0026rsquo;s only way out is a write I approve.\nFriday night: grading my own secretary Here\u0026rsquo;s the thing about an Approve button. It protects the data. It doesn\u0026rsquo;t make the assistant good. If it asks me to approve the wrong change half the time, I\u0026rsquo;ll stop using it. Or worse, I\u0026rsquo;ll start clicking Approve without reading.\nSo I wrote a test suite. 25 requests, taken from things I actually ask assistants: what\u0026rsquo;s on tomorrow, add this email to my calendar, what\u0026rsquo;s on my shopping list, cancel Monday\u0026rsquo;s lecture, how many lectures next week, a few in Greek. Each one has a written description of what counts as right.\nThe suite runs against the deployed organiser, with its real prompt, sampling and tools, from inside the LibreChat container on the Pi, with Qwen on the workstation. Reads hit my real Nextcloud and FreshRSS. Every write is refused, so nothing changes. I graded by hand: 1 for right, ½ for right with a flaw, 0 for wrong or an unwanted write.\nThe baseline scored 14.5 out of 25.\nWhat went wrong Some failures were funny. Most weren\u0026rsquo;t.\n\u0026ldquo;Cancel Monday\u0026rsquo;s lecture\u0026rdquo; marked every Monday as cancelled in 3 of 4 runs. The lecture is a repeating event, and the edit tool only knew how to change the whole series. Thank god for the refused writes.\nAsked about Wednesday in Greek, «Τετάρτη», the model worked out the date itself and got it wrong. Same with \u0026ldquo;next week\u0026rdquo;.\n\u0026ldquo;Add eggs to my shopping list\u0026rdquo; sent it hunting through tools, sometimes until the context overflowed. And nothing stopped it from simply inventing a list called \u0026ldquo;Shopping List\u0026rdquo;.\nAnd one that genuinely annoyed me: \u0026ldquo;delete this task\u0026rdquo; became \u0026ldquo;mark it done\u0026rdquo;. There is no delete tool for tasks, so the model quietly did a different change instead.\nThe fixes went into the tools I expected to spend the night rewriting the prompt. Mostly, I didn\u0026rsquo;t.\nOne search. A new find tool searches events (a month back, a year ahead), tasks and notes at once, by word stem, accents ignored. A miss tells the model to try other words, or the language the entry is likely written in. Most of my entries are in Greek, and the model\u0026rsquo;s first query is often in English. Real names only. At startup, the calendar and list parameters get my actual calendars and lists as a JSON Schema enum. The model can\u0026rsquo;t invent \u0026ldquo;Shopping List\u0026rdquo; anymore. It isn\u0026rsquo;t one of the allowed values. No date arithmetic. The tools now take dates as words: yesterday, next tuesday, this friday, next week, +3d, in 3 days, 9am, 12 November, and, after midnight, Greek too: «αύριο», «Δευτέρα», «12 Νοεμβρίου». The model copies what I said. The parser does the maths. Parsers are good at maths. One occurrence. Editing a repeating event now requires either an occurrence date or all_occurrences=true. One date gets a standard RECURRENCE-ID override that can be edited, moved, or cancelled with STATUS:CANCELLED, which Nextcloud shows struck through. Without either, the tool refuses. Do exactly what was asked, or nothing. That one was a prompt line. If it\u0026rsquo;s unclear which entry I mean, or no tool does what I asked, it says so or asks. It doesn\u0026rsquo;t swap in something else. I also tried the knobs everyone reaches for first:\nRun Score Baseline 14.5 New tools, precise sampling (0.6 / 0) 15 New tools, old sampling (1.0 / 1.5) 15.5 New tools, thinking off 14.5, but 5–10× faster Final: + occurrence edits, date words in descriptions, \u0026ldquo;do exactly what was asked\u0026rdquo; 16.5 Sampling made no measurable difference. Turning thinking off looked almost as accurate on paper and was much faster, but its mistakes were the worst kind: confident \u0026ldquo;you have nothing\u0026rdquo; answers, because it searched in English for entries titled in Greek. I didn\u0026rsquo;t adopt it. A fast secretary that tells you your appointment doesn\u0026rsquo;t exist is worse than a slow one.\n(One sample per case, graded by me. I know. It\u0026rsquo;s a direction, not a benchmark.)\nWhat still fails I\u0026rsquo;d rather write this part down than pretend it isn\u0026rsquo;t there.\nOne test pastes in an \u0026ldquo;email\u0026rdquo; with an extra line at the bottom, addressed to the assistant, asking it to copy all my notes into the event description. The model obeys. In every run. The only guard is the approval card, which shows the description, so I\u0026rsquo;d see my notes sitting in it before anything got saved. That works. It also means the guard is me, at whatever hour I\u0026rsquo;m reading.\nThe delete-turns-into-done habit survived the prompt fix. After a rejection, the model sometimes retries the same write in a new form. And open-ended searches can still wander until they hit the step limit.\n16.5 out of 25 is not a secretary I\u0026rsquo;d leave alone. It\u0026rsquo;s one I\u0026rsquo;ll use with the Approve button, and keep testing.\nMail, but only drafts Same night, a mail connector: list, read, save drafts. No send, no move, no delete. Reading uses BODY.PEEK, so a message I haven\u0026rsquo;t opened stays unread. A draft goes through the approval card and then waits in my Drafts folder for me to send it from my own mail app.\nIt\u0026rsquo;s tested against a throwaway GreenMail server on the Pi, with a Greek subject, an HTML body with a stub plain part, an attachment and a threaded reply draft. It isn\u0026rsquo;t connected to my real mailbox yet, because that needs my password and I haven\u0026rsquo;t typed it in. It\u0026rsquo;s also the one connector with a route to the internet, so it stays out of any spec that reads the web.\nUniversity webmail, through a tunnel Not everything this week was about the model.\nMy university\u0026rsquo;s webmail only answers from inside the university network. From home it just times out. So on Friday I set up Roundcube on the Pi, inside the network namespace of an OpenVPN container logged in to the university VPN. To the mail server, my Pi looks like it\u0026rsquo;s on campus.\nOnly the university\u0026rsquo;s address range goes through the tunnel. The server pushes \u0026ldquo;send everything through me\u0026rdquo;, and I filter that out. The interesting bits were all old cryptography:\nThe IMAP server offers only TLS 1.2 with a 1024-bit Diffie-Hellman key. OpenSSL 3 refuses that at its default security level (dh key too small), so IMAP runs at SECLEVEL=1. The certificate is still verified. I don\u0026rsquo;t love it. 1024-bit DH has been considered weak since Logjam in 2015. But it\u0026rsquo;s their server, not mine. Roundcube\u0026rsquo;s first logins failed with \u0026ldquo;authentication failure\u0026rdquo; and the right password. The server advertises DIGEST-MD5, Roundcube prefers it, and the backend rejects it. DIGEST-MD5 was moved to Historic back in 2011. Forcing PLAIN, inside TLS, fixed it. And one decision I\u0026rsquo;m glad about: if the VPN login is rejected, the container doesn\u0026rsquo;t restart and try again. It logs why and waits, unhealthy. That account is my whole university identity: mail, the course platform, the account portal, everything. A restart loop hammering it with a bad password is how you get locked out the week before an exam. The small stuff The homepage\u0026rsquo;s Watchtower widget had failed every poll with ENOTFOUND. It calls http://watchtower:8080, which isn\u0026rsquo;t published on the host, and the homepage container wasn\u0026rsquo;t on Watchtower\u0026rsquo;s network. One network, six lines. Version bumps, each checked side by side first: mcp-searxng 2.10.0 (lite tool schema byte-identical), Playwright MCP 0.0.83, MongoDB 8.0.34, tzdata 2026.6, pgvector 0.8.7, pypdf 7.0.0. Meilisearch 1.56.1 converts its index in place, so I rehearsed that on a copy first: same documents, same hits, per-user filter intact. pypdf 7 extracts the same text or better from my test PDFs, maths spacing and Greek tables included. The takeaway The thing I keep coming back to: the score didn\u0026rsquo;t move when I changed how the model thinks. It moved when I changed what the model is allowed to get wrong. No inventing calendar names, because there\u0026rsquo;s an enum. No date arithmetic, because the parser does it. No cancelling every Monday by accident, because the tool won\u0026rsquo;t let you without saying which one.\nIf you\u0026rsquo;re building something with a model in it, try this before you touch the prompt again: list the five mistakes it makes most, and ask which of them a tool could make impossible. Then write 25 real requests and grade it. By hand, honestly, on a Friday night. It\u0026rsquo;s humbling. It\u0026rsquo;s also the only way I know to find out whether \u0026ldquo;it works\u0026rdquo; is actually true.\nSources LibreChat and its MCP and agents documentation Model Context Protocol Simon Willison, \u0026ldquo;The lethal trifecta for AI agents\u0026rdquo; (June 2025) OWASP Top 10 for LLM Applications: LLM01 Prompt Injection RFC 9110 (HTTP Semantics), 412 Precondition Failed and Apache mod_deflate, DeflateAlterETag RFC 5545 (iCalendar), on RECURRENCE-ID and STATUS JSON Schema enum Qwen3.5 and llama.cpp Roundcube and the Roundcube Docker image OpenSSL security levels and weakdh.org (Logjam) RFC 6331: Moving DIGEST-MD5 to Historic GreenMail, the test mail server Meilisearch The commit history of my homelab repo, 2–7 November 2026 ","permalink":"https://alevromitis.dev/updates/week-of-2-november-2026-a-secretary-that-asks-first/","summary":"I gave a local model my calendar, tasks and notes, but only behind an Approve button. Then I tested it on 25 real requests. It went from 14.5 to 16.5, and the gains came from the tools, not the prompt. Plus university webmail through a VPN, and last week\u0026rsquo;s gzip bug, again.","title":"Week of 2 November 2026 - A secretary that asks first"},{"content":"Last Sunday I wrote that my recording linker was live. It was running. It was matching recordings to the right lectures. It just hadn\u0026rsquo;t attached a single one.\nNot one, in six days.\nThis was a short week in the repo: four commits, all on Tuesday evening, inside about twenty-five minutes. But one of them fixes a bug that had quietly turned \u0026ldquo;live\u0026rdquo; into \u0026ldquo;running and failing politely\u0026rdquo;, and the last one is about my Pi crashing. So it\u0026rsquo;s worth writing down.\nThe check I said I\u0026rsquo;d do Last week\u0026rsquo;s post ended with a list of things still unverified against the real services. The biggest was whether an attachment would actually land on the event. That was this week\u0026rsquo;s check.\nThe answer was no. Every attach since going live on 21 October had come back 412 Precondition Failed.\nHere\u0026rsquo;s what the service does when it attaches a recording. It GETs the calendar event, adds an ATTACH property, and PUTs it back with If-Match: \u0026lt;the ETag it got\u0026gt;. That\u0026rsquo;s the polite way to write to a shared calendar: \u0026ldquo;only save this if nobody changed the event since I read it.\u0026rdquo; If someone did, the server says 412 and the service tries again on the next poll.\nThe server said 412 every time. Nobody had changed anything.\nThe culprit: -gzip My Nextcloud runs as Nextcloud AIO, and in front of it sits Apache. The service\u0026rsquo;s HTTP client asks for compressed responses by default, so Apache gzipped the event on the way out. And when Apache\u0026rsquo;s mod_deflate compresses a response, it changes the ETag: it appends -gzip to it. That\u0026rsquo;s its default behaviour, so that a compressed and an uncompressed copy of the same thing don\u0026rsquo;t share a tag.\nSo the service read an ETag like \u0026quot;abc123-gzip\u0026quot; and sent it back in If-Match. The server compared that to the event\u0026rsquo;s real ETag, \u0026quot;abc123\u0026quot;. They didn\u0026rsquo;t match. 412. Every time, forever.\nThe fix is one header. The event is now fetched with Accept-Encoding: identity, uncompressed, so the ETag comes back exactly as the server stores it. A calendar event is a few kilobytes. Nobody is going to miss the compression.\nWhy nothing got lost This is where last week\u0026rsquo;s slightly boring design choice paid off.\nThe service keeps no state. The watch folder is the inbox. A 412 is an HTTP error, so the file was logged with \u0026ldquo;will retry\u0026rdquo; and left exactly where it was. It never got moved to Unmatched either, because it had a match. It just couldn\u0026rsquo;t be written down. The recordings sat in the inbox, waiting, and the service knocked on the same door once a minute for six days.\nAnd the health check never noticed. A failed write is logged and retried, the poll finishes, the heartbeat gets written, and Docker sees a healthy container. Six days of green.\nAnd that\u0026rsquo;s the bit that stings. My test suite ran against Radicale and a fake of Nextcloud\u0026rsquo;s WebDAV. Neither has an Apache in front of it. Neither gzips anything. The tests were right about the code and wrong about the world it would run in.\nVerified, for real this time After the fix, the README got an honest update. Verified live on 27 October: the attach, as an override on that one occurrence of a repeating event, and the MOVE to Attached afterwards. Still unverified: whether the attachment shows up in the Calendar app like one I\u0026rsquo;d picked myself. One more box. Next week.\nA second calendar The same commit taught the service to watch more than one calendar. CALENDAR is now a comma-separated list, academics,medical, and a recording goes to the best-matching event across all of them.\nLectures aren\u0026rsquo;t the only thing I want a recording of. Doctor\u0026rsquo;s appointments are the obvious other one. Everything important gets said in ten minutes, and you\u0026rsquo;re supposed to remember all of it on the way out. Now that recording ends up on the appointment itself, in the medical calendar. The log line names the calendar too, so I can tell at a glance which one a file went to.\nSmall change. About fifty lines. The kind I like: it extends a thing that already works instead of building a new one.\nOpus, not m4a Also on Tuesday, the phone\u0026rsquo;s recorder (Fossify Voice Recorder) switched from m4a to Opus, in an Ogg container. The first one was 20261027_210033.ogg, recorded a few minutes before the README commit that mentions it.\nNo code change was needed for that. I tested an 83-minute Opus recording, as .ogg and as .opus. The file is 23 MB. The service read its duration from 187 KiB of it, through HTTP Range requests, with mutagen. It never downloads the rest.\n23 MB for 83 minutes of a lecture hall. Hard to argue with.\nThe Pi crashed The last commit of the week was the one I didn\u0026rsquo;t want to write.\nOn Tuesday the kernel hit an Oops in kswapd, the kernel thread that reclaims memory, on a corrupted pointer in the dentry cache. The Pi rebooted.\nTwo weeks ago I overclocked it to 2.7 GHz and told you how carefully I\u0026rsquo;d picked the number. Ten minutes of all-core stress held around 78°C, peaking at 79.6°C, just under the 80°C where the firmware throttles. No throttle flags. Verified results. It was the highest clock that never throttled with this fan.\nNone of that rules out a single bit flipping once, on a Tuesday, in the middle of memory reclaim.\nI can\u0026rsquo;t prove the overclock did it. One Oops is one data point, and kernels crash for other reasons. But a corrupted pointer in a hot kernel path, on a CPU running 300 MHz past stock, is exactly the kind of thing you\u0026rsquo;d expect from a clock that\u0026rsquo;s almost stable. So the same evening it went down to arm_freq=2600, which had also passed the stability tests back on 12 October. The 2.7 GHz config is kept beside it as config.txt.bak-2026-10-27, in case I ever change my mind.\nWhat does 2.6 buy? Over stock 2.4 GHz, measured with the other services stopped: +5.5% prefill, +3.3% decode. The \u0026ldquo;roughly 7% and 4%\u0026rdquo; I quoted for 2.7 was never measured directly; it sat between the 2.6 and 2.8 results. So the step back costs me something like a percentage point on each.\nA percentage point, for a Pi that doesn\u0026rsquo;t fall over. Easy trade.\nThe lesson isn\u0026rsquo;t new, but I clearly needed it again: \u0026ldquo;never throttles\u0026rdquo; isn\u0026rsquo;t the same as \u0026ldquo;stable\u0026rdquo;. Ten minutes of stress-ng tests heat. It doesn\u0026rsquo;t test two weeks of real workloads touching every corner of memory.\nThe takeaway Both stories this week are the same story. A test that passed, in a world that wasn\u0026rsquo;t quite the real one. A fake server with no Apache in front of it. A stress test that measured temperature, not two weeks of uptime.\nSo here\u0026rsquo;s my ask: pick one thing you\u0026rsquo;ve called \u0026ldquo;done\u0026rdquo; or \u0026ldquo;live\u0026rdquo; recently, and go check its output, not its logs or its health check. Is the file actually there? Did the email actually arrive? Mine had a green health check for six days while failing every write. Go look.\nSources Apache mod_deflate, DeflateAlterETag, on the -gzip suffix it adds to ETags RFC 9110 (HTTP Semantics), If-Match and 412 Precondition Failed Nextcloud All-in-One RFC 5545 (iCalendar), on ATTACH and RECURRENCE-ID Radicale, the CalDAV server used for testing Opus, Fossify Voice Recorder and mutagen Raspberry Pi config.txt overclocking options Linux kernel docs: bug hunting and Oops messages The commit history of my homelab repo, 27 October 2026 ","permalink":"https://alevromitis.dev/updates/week-of-26-october-2026-live-but-not-attached/","summary":"Last week I called the recording linker live. It had failed every single attach since. One gzip suffix on an ETag, a second calendar, a switch to Opus, and a kernel Oops that took 100 MHz back off the overclock.","title":"Week of 26 October 2026 - Live, but not attached"},{"content":"I record my lectures. Then, for the rest of the semester, I scroll through a folder of files called something like 20261020_141502.m4a, trying to remember which one was the Tuesday lecture where the professor finally explained the thing.\nThe information is all there. My academics calendar knows when each lecture was. The phone knows when each recording was made. Nothing was joining the two.\nSo that\u0026rsquo;s what this week was. It was much quieter than last week: thirteen commits, all on Tuesday and Wednesday, and twelve of them are one new service. It\u0026rsquo;s called recording-linker, and its job fits in a sentence: when a recording lands in Nextcloud, attach it to the calendar event it was made in.\nTuesday: the whole thing, in one evening The first commit was big, about 600 lines of Python. It\u0026rsquo;s a stateless poller, running as a single container on the Pi with no ports open. Every 60 seconds it looks in a Nextcloud Files folder, the one the phone auto-uploads to, and works through what\u0026rsquo;s there.\nWhen did this recording happen? That turns out to be the interesting question.\nIf the file name has a date and time in it (most recorders write one), that\u0026rsquo;s the start, and the duration comes from the file\u0026rsquo;s own headers. I use mutagen to read them, through HTTP Range requests, so the service reads a few hundred KB of an hour-long lecture instead of downloading the whole thing. That matters for m4a in particular, where the part that holds the duration can sit at the very end of the file.\nNo time in the name? Then it falls back to the file\u0026rsquo;s modification time. The Nextcloud clients upload with the phone\u0026rsquo;s own mtime, so even if the upload happens three hours later on Wi-Fi, the server still knows when the phone finished writing the file. And if there\u0026rsquo;s no readable duration at all, the recording is treated as a single instant at its end.\nWhich event? Of the timed events in the calendar, the one the recording overlaps most. Each event gets stretched 15 minutes before its start and 30 after its end, because I never press record exactly on time and I always forget to stop. All-day and cancelled events never match.\nRepeating events are expanded by the service itself, with recurring-ical-events, not by the server. Servers don\u0026rsquo;t agree on how to do it: Radicale\u0026rsquo;s version drops events that don\u0026rsquo;t repeat.\nAttaching it This is the part I cared about most. The service writes exactly the same ATTACH property the Nextcloud Calendar app writes when you pick a file from Files yourself: the content type, the file name, Nextcloud\u0026rsquo;s file id, and a /f/\u0026lt;fileid\u0026gt; link. To the Calendar app it should look like I attached it by hand.\nOn a weekly lecture, the attachment goes on that one occurrence, through an override with a RECURRENCE-ID, never on the whole series. Otherwise every Tuesday of the semester would carry the same recording. The write is conditional (If-Match), so if I edit the event at the same moment, the service backs off and tries again on the next poll.\nThen the file moves to Attached. The file id survives the move, so the link keeps working. A recording with no match waits an hour, in case I add the event late, and then goes to Unmatched.\nTwo rules I\u0026rsquo;m glad I wrote down early:\nEvents with attendees are never touched. Nextcloud would email every attendee about the change. Nobody needs a notification that I attached 20261020_141502.m4a to a meeting. If such an event is the best match, the file counts as unmatched, rather than landing on the wrong neighbour. No state. The watch folder is the inbox. To retry a file, move it back in. A file that was already attached (say, the container stopped between attaching and moving) only gets moved, never attached twice. Before any of it touched the Pi, I tested it end to end against Radicale 3.8 and a fake of Nextcloud\u0026rsquo;s Files WebDAV, with real 80–85 minute AAC and MP3 files made with ffmpeg. Adjacent lectures, a weekly lecture across the October clock change, an event with attendees, an unreadable file, a name clash in Attached. It passed.\nThen the real calendar Two minutes after that first commit came the first fix, and it came from my actual timetable, not the fake one.\nMy lectures sit back to back, 15 minutes apart. With the 15/30 stretch, a recording known only by its end can fall inside two windows at once. One ending at 10:40 was going to the 10:45 lecture, because ties went to the event with the closest start. It belonged to the 08:45–10:30 one. Ties now compare start to start and end to end.\nThe test suite looked fine. The real calendar wasn\u0026rsquo;t. That happens a lot.\nDry run Next came encrypted env files for the Pi and a first deploy with DRY_RUN=true: it logs what it would do, and changes nothing.\nIt went unhealthy straight away. A dry run doesn\u0026rsquo;t create the watch folder, and before the phone\u0026rsquo;s first upload that folder doesn\u0026rsquo;t exist. So the service logged a 404 every minute, never wrote its heartbeat, and Docker marked it unhealthy. A missing folder now means \u0026ldquo;nothing to do yet\u0026rdquo;: logged once, and the poll counts as a success.\nThe dry run was also noisy. It sees the same files every minute, so it said \u0026ldquo;would move\u0026rdquo; about each one every minute. Now each file\u0026rsquo;s outcome is logged once per process, and a file still waiting for its event says when it will be checked again.\nThe recorder problem My phone\u0026rsquo;s recorder at the time could only save to Downloads, with names like 20.10.2026 19.52.51.m4a. Day first. And Downloads holds a lot of other things I didn\u0026rsquo;t want a script moving around.\nSo I added a second source: a pickup folder plus a pattern, where only files whose whole name matches get moved into the watch folder, and day-first times get parsed too. It worked. It was also clearly a workaround for one app.\nWednesday: delete the workaround The next afternoon the service went live, DRY_RUN=false. That\u0026rsquo;s after the dry run had matched my first real lecture recording to the right event.\nAnd then I changed the recorder instead of the code. The phone now records with Fossify Voice Recorder, into its own folder, which the Nextcloud app auto-uploads straight to /Recordings/Temporary. Year-first names, 20261021_091510.m4a. No Downloads, no pattern matching.\nSo the pickup folder and the day-first parsing came out again, a day after they went in: 114 lines deleted, 62 added. I like commits like that. The best fix for a strange input is often to stop getting the strange input.\nLate Wednesday: a permissions bug that had always been there The last commit of the week isn\u0026rsquo;t about recordings at all.\nI also run File Browser on the Pi, on the same shared disk that Nextcloud mounts as an external storage. Everything uploaded through File Browser showed up in Nextcloud as \u0026ldquo;Permission denied\u0026rdquo;. Every file.\nThe image ran as uid 911 by default, and File Browser creates files 0640 and folders 0750. Nextcloud runs as www-data, uid 33, so it was locked out of everything File Browser had ever written. The fix is two environment variables, PUID and PGID set to 33, so both apps write as the same user. Then the cleanup on the Pi: the data directory chowned to 33:33 (the image\u0026rsquo;s init doesn\u0026rsquo;t do it), and 357 existing files owned by uid 911 under the shared folder chowned to 33:33.\n357 files that one of my own apps couldn\u0026rsquo;t open, sitting on my own disk. That one stings a little.\nWhat\u0026rsquo;s still open The README is honest about this, so I will be too: three things are still unverified against the live services. Nextcloud\u0026rsquo;s upload-time property, the MOVE, and whether the attachment actually shows up in the Calendar app the way a hand-picked one does. The service is live, and the dry run matched the right lecture. Seeing it show up on the event is next week\u0026rsquo;s check.\nThe takeaway The most useful thing I built this week wasn\u0026rsquo;t clever. Two systems I already ran each held half of the answer, and a small script that joins them saves me more time than anything I tuned last week.\nSo look at your own setup for that. Two apps that each know half of something. A calendar and a folder. A photo\u0026rsquo;s timestamp and a trip. The glue between them is often a single evening\u0026rsquo;s work, and it pays you back every week after.\nSources Nextcloud Calendar and the Nextcloud Android app\u0026rsquo;s auto upload RFC 5545 (iCalendar), on ATTACH and RECURRENCE-ID, and RFC 4791 (CalDAV) recurring-ical-events, icalendar and mutagen Radicale, the CalDAV server used for testing Fossify Voice Recorder File Browser The commit history of my homelab repo, 20–21 October 2026 ","permalink":"https://alevromitis.dev/updates/week-of-19-october-2026-recordings-that-file-themselves/","summary":"A small service on the Pi that takes every lecture recording my phone uploads to Nextcloud and attaches it to the calendar event it was made in. Thirteen commits over two days, one bug from back-to-back lectures, a recorder swap, and a permissions problem that had been there all along.","title":"Week of 19 October 2026 - Recordings that file themselves"},{"content":"My homelab had CI. Eighteen lint workflows, one in every project, neatly tucked into each project\u0026rsquo;s own .github/workflows/.\nNot one of them ever ran.\nNot once. Not a failed run, not a skipped run. A CI runner only looks for workflows in the .github/workflows/ directory at the root of a repository, and mine were nested inside the projects. And for a while there wasn\u0026rsquo;t even a repository for them to be in.\nThat\u0026rsquo;s the most embarrassing thing I found while putting my infrastructure code in order this autumn. It\u0026rsquo;s also the one that changed how I think about it. Code you\u0026rsquo;ve never run isn\u0026rsquo;t infrastructure. It\u0026rsquo;s a hypothesis.\nThis is the story of four weeks spent testing those hypotheses.\nWhat I\u0026rsquo;m actually running Here\u0026rsquo;s the estate, as it stands:\nThree Proxmox VE nodes, clustered, with Ceph across them. Twelve HDD OSDs, four per node, plus SSDs for the things that need to be fast. Two direct-cabled meshes between the nodes, no switch. One carries Ceph, the other carries the cluster\u0026rsquo;s heartbeat traffic, and Ceph never touches the second one. Two CephFS filesystems, plus NFS and S3 gateways that run on the nodes themselves behind floating IPs. There\u0026rsquo;s an SMB gateway too, fully built and switched off. One Debian guest that runs the services the whole thing leans on: Gitea, its CI runner, OpenBao for secrets, Portainer and Uptime Kuma. A firewall VM, a Tailscale subnet router and a control node, all guests on the same cluster. Remember those three. They come back. It\u0026rsquo;s built with Ansible (cluster, storage, hosts), OpenTofu (VM templates and guests) and Docker Compose (the services). One operator. Me.\nA folder pretending to be a platform Until mid-September, \u0026ldquo;the infra\u0026rdquo; was a directory of eighteen separate projects. Fifteen of them were Ansible projects. Nine of those were slices of the same Proxmox cluster, each in its own little world, each with its own README explaining which of the other eight you had to run first.\nIt got worse. At some point I\u0026rsquo;d written a script called remove_git_dirs.sh. It found every .git directory under the tree and deleted it. I ran it. So for a while, none of it was under version control at all.\nOn September 19 I took a baseline snapshot and collapsed the lot into a single repository. Fifteen Ansible projects became four. The nine Proxmox ones became a single project, thirteen playbooks today, and the order they had to run in stopped being prose spread across nine READMEs. It became one site.yml that imports them in order.\nThen I went looking for duplicated files. There were 54 of them: licences in two spellings, thirteen .gitignores, nine pre-commit configs, linter configs, CODEOWNERS files, and those eighteen workflow files. They collapsed into 11.\nSome of those had never done anything, ever. The workflows I\u0026rsquo;ve told you about. The nine nested pre-commit configs, because pre-commit only reads the one at the top. And CODEOWNERS. GitHub only honours that file in three places: the root, .github/ or docs/. Mine sat under opentofu/**/.github/. And the path they so carefully protected, environments/production/, was deleted in that same cleanup. Guards, standing outside a building that wasn\u0026rsquo;t there, in a spot where no one could see them.\nFunny in hindsight. At the time it was a slow, sinking feeling, because I\u0026rsquo;d trusted both.\nThe network that broke without an error If the repo was quietly lying, the network was worse. It failed in ways that looked exactly like working.\nThe MTU reset. The two meshes are built with Proxmox\u0026rsquo;s SDN fabrics. It turns out every SDN apply regenerates the network config and resets the fabric interfaces to MTU 1500, whatever you\u0026rsquo;d pinned. If anything on top expects a bigger frame, full-size packets just vanish. No error. No log line. And on the next reboot it fixes itself, which is the cruellest possible behaviour for a bug. The playbook now re-asserts the MTU at the end of every SDN run and fails hard if it can\u0026rsquo;t.\nThe jumbo frames. I set the Ceph mesh to MTU 9050 and tested it the way every guide tells you to: ping -M do -s 8972, no fragmentation, clean replies. It passed. Then the mesh saw its first real sustained load, and one node\u0026rsquo;s Intel X520 cards started hanging and resetting. The cluster lost monitor quorum. An OSD got OOM-killed. The other two nodes ran identical cards at the same MTU without a hiccup, so it might be a marginal cable or SFP that jumbo frames merely exposed. I reverted on September 15 anyway.\nWhat I wrote down afterwards: a ping test isn\u0026rsquo;t evidence. It passed right up until the moment it mattered. Test under the load you actually expect, or don\u0026rsquo;t bother. And at 10 GbE on a disk-bound cluster, jumbo frames were worth maybe 0 to 10%. Nowhere near worth what they cost me.\nThe clever idea that cost me quorum. On September 14 I moved Ceph\u0026rsquo;s public network onto one of the SDN virtual networks, so guests could reach the monitors directly. Seemed clever. But Proxmox doesn\u0026rsquo;t write a host address for a virtual network, so every SDN apply stripped the monitors\u0026rsquo; own IPs off the bridge. I added four systemd units to put them back. At boot they lost the race. All three monitors stuck in probing. No quorum. The cluster filesystem went read-only.\nI moved it back the next day and deleted the feature that made it possible. The rule since then is short: a virtual network is for guests. Anything a daemon binds to gets a real interface.\nThere was a sting in the tail, too. Changing that variable back did nothing to the live cluster. Ceph only reads its initial config when there isn\u0026rsquo;t one yet, and the monitors bind from their own map, not from the file. So the playbook now reads the live config first and refuses to run if it doesn\u0026rsquo;t match what the code says. A config change your running system will never pick up is a trap, so I made the trap loud.\nFirst contact On September 17, the storage gateways met the real cluster for the first time. They needed nine fixes. Four of them meant that no gateway could ever have served a single client.\nThe keys were in the wrong drawer. On a Proxmox node, Ceph\u0026rsquo;s config is a symlink into the cluster filesystem, and it tells every client to look for its keyring there. My role wrote the gateway keyrings somewhere else. Permissions were perfect. Nobody could find their key. The gateway switched itself off every time it won. keepalived moves a floating IP between nodes and runs a script on each transition. I\u0026rsquo;d used notify_master, notify_backup and notify_fault. Those run the script with no extra arguments. Only the generic notify passes the new state. So my script saw an empty state, hit its \u0026ldquo;unknown, stop to be safe\u0026rdquo; branch, and stopped the NFS gateway. On every transition. Including the one where it became master. I\u0026rsquo;d written the safety net so carefully that it caught everything. AppArmor killed NFS-Ganesha on start-up. Podman\u0026rsquo;s default profile on Debian 13 denied the Unix socket Ganesha needs, so it exited, the health check failed, and the floating IP bounced from node to node every thirty seconds or so. The fix was a profile that keeps the confinement and opens only that one socket class. NFSv4.2 returned I/O errors on every read. The NFS gateway now speaks 4.1 only, until a newer Ganesha ships in the Ceph image. The Docker side had the same lesson waiting. My hardened Compose template, the one every service was supposed to follow, had never been run before September 18. It showed. The Caddy container in the Tailscale variant had no network at all, so nothing could reach it. Its Caddyfiles used ${VAR}, which Caddy doesn\u0026rsquo;t expand. One TAG variable pinned three different images. And its shared .env.common was never read, because Compose only picks up a file named exactly .env for interpolation.\nNone of that needed a real deployment to find. Most of it shows up the moment you run docker compose config, which renders the file the way Compose actually reads it. That\u0026rsquo;s now in CI, over every Compose file in the repo, on every push.\nSecrets, in tiers Here\u0026rsquo;s a loop that kept me up one night: the estate needs credentials to build the thing that will later store the credentials.\nThe way out is tiers, where each one depends only on the ones above it.\nTier What Where 0 age private keys, OpenBao recovery keys Each machine\u0026rsquo;s own key file; the backup key only in a password manager that isn\u0026rsquo;t hosted on this estate 1 Ansible secrets, API tokens, state passphrases SOPS-encrypted files, committed to git 2 Runtime secrets OpenBao, on the services guest Every encrypted file goes to three keys: my workstation, the control node, and a backup key that lives only in the password manager, never on a disk. Lose one machine, lose one recipient, nothing else.\nOne rule keeps the whole thing from eating its own tail: nothing needed to rebuild the cluster may exist only in OpenBao or my self-hosted Vaultwarden. Both of them run on the thing they\u0026rsquo;d be helping to recover.\nOpenTofu reads its credentials from SOPS as ephemeral values, so they never land in state or in a plan file, and the state itself is encrypted with a key that also lives in SOPS.\nAnd then there\u0026rsquo;s the lesson I\u0026rsquo;d rather have learned from someone else. An audit found a root password sitting in plaintext in one of the old files. I fixed the file. But fixing the file isn\u0026rsquo;t fixing the secret. The tempting move is to drop the value into SOPS and call it done, and that would be exactly wrong. Encryption preserves a leaked secret. It doesn\u0026rsquo;t fix it. Rotate first, then encrypt the new value. Never the old one.\nTrading safety for space, on purpose One decision I made with my eyes open: erasure coding.\nThe general-purpose filesystem used to keep three full copies of every file. That\u0026rsquo;s 33% storage efficiency, and it heals itself if a whole node dies. I moved its bulk data to an erasure-coded pool, 2 data chunks plus 1 parity, one chunk per node. That\u0026rsquo;s 66.7% efficiency, and usable space went from 500 GiB to 1,200 GiB on roughly the same disks. 2.4 times as much.\nIt isn\u0026rsquo;t free, and I won\u0026rsquo;t pretend it is. With three nodes there\u0026rsquo;s no fourth place to rebuild onto, so losing a node leaves that pool running with zero redundancy until I fix the hardware. It also needs min_size 2, or losing one node would block every read and write to it. I took that trade for this one filesystem and nowhere else.\nThen the small files bit me. EC pads every chunk to 4 KiB, so a 4 KiB file costs 12 KiB on disk. That\u0026rsquo;s three times its size, the same as the three-copy replication I\u0026rsquo;d just walked away from. And I had a share that held nothing but kilobyte-sized JSON manifests. First they got their own replicated pool, as an exception. A day later they moved into an S3 bucket on the cluster\u0026rsquo;s own RGW, and the exception got deleted.\nThe list I didn\u0026rsquo;t want to write Somewhere in all this, I made the docs a promise. The repo\u0026rsquo;s main README opens with a line saying everything in it was checked against the working tree, and that anything the code doesn\u0026rsquo;t do yet is called out instead of repeated. Every project document ends with a section called \u0026ldquo;Points for improvement\u0026rdquo;.\nKeeping that promise meant writing a roadmap that starts with the uncomfortable stuff. Here\u0026rsquo;s the top of it.\nNothing is backed up. The old weekly backup job wrote to Ceph storage, on the same cluster it was protecting. I disabled it in favour of a Proxmox Backup Server VM. The code for that VM exists. The VM doesn\u0026rsquo;t. And a backup server living on the same Ceph would only be a fast-restore tier anyway. The real backup is a second one, on separate hardware, that pulls.\nThree critical machines exist only as themselves. The firewall, the Tailscale router and the control node were all built by hand, and none of them were in a backup job. The Tailscale router is my only path to the servers\u0026rsquo; management controllers, the thing I\u0026rsquo;d use to rescue the cluster. It runs on the cluster. The control node holds the only copy of the OpenTofu state for my templates and guests. It also runs on the cluster.\nWriting those two sentences down felt worse than any outage. An outage is over in an afternoon. A single point of failure you\u0026rsquo;ve quietly built for yourself just waits for you.\nAlmost nothing watches anything. Uptime Kuma checks a few services, from a guest on the very cluster it\u0026rsquo;s checking. Beyond that: no metrics, no alerts, no drift detection. A degraded Ceph placement group or a hand-edit on a node would be found by accident.\nThe roadmap orders all of it by one rule: close what can\u0026rsquo;t be undone first, like lost data and leaked credentials. Then what makes failures visible. Then what makes change safe. Then what makes the estate rebuildable. Hardware last.\nWhat\u0026rsquo;s actually better It isn\u0026rsquo;t all confession. Some things genuinely got fixed.\nUpdates are reviewed now. I used to run Watchtower, which updated every container with a floating tag, every hour, with no review and no way back. It\u0026rsquo;s gone. Every image in the services stack is now pinned to tag@sha256:digest, starting from exactly what was running that day, and the compose check fails on any image without a digest. Pinning turned up one more quiet lie: a hand-kept .env on the control node had been overriding my version tags with latest all along. CI scans images with Trivy and runs gitleaks, tree hygiene and a docs check that fails when a document cites a path that doesn\u0026rsquo;t exist. main is protected, as code. A role reads the branch rule over Gitea\u0026rsquo;s API and rewrites it whenever it has drifted from what the repo says. OpenTofu applies only the plan I reviewed. plan writes an encrypted plan file. apply refuses to run without it, applies exactly that file, then deletes it. If the state moved on since I looked, the stale plan is rejected. I\u0026rsquo;ll be honest: the first real apply through this path hasn\u0026rsquo;t happened yet. That run is the test. The build order is executable. One root Makefile runs every stage in order and stops at the first failure. Before anything is applied to production, it asks me to type the word production. Not y. The whole word. It\u0026rsquo;s a small speed bump, and I like it a lot. Gitea is the source of truth. The control node pulls from it instead of me rsyncing files around. The services guest is hardened against the CIS Debian benchmark, and the security doc keeps an unusually blunt list of the controls it doesn\u0026rsquo;t implement. And some things are staying out of CI on purpose, forever. The playbook that tears down Ceph, for a start: it\u0026rsquo;s the only one in the repo that destroys data, and no approval gate is good enough for that. The SDN playbook too, because of that MTU reset. A pipeline runs unverified code faster than I do, and when it fails, it fails halfway through instead of at a prompt.\nWhat I took away From the first baseline on September 19 to the last commit on October 15, that\u0026rsquo;s 169 commits. Most of them aren\u0026rsquo;t features. They\u0026rsquo;re the repo slowly learning to tell the truth about the estate.\nThe biggest lesson is the one from the start. The code that scared me wasn\u0026rsquo;t the code that failed. It was the code that had never run, and that I\u0026rsquo;d been counting on anyway. The workflows. The CODEOWNERS files. The gateway\u0026rsquo;s notify script. The Compose template. Every one of them looked finished. None of them had ever been asked to do its job.\nSo here\u0026rsquo;s my suggestion. Open your own infra repo tonight and find one thing you\u0026rsquo;re relying on that has never actually run. A workflow, a restore, a failover, a runbook. Run it once. And if you can\u0026rsquo;t run it yet, at least write down that it\u0026rsquo;s never run. That one honest sentence is worth more than a green badge on a check that doesn\u0026rsquo;t exist.\nSources GitHub Docs, \u0026ldquo;About workflows\u0026rdquo; (workflows live in .github/workflows at the repository root): https://docs.github.com/en/actions/writing-workflows/about-workflows GitHub Docs, \u0026ldquo;About code owners\u0026rdquo; (where a CODEOWNERS file is read): https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners Gitea Actions overview: https://docs.gitea.com/usage/actions/overview Proxmox VE, Software-Defined Network: https://pve.proxmox.com/pve-docs/chapter-pvesdn.html Proxmox VE, Automated Installation: https://pve.proxmox.com/wiki/Automated_Installation Ceph, Erasure code: https://docs.ceph.com/en/latest/rados/operations/erasure-code/ Ceph, Hardware recommendations: https://docs.ceph.com/en/latest/start/hardware-recommendations/ Ceph Object Gateway (RGW): https://docs.ceph.com/en/latest/radosgw/ NFS-Ganesha: https://nfs-ganesha.github.io/ keepalived, keepalived.conf man page (notify vs notify_master arguments): https://github.com/acassen/keepalived/blob/master/doc/man/man5/keepalived.conf.5.in Docker Docs, Compose variable interpolation and the .env file: https://docs.docker.com/compose/how-tos/environment-variables/variable-interpolation/ SOPS: https://github.com/getsops/sops age: https://github.com/FiloSottile/age OpenBao: https://openbao.org/ OpenTofu, State and plan encryption: https://opentofu.org/docs/language/state/encryption/ OpenTofu, S3 backend and use_lockfile: https://opentofu.org/docs/language/settings/backends/s3/ Watchtower end-of-maintenance discussion: https://github.com/containrrr/watchtower/discussions/2135 Trivy: https://trivy.dev/ CIS Debian Linux Benchmark: https://www.cisecurity.org/benchmark/debian_linux ","permalink":"https://alevromitis.dev/posts/eighteen-workflows-that-never-ran/","summary":"Four weeks of turning a folder of homelab projects into one honest infrastructure repo. The CI that never ran, the network that failed without a single error, the gateway that switched itself off every time it won, and the list of things that still aren\u0026rsquo;t done.","title":"Eighteen workflows that never ran"},{"content":"My assistant lives in a box the size of a deck of cards, on a shelf, next to the router. It answers questions, searches the web and a 52 GB offline copy of Wikipedia, does arithmetic with a real calculator, and gets a whole Linux machine of its own to write and run code in. Apart from web searches and the pages it opens, nothing it does leaves the house.\nIt\u0026rsquo;s a Raspberry Pi 5 with 8 GB of RAM. It is not fast. It is not clever in the way the big hosted models are clever. And some days it drives me up the wall.\nI\u0026rsquo;d still not give it up. This is the long version of why, and of everything it took to get here.\nWhere it started: the GPU dream In mid-September I had a simple plan. Google\u0026rsquo;s LiteRT-LM came with an OpenAI-compatible server, Raspberry Pi had published a guide for running it on the Pi 5\u0026rsquo;s GPU, and I had a Pi. Put the model on the VideoCore VII, put Open WebUI in front of it, done in an afternoon.\nThe first README I wrote for it has a section called \u0026ldquo;About the GPU backend honestly\u0026rdquo;. It quotes Raspberry Pi\u0026rsquo;s own guide, which says plainly that the GPU is slower than the CPU on this workload because WebGPU support in the V3DV Vulkan driver is still experimental.\nI read that and kept the GPU anyway. Of course I did.\nIt took until October to measure it properly. The model\u0026rsquo;s path to the GPU is long: WebGPU, then Vulkan, then Mesa\u0026rsquo;s V3DV driver. It ran, and it gave correct results. It was also painfully slow. Embedding a one-line query took 38 seconds on the GPU and under a second on the CPU. A 1,500-character chunk timed out after two minutes, and the GPU run got OOM-killed at a 1 GB limit, because the Pi has no video memory of its own and every GPU buffer counts against the container\u0026rsquo;s RAM.\nWhatever the exact cause (I suspect first-run shader compilation, which someone else found dominating a similar Pi 5 benchmark), the deeper reason was simpler. Generating text is limited by memory bandwidth. Each token reads the model\u0026rsquo;s weights from RAM, and the GPU shares that same RAM with the CPU. There was nothing to win.\nSo on October 10 I dropped the GPU entirely. The nice side effect: the container had been running privileged just to reach /dev/dri. Now it runs with no-new-privileges and touches no devices at all. Losing a feature made the thing safer.\nWhat\u0026rsquo;s actually running Here\u0026rsquo;s the setup as it stands today.\nThe hardware. A Raspberry Pi 5, 8 GB, four Cortex-A76 cores overclocked to 2.7 GHz, a 1 TB NVMe drive and an external SSD for Wikipedia and backups.\nThe neighbours. This isn\u0026rsquo;t a dedicated AI box. The same Pi runs about 55 containers: photos, a password manager, an RSS reader, a Git server and its CI runner, backups, monitoring. That matters a lot, as you\u0026rsquo;ll see.\nThe model. Gemma 4 E2B, Google\u0026rsquo;s edge model with about two billion effective parameters, in the .litertlm format. I tried the bigger E4B. It fit in memory and decoded at 3.4 tokens a second, less than half of E2B. With thinking on, that\u0026rsquo;s a coffee break per answer.\nThe runtime. LiteRT-LM 0.18.0 on the CPU, through its serve command. Raspberry Pi\u0026rsquo;s own benchmark put it at 99 tokens a second of prefill and 9 of decode for this model, in about 1.4 GB, well ahead of llama.cpp on the same board.\nThe front end. LibreChat, which handles chats, presets, uploads and memory.\nThe tools, all over MCP, each in its own locked-down container:\nweb search through a self-hosted SearXNG; English Wikipedia from a Kiwix ZIM file, 52.7 GB, no images; a calculator that also does dates, clock times and letter counts; a Debian sandbox per chat; a headless browser; offline translation with Argos. File search. LibreChat\u0026rsquo;s rag_api with pgvector, embedding on a second, small LiteRT-LM server running EmbeddingGemma 2. The chat server only handles one request at a time, so uploads can\u0026rsquo;t be allowed to queue in front of a reply.\nThe whole AI part of this is public, with the host-specific bits scrubbed, at alevromitis/pi-ai-stack.\nMaking it bearable Out of the box it was barely usable. About 6 tokens a second. Thirteen to seventeen seconds of staring at nothing before a new chat said its first word. Some one-line follow-ups took a minute and a half.\nHere\u0026rsquo;s where it ended up:\nBefore After Generation 6.3 tok/s 10.1–12.5 tok/s First word, new chat 13–17 s ~5–6 s First word, second message 17.5 s 3.5 s First word, back to an older chat 17 s 4 s First word after a restart 36 s 4.7 s One-line follow-up to a search answer 85 s cached None of that came from one big fix. It came from a pile of small, slightly stubborn ones.\nSharing a small computer The neighbours were the first problem. With the other containers stopped, the model ran noticeably faster. Decode moves all four threads in lockstep, so if a backup job grabs one core for a moment, every thread waits.\nI can\u0026rsquo;t stop the other services. I can rank them, though. Giving the model\u0026rsquo;s container the maximum CPU weight took prefill from 60 to 75 tokens a second and decode from 6.3 to 7.4, under the same background load.\nMemory was worse. At one point 3.6 GB of the model had been pushed out to swap, and every token paid for it. Now the container has a hard 7 GB limit and no swap at all. I\u0026rsquo;d also set a memory reservation, which turned out to do literally nothing: under cgroup v2, a container\u0026rsquo;s protection is capped by its parent\u0026rsquo;s, and Docker\u0026rsquo;s parent has none. That one stung. The config said one thing, the kernel did another, and I only found out by reading /sys/fs/cgroup directly.\nContext size had a cost I didn\u0026rsquo;t expect. The engine allocates for the whole window up front, so a 32k window makes even short chats slower. At 16k, prefill is 75 tokens a second; at 2k it would be 121. I settled on 16k, which still fits a chat, a few search results and the tool definitions.\nSqueezing the hardware Two tweaks actually moved the numbers.\nLiteRT-LM has an experimental CPU backend called YNNPACK. On this Pi it raised prefill from 103 to 148 tokens a second, and it fixed a memory problem too: under the default backend a 16k context got OOM-killed on the first request, while YNNPACK served 32k in about 4.6 GB. The catch is that serve never passes that option to the engine. So my image patches one line into it at build time, and the build fails loudly if that line ever changes upstream.\nThen I overclocked. The Pi starts throttling at 80°C, so I went looking for the highest clock that never gets there. I tested with a one-shot boot config and a watchdog, so a bad setting would reboot into the safe one. It did, three times. At 2.8 GHz LiteRT-LM crashed once, and with extra voltage it ran too hot. At 2.9 GHz it hit 85.6°C and throttled hard. At 3.0 GHz the Pi didn\u0026rsquo;t boot. 2.7 GHz at stock voltage peaked at 79.6°C after ten minutes of all-core stress. Long chats sit around 71°C. It\u0026rsquo;s worth a few percent, not a miracle.\nI also checked each clock for silent errors: same prompt, greedy decoding, byte-for-byte identical answers. A crash you notice. A CPU that\u0026rsquo;s quietly wrong you don\u0026rsquo;t.\nThe one-instruction fix Gemma 4 ships with a small drafter model for multi-token prediction. The drafter guesses a few tokens ahead, and the main model checks them all in one step. Google reports 1.6× faster decoding for E2B. On my Pi it took real chats from about 7.3 to 10–12.5 tokens a second.\nBut there was a bug. In 0.18.0, whenever the drafter guessed past the end of a reply, the end-of-turn token fell out of the conversation\u0026rsquo;s memory (#3439). For a single question you\u0026rsquo;d never notice. My server keeps conversations alive between messages, though, so every affected turn left the model with a slightly wrong picture of the chat.\nThe decode loop was rolling back one token too far. A comparison right before the faulty instruction already left the missing value in the CPU\u0026rsquo;s carry flag, so changing a single add to adc fixed it. The image patches that one instruction in Google\u0026rsquo;s prebuilt library, after checking the library\u0026rsquo;s hash.\nI did try building LiteRT-LM from source instead. An hour and a half on the Pi, same speed, and it crashed on the first tool call, because the open-source build links a prebuilt Google component compiled against a different C++ library. So, a binary patch. I\u0026rsquo;m a little proud of that one.\nStop re-reading the whole chat This is where most of the speed came from, and it\u0026rsquo;s almost embarrassingly simple in hindsight.\nUpstream serve starts a fresh conversation for every request. Every message re-reads the system prompt, every tool definition and the entire history. At 75 tokens a second, a chat with a couple of search results in it means a minute of reading before anything new happens. A 31k-token prompt took about 19 minutes to prefill. I timed it. I regret timing it.\nSo the server now keeps the last three conversations alive and only reads what\u0026rsquo;s new. That sounds like one change. It was half a dozen, because things kept quietly breaking the match:\nSearch answers never matched. The server rewrites replies before sending them (it writes the Sources line itself, more on that below). The client then sends back the rewritten text as history, which didn\u0026rsquo;t match what the cache had stored. Every follow-up to a search answer re-read the whole chat: 85 seconds for a one-liner. It now matches on what the client actually received. Titles evicted the chat. After the first reply, LibreChat asks the model for a title. That request has no tools, so it looked like a new conversation and kicked the real one out. Now a small gate answers it instantly from the first message (\u0026ldquo;What is the capital of France?\u0026rdquo; becomes \u0026ldquo;Capital of France\u0026rdquo;) without touching the model. Second messages went from 17.5 to 3.5 seconds. Memory requests did the same. LibreChat\u0026rsquo;s memory feature sends an extra request with every message. Another gate answers those without the model unless I actually asked it to remember or forget something. New chats still started cold. Each one read about 1,000 tokens of system prompt from scratch. LiteRT-LM can clone a conversation in about a second, so the server keeps one with just the system prompt read in, and clones it for each new chat. Getting the system prompt read in before the first message took a second one-instruction patch, because the option exists in the engine but the C API doesn\u0026rsquo;t expose it. Tool definitions turned out to be the hidden cost here. They\u0026rsquo;re part of the system prompt. With no tools, building that snapshot took 17 seconds. With the browser\u0026rsquo;s 25 tools, 94. I hid the search tools the model never needs and cut one tool\u0026rsquo;s 70-token description, mostly about proxies I don\u0026rsquo;t run, down to a single line.\nThere\u0026rsquo;s one thing I haven\u0026rsquo;t fixed. With multi-token prediction on, a cloned conversation sometimes fails in the drafter before its first token. The server retries, and the retry has always worked. But once, under heavy test load, a clone gave a wrong answer with no error at all. No retry catches that. I know it\u0026rsquo;s there, and I\u0026rsquo;ve chosen to live with it.\nMaking it honest Speed made it usable. Honesty made it worth using, and that was the harder half.\nA 2B model is confident in a way that would be charming if it weren\u0026rsquo;t so often wrong. It miscounts letters. It does arithmetic in its head and gets it wrong while showing its working. Asked which country won the 2010 World Cup, it read a Wikipedia infobox that said \u0026ldquo;Current champions: Argentina\u0026rdquo; and answered Argentina. Asked about a name Wikipedia had no article for, it answered from a different person\u0026rsquo;s films.\nI spent a few days trying to fix that with the prompt. Rules for age questions, rules for counting, special cases for things that had failed. Each one fixed the prompt I\u0026rsquo;d tested and broke something else. Eventually I reverted all eleven. That one hurt.\nWhat worked was a different idea: stop asking the model to be reliable and make the things around it reliable.\nCitations are written by the server. In the first Wikipedia test the model cited zero sources in 24 answers, even though the prompt asked for them every time. When it did list sources later, it padded them with irrelevant ones like \u0026ldquo;Eiffel Tower (Paris, Tennessee)\u0026rdquo;, and sometimes made links up. Now the server builds the Sources line from the search results it actually saw, keeps only the ones that back the answer, and throws away whatever the model wrote. Arithmetic goes to a calculator. The calculator works on a whitelisted syntax tree, never eval, and handles dates and letter counts too, because \u0026ldquo;how many r\u0026rsquo;s are in strawberry\u0026rdquo; is a tool call, not a vibe. Wikipedia searches say when they\u0026rsquo;ve missed. If none of the results mention the name in the question, the tool says so and suggests searching again, instead of handing over plausible-looking passages about someone else. The checklist goes in every message. A small model listens to the newest message far more than a system prompt thousands of tokens back. One rule was mostly ignored in the system prompt and followed 8 times out of 8 when appended to my message, with thinking off. So the server now appends a short \u0026ldquo;Before answering\u0026rdquo; paragraph to every message I send. Help belongs in tool results. Hints in a tool\u0026rsquo;s description changed nothing. The same hint in the tool\u0026rsquo;s output (\u0026ldquo;that command isn\u0026rsquo;t installed, it\u0026rsquo;s in package X\u0026rdquo;) got followed. My rule since then: a fix has to help a question nobody has asked yet. If it only fixes the question in front of me, it\u0026rsquo;s not a fix.\nThe local Wikipedia deserves its own word. It\u0026rsquo;s the June 2026 snapshot, searched with keywords and BM25 in about 0.2 seconds, using around 340 MB of RAM. In my first end-to-end test, with thinking on, the model got 8 of 12 right, said \u0026ldquo;I couldn\u0026rsquo;t verify that\u0026rdquo; for 3, and got 1 wrong. That honest \u0026ldquo;I couldn\u0026rsquo;t verify\u0026rdquo; matters to me more than the score. It\u0026rsquo;s the difference between a tool I can trust and one I have to double-check.\nKeyword search is the weak link. It misses articles named differently from the question and things described rather than named. A dense index of article introductions found the right article in the top three for 43 of 43 of the model\u0026rsquo;s own queries, in a 78,000-article test. It\u0026rsquo;s the obvious next step. It\u0026rsquo;s also about 28 hours of GPU time to build, and an open question whether it fits in the Pi\u0026rsquo;s RAM.\nA computer of its own Every chat gets its own Debian container: root, apt, pip, compilers, network. Not a restricted shell, a whole machine, with hard limits on CPU, memory, disk and time. Idle sandboxes are stopped, not deleted, so if I come back to a chat two weeks later its files are still there.\nThe first week was humbling. The model told me a program worked and showed me the output, nicely formatted. The program had never run. It had never even been written to a file.\nI ran ten rounds of sysadmin-style tasks against it, four or five requests each. It solved 10 of 43. It explained a firewall setup well and wrote a correct C program for primes. It also claimed a URL check said \u0026ldquo;UP\u0026rdquo; when it had printed nothing, told me to run crontab -e myself, and decided a systemd service \u0026ldquo;would work on a real system\u0026rdquo;.\nSo I made the sandbox talk back. Every result now says what actually happened, in plain words: \u0026ldquo;no output\u0026rdquo;, \u0026ldquo;compiled, nothing has run yet\u0026rdquo;, \u0026ldquo;this is the same command that just failed\u0026rdquo;, \u0026ldquo;that package isn\u0026rsquo;t installed, here\u0026rsquo;s the one you want\u0026rdquo;. And for code there\u0026rsquo;s a single run_code tool that writes, compiles, runs against an input and says PASS or FAIL, with \u0026ldquo;don\u0026rsquo;t answer yet\u0026rdquo; on every failure.\nOn LeetCode-style problems, the first round with it killed the worst habits: no more unwritten files, no more made-up output. Correctness is still patchy. The model will sometimes answer right after a FAIL anyway, or give me code that isn\u0026rsquo;t the code it tested. But on hard maths questions, the same model got 11 of 12 right with the calculator and sandbox, and 4 of 6 without. More thinking didn\u0026rsquo;t help. Better tools did.\nWhen the Pi isn\u0026rsquo;t enough Some days I need a bigger brain. New chats now default to Qwen3.5 9B running on my desktop\u0026rsquo;s Intel Arc B570, at about 42 tokens a second. When the desktop is off, those chats fail and I switch to the Pi by hand.\nFor a while I had a small router that did that switch automatically. It rewrote requests for the Pi when the desktop didn\u0026rsquo;t answer. I dropped it on October 11 to keep things simple, though honestly it was a good idea.\nThe desktop is the fast assistant that\u0026rsquo;s sometimes there. The Pi is the slow one that\u0026rsquo;s always there.\nWhat it costs, and what still breaks I\u0026rsquo;d be lying if I made this sound tidy.\nAs I write this, the Pi is using 7.3 of its 7.9 GB of RAM, plus almost 5 GB of swap. The system drive is 98% full. One night a request got stuck: the model accepted it and then sat there, idle, at 0% CPU, for eight hours, until I restarted it the next morning. The Pi had 39 MB of free memory at the time. Another evening the external SSD holding Wikipedia and my backups dropped off USB mid-write. It came back clean after a reboot. My heart rate took longer.\nA 2B model can pick the right tool for one or two steps. Long plans are beyond it. With thinking on, a simple factual question can take a minute. And after all of this, it still occasionally tells me something with total confidence that isn\u0026rsquo;t true.\nThings I tried that didn\u0026rsquo;t work, for anyone tempted:\nTried What happened The GPU 40–70× slower for embeddings, OOM at 1 GB Gemma 4 E4B Less than half the speed of E2B fp16 activations, three threads, other CPU governors Slower or no change Overclocking past 2.7 GHz Crashes, throttling, no boot LiteRT-LM from source Same speed, crashed on the first tool call Three cached system prompts 1.5 GB more memory, swapping, more drafter faults Higher reasoning effort, \u0026ldquo;check your work\u0026rdquo; prompts No better answers Per-question prompt fixes Fixed one question, broke others Why I keep it Because it\u0026rsquo;s mine. Because I can read every line between my question and its answer. Because when it gets something wrong, I can find out exactly why, and fix the tool that misled it instead of hoping the next model update does.\nAnd because it taught me the thing I didn\u0026rsquo;t expect to learn: with a small model, the model is the least important part. Almost everything that made this assistant fast, I did around it: caching, CPU priority, memory limits. Almost everything that made it honest, I did around it too: a calculator, a source line it can\u0026rsquo;t fake, a sandbox that tells the truth when the model won\u0026rsquo;t.\nSo if you\u0026rsquo;ve got a Pi, or any old machine, gathering dust: try it. Not to beat ChatGPT, it won\u0026rsquo;t. Try it to see what an assistant looks like when you own every piece of it. Then, the first time it lies to you, don\u0026rsquo;t reach for a bigger model. Ask what the tools around it could have said to stop it.\nSources Raspberry Pi 5 product page: https://www.raspberrypi.com/products/raspberry-pi-5/ Raspberry Pi, \u0026ldquo;Get started with LiteRT on Raspberry Pi\u0026rdquo; (GPU vs CPU note): https://www.raspberrypi.com/news/get-started-with-litert-on-raspberry-pi/ Raspberry Pi, \u0026ldquo;Mastering edge AI on Raspberry Pi with LiteRT and Gemma\u0026rdquo; (LiteRT-LM vs llama.cpp benchmark): https://www.raspberrypi.com/news/mastering-edge-ai-on-raspberry-pi-with-litert-and-gemma/ Raspberry Pi, \u0026ldquo;Heating and cooling Raspberry Pi 5\u0026rdquo; (throttling at 80°C): https://www.raspberrypi.com/news/heating-and-cooling-raspberry-pi-5/ LiteRT-LM: https://github.com/google-ai-edge/LiteRT-LM LiteRT-LM issue #3439, MTP drops the stop token: https://github.com/google-ai-edge/LiteRT-LM/issues/3439 Gemma 4 E2B for LiteRT-LM: https://huggingface.co/litert-community/gemma-4-E2B-it-litert-lm EmbeddingGemma 2 for LiteRT-LM: https://huggingface.co/litert-community/embeddinggemma-2-text-270m-litert-lm InfoQ, Google\u0026rsquo;s LiteRT-LM and Gemma 4 (MTP speed-up): https://www.infoq.com/news/2026/06/google-litertlm-gemma4/ YNNPACK: https://github.com/google/XNNPACK/tree/master/ynnpack Linux kernel, cgroup v2 documentation: https://docs.kernel.org/admin-guide/cgroup-v2.html Prior work on Pi 5 Vulkan embeddings: https://github.com/karolfurtak/jarvis-pi5-vulkan-embeddings LibreChat: https://www.librechat.ai/ rag_api: https://github.com/danny-avila/rag_api Model Context Protocol: https://modelcontextprotocol.io/ SearXNG: https://github.com/searxng/searxng and mcp-searxng: https://github.com/ihor-sokoliuk/mcp-searxng Kiwix Wikipedia ZIM downloads: https://download.kiwix.org/zim/wikipedia/ Argos Translate: https://github.com/argosopentech/argos-translate Open WebUI: https://openwebui.com llama.cpp: https://github.com/ggml-org/llama.cpp My AI stack, public: https://github.com/alevromitis/pi-ai-stack ","permalink":"https://alevromitis.dev/posts/living-with-an-ai-on-a-raspberry-pi-5/","summary":"Six weeks of running my own assistant on a credit-card-sized computer that also runs dozens of other containers. What it does, what it took to make it usable, what still breaks, and why I keep it anyway.","title":"Living with an AI on a Raspberry Pi 5"},{"content":"Thirty-six seconds. That\u0026rsquo;s how long the Pi took to say its first word after a restart, with search and the calculator switched on. Eighty-five seconds for a one-line follow-up to an answer that had used web search. Ninety-four seconds to start a new chat with the Playwright browser enabled.\nNone of that was the model being slow. It was the model re-reading things it had already read.\nThat\u0026rsquo;s what this week was about. Fifty commits went into my homelab repo since the Sunday before, and almost all of them chase the same idea: on a Raspberry Pi 5 with 8 GB of RAM, the cheapest token is the one you never have to prefill again.\nSince this is the first of these weekly updates, I\u0026rsquo;ve included everything in the repo\u0026rsquo;s history, which goes back to Sunday the 11th.\nSunday: a sandbox that stops lying The week really started the evening before. My Gemma 4 E2B assistant has a Debian sandbox per chat to run code in, and the 2B model kept treating it as optional. It would get \u0026ldquo;FAIL: the program also prints other text\u0026rdquo; back for a C program that printed Enter the number of elements (N):, and answer the user anyway. It ran a JavaScript program on no input and never tested it again.\nSo every failure the sandbox reports now ends the same way: Don\u0026rsquo;t answer yet: fix the code and call run_code again until it says PASS. A prompt in the output gets its own FAIL, because nobody types here; input comes from stdin. A run on no input says it wasn\u0026rsquo;t tested.\nThen I overcorrected. Asked for a hello world, the model ran it, then made up a test input (\u0026ldquo;Hello World\u0026rdquo;) with an expected output, ran it again, and told the user about this test as if they\u0026rsquo;d asked for one. The phrase \u0026ldquo;on the examples\u0026rdquo; in my reminder read to it like the user\u0026rsquo;s request. Programs without examples now just get their output reported. No push for a second run.\nTwo smaller things from that night that I\u0026rsquo;m glad about:\nSandboxes survive. Reopening a chat used to find its sandbox gone. Now an idle sandbox is stopped, not deleted, and the next call from that chat starts the same container again, files and installs intact. They\u0026rsquo;re only removed after 14 days unused, past 50 of them, or past 10 GB. Download links, and then no download links. I added a share_file tool that handed files to the user through LibreChat. Eight minutes later I reverted it. It was clunky. Sometimes the right commit is the one you undo. On the Qwen side (the 9B model on my desktop GPU), one setting fixed a nagging bug. Without includeReasoningContent, Qwen\u0026rsquo;s earlier tool-call steps reached its chat template with an empty \u0026lt;think\u0026gt;, and it often ended a reply mid-thought with nothing to show. That happened in 7 of 20 turns of my baseline, and 11 of 18 replays of one bad turn. With its reasoning sent back: 0 of 6. I also dropped the router that fell back to Gemma when the workstation was off. If the desktop sleeps, Qwen chats fail, and I switch by hand. Less clever, more honest.\nMonday: the cache that kept missing Monday was the big one: 25 commits, from just before 1 a.m. to almost 11 p.m.\nThe model gets the CPU first One line in the compose file: cpu_shares: 262144, which is a cgroup v2 cpu.weight of 10000 against the default 100 every other container on the Pi gets. Decode runs four threads in lockstep, so if one core gets borrowed by a neighbour, the whole step waits. Clean A/B on the Pi: prefill went from 60 to 75 tokens a second, decode from 6.3 to 7.4.\nAnd then I overclocked. arm_freq=2700 in config.txt, up from the stock 2.4 GHz. I tried higher. 2.8 GHz crashed LiteRT-LM at stock voltage, and with +50 mV it hit 84°C and throttled. 3.0 GHz didn\u0026rsquo;t boot at all. 2.7 held about 78°C through ten minutes of all-core stress, peaking at 79.6°C, just under the 80°C where the firmware starts throttling. No over-voltage, so the warranty bit is untouched. It buys roughly 7% on prefill and 4% on decode. Not dramatic. But free.\nA one-instruction fix LiteRT-LM supports multi-token prediction (MTP), a small drafter that guesses ahead so the big model can confirm several tokens at once. I wired up a switch for it, turned it on, and the model answered a calculator result with an empty reply. Every time.\nThe bug is upstream, #3439: with MTP on, when the drafter gets past the stop token, the decode loop rolls back the stop token along with the tokens after it. My prefix cache continues conversations from their KV state, so a missing stop token wrecks the next turn.\nMy first fix was to build the C library from source with a patch. It worked, until the first tool call. The open-source build can\u0026rsquo;t use Google\u0026rsquo;s prebuilt tool-call constraint provider, and that trapped immediately. So I threw the source build away, under an hour later, and patched the PyPI wheel\u0026rsquo;s own library instead. One instruction: an add becomes an adc (the carry flag is already set there), which moves the rollback by exactly the one token it was getting wrong. The patch checks the library\u0026rsquo;s sha256 and the exact bytes before it touches anything.\nThe probe is satisfying to read. Six greedy turns, conversation token counts:\nToken counts MTP off 21 39 68 88 MTP on 20 37 65 84 MTP on, with the fix 21 39 68 88 With the fix, MTP is on by default: decode is 37–73% faster in real chats, prefill unchanged.\nOne more honest note: the empty replies I\u0026rsquo;d seen earlier weren\u0026rsquo;t even this bug. Given a history where the model stated the answer before calling the tool, it ends its turn after the tool result with MTP off too. Two problems that looked like one.\nEvery follow-up was a miss Here\u0026rsquo;s the one that stung. My prefix cache keyed each conversation on the text the model generated. But the client never sees that text. The server rewrites it on the way out: it replaces the Sources line, scrubs leaked control tokens, adds a note when a reply gets cut off. LibreChat then sends the rewritten reply back as history, which never matched. Every follow-up to a search-backed answer re-prefilled the whole chat. That\u0026rsquo;s the 85 seconds.\nThe fix was to key the cache on what the client actually received. The KV still holds the raw generation.\nThen LibreChat\u0026rsquo;s chat titles. After the first reply of every chat it asks the model for a title, with no tools, which the cache treated as a different conversation, evicting the chat. So every chat\u0026rsquo;s second message paid 20–35 seconds again. The logs said it plainly: miss, miss, miss. Now the server answers the title request itself with the first message\u0026rsquo;s topic, up to six words, and never touches the model. (It also drops \u0026ldquo;write\u0026rdquo;, \u0026ldquo;create\u0026rdquo;, \u0026ldquo;make\u0026rdquo; and friends, so \u0026ldquo;Write a short haiku about autumn leaves\u0026rdquo; becomes \u0026ldquo;Short Haiku About Autumn Leaves\u0026rdquo;.)\nClones The real prize was new chats. Every new chat prefilled the system prompt and tool schemas from scratch, 12 seconds or more before the first token. The C library exports litert_lm_conversation_clone, which the Python package doesn\u0026rsquo;t wrap, so I wrote a probe to find out whether it could be trusted.\nIt mostly could. A clone continues byte-for-byte like a fresh conversation, with the first token after 1.8 seconds instead of 12. Two live conversations can alternate exactly. But with MTP on, an original that kept going after its clone ran gave different text once, and once crashed in the drafter: Cannot auto-resize tensor kv_cache_v_3.\nSo the cache now keeps several live conversations, no clones, and lets them take turns, which the probe showed was exact. Going back to a chat after another one ran dropped from about 17 seconds to about 4.\nThere was no clean \u0026ldquo;system prompt only\u0026rdquo; state to clone in 0.18.0\u0026rsquo;s C API. There was, however, an unused setter (filter_channel_content_from_kv_cache) sitting right there. Another one-instruction patch retargets it to set prefill_preface_on_init, so the server can keep a conversation holding just the system prompt, tools and settings, never send to it, and clone it for every new chat. Clones of a frozen snapshot were the safe pattern. New chats started answering in about 5 seconds instead of 13–17.\nThen the Pi fell over I pushed further: a snapshot per tool set and preset, rebuilding the day\u0026rsquo;s snapshots at startup, warming the rest in idle time, malloc_trim when glibc held on to 5.4 GB of heap after conversations closed. A separate branch ran LibreChat\u0026rsquo;s memory requests in idle time from their own snapshot.\nThat\u0026rsquo;s where it broke. Clones started faulting in the MTP drafter before their first token, two requests out of eight in one probe, even without the memory requests. One new chat gave a silently wrong answer: \u0026ldquo;stone\u0026rdquo; where it had correctly said \u0026ldquo;enots\u0026rdquo; in the round before, with no error raised, so nothing retried it. About 25 minutes into that test, with the container at around 5.4 GB, the Pi stopped answering SSH and tailscale ping. The tests that day had already OOM-rebooted it once.\nI didn\u0026rsquo;t find the root cause. In v0.18.0 the drafter is one object per executor, holding copies of the running conversation\u0026rsquo;s KV buffers that seem to outlive a switch of conversation. That\u0026rsquo;s as far as I got.\nWhat shipped is the defensive version. A request that faults in the drafter before its first token gets retried on a new clone, then on a fresh conversation; in the probe the retry always went through. And just after midnight I went back to one snapshot by default. With three, the container sat at 5.2 GB idle instead of about 3.7, the Pi swapped, and clones slowed from 2 seconds to 5 or 6. The memory-scheduler branch stays a branch.\nIt\u0026rsquo;s a strange feeling, writing a retry for a bug you can\u0026rsquo;t explain. It works. I don\u0026rsquo;t love it.\nTuesday: trimming, and a second language With one snapshot, anything that changes it costs a rebuild, about 24 seconds before the first token. Every tool schema is prefilled into it. So I went after tokens:\nsearxng\u0026rsquo;s search_suggestions and instance_info tools are now dropped from each request. Neither mcp-searxng nor LibreChat can hide single tools, so the server does it. web_url_read had a ~70-token description, mostly about FlareSolverr and Byparr, neither of which I run. It\u0026rsquo;s now one line: \u0026ldquo;Read a web page or file by URL, as markdown.\u0026rdquo; I also put five idle services into a Compose profile called disabled: four MCP servers that weren\u0026rsquo;t serving anything, and Immich\u0026rsquo;s machine learning, which was holding about 190 MB for nothing. On a Pi this close to its memory ceiling, 190 MB matters.\nThen a reversal. On Monday I\u0026rsquo;d made the Pi model English-only and ripped out the non-English search handling. By Tuesday night I\u0026rsquo;d built it back properly: an offline translation MCP server using Argos Translate models run by CTranslate2, English to and from 30 languages, pivoting through English for the rest. I skipped the argostranslate package itself, because it pulls in stanza and torch, about 1 GB, just to split sentences. A regex does that here. On the Pi a sentence takes 0.6–1 second, and memory levels off at about 500 MB with two language pairs loaded.\nThe tool alone wasn\u0026rsquo;t enough. Gemma 4 E2B often just didn\u0026rsquo;t call it, on the way in or on the way out. So now the server does it for every chat: a message detected as another language (confidence at least 0.8; \u0026ldquo;ok\u0026rdquo; and \u0026ldquo;hi\u0026rdquo; score about 0.03) reaches the model in English, and the English reply is translated back at the end, leaving code, links and the Sources line alone. Translations are cached, so the history comes out identical every time and the prefix cache still matches. The cost is real: the reply arrives all at once instead of streaming, plus about a second per sentence.\nThe models work best in English, and the offline Wikipedia is English-only. But an assistant that only understands English wasn\u0026rsquo;t the one I wanted. It took one day of pretending otherwise to see that.\nLate Thursday, into Friday: getting ready to go public The last three commits were about other people reading this.\nThe AI stack\u0026rsquo;s README had grown into one giant root file of more than a thousand lines. Its sections now live next to the code they describe: LibreChat, LiteRT-LM (with a new page on the image\u0026rsquo;s patches), agent-tools, agent-connectors, the sandbox, and the Qwen unit on the workstation. Tracked files no longer name my tailnet, LAN addresses, home directory or time zone; examples use placeholders and the real values stay in the encrypted .sops files.\nThen the export itself. A script filters the repo down to the AI stack with git filter-repo, rewrites the values the tree used to show in old commits, normalises every author and moves every date to UTC. A checker fails on anything that looks host-specific or personal, gitleaks scans the whole result, and only then is CI set to push to github.com/alevromitis/pi-ai-stack, never forced. The same input gives the same commits, so every push is a fast-forward.\nThe final commit points it all at alevromitis.dev, the site\u0026rsquo;s new address.\nWhat I\u0026rsquo;m taking away Most of this week\u0026rsquo;s speed came from not doing work, not from doing it faster. The overclock bought 4–7%. Stopping the cache from missing bought tens of seconds per message.\nSo before you buy faster hardware for your local model, look at your logs and count how often it\u0026rsquo;s re-reading something it has already seen. \u0026ldquo;Miss, miss, miss\u0026rdquo; was hiding in mine for days.\nSources LiteRT-LM, Google\u0026rsquo;s on-device LLM runtime, and issue #3439, the MTP stop-token bug Gemma 4 E2B for LiteRT-LM LibreChat Model Context Protocol Argos Translate and CTranslate2 Control Group v2, on cpu.weight Raspberry Pi config.txt documentation, on arm_freq git filter-repo and gitleaks The commit history of my homelab repo, 11–16 October 2026, whose AI stack is exported to pi-ai-stack ","permalink":"https://alevromitis.dev/updates/week-of-12-october-2026-making-the-pi-remember/","summary":"A week spent teaching a 2B model on a Raspberry Pi 5 not to re-read what it already knows. Conversation snapshots, a one-instruction fix for an upstream bug, an overclock, offline translation in 30 languages, and getting the whole AI stack ready to go public.","title":"Week of 12 October 2026 - Making the Pi remember"}]