I run a lot of projects in parallel. Family, company, a relocation, training, the usual chaos. The pile kept growing past what I could hold in my head, so I built an always-on assistant to externalize the load. Calendar, todos, project context, reminders, email triage, all behind a single chat I can reach from any device. Nothing exotic about the goal. The interesting part is what it took to build one I'd actually trust with the contents of my real life.

This post is the architecture and the security choices. It's opinionated and specific. If you're considering the same thing, the trade-offs below will save you a few weekends.

Before tooling: write down what you're protecting against. Mine, roughly:

• A compromised laptop or phone shouldn't immediately leak family schedules, visa documents, contacts, or work strategy.

• A compromised API provider shouldn't have a copy of any conversation that touches real data.

• The agent itself, if it misbehaves, shouldn't have free reign over my home directory, my SSH keys, or my cloud accounts.

• A leaked git history shouldn't be readable to whoever finds it.

That last one matters because the assistant's context lives in a git repo I sync across devices. The repo is the brain.

I picked the Hermes Agent framework from NousResearch. Open source, batteries included (tool use, memory, skills, hooks), self-hostable, well documented. It's the only framework I found where "run a real personal agent on my own hardware" was an obvious target rather than an afterthought.

Around it:

• Inference: nano-gpt as the default provider, routing GLM-5.1 through their :wafer suffix for ZDR (more on this below). Optional secondary route through uomirouter on Qwen3.6-27B when I want speed over privacy.

• Messaging: Matrix with end-to-end encryption (mautrix + olm). Bot account on matrix.org for now, will self-host Synapse later.

• Calendar: Google Calendar via an MCP server (the @cocal/google-calendar-mcp package) running as a local subprocess. Scope restricted to read + create.

• Workspace: a single private git repo holds todos.md, projects/<slug>.md, a system prompt (SOUL.md), config, and a few custom skills. Hosted on Codeberg, encrypted in transit and at rest in git via git-crypt.

• Sandbox: every shell command the agent runs goes through rootless Podman with the workspace mounted in and nothing else from the host visible.

The host is a Linux box with LUKS full-disk encryption that stays on 24/7.

Most of the work in this build was making the agent useful AND making sure I wouldn't be embarrassed if any one component got rooted. A few specific decisions that did the heavy lifting:

The default model is zai-org/glm-5.1:wafer, routed through nano-gpt's Wafer tier. The :wafer suffix is the part that matters: it's their zero-data-retention path. Requests are processed and dropped. No training, no logs of prompt content. I verified this against their published policy and I plan to re-verify periodically rather than take it on faith.

GLM-5.1 itself is capable enough for everything I throw at the assistant in practice: planning a week, summarizing an email thread, drafting a follow-up, querying my todos, deciding which project a new task belongs to. It's not the best model on every benchmark, but it's the best ZDR model I have access to with a price that lets me run an always-on bot without anxiety about cost.

If you want to try the same stack: nano-gpt referral link.

The opt-in alternate route is uomirouter on Qwen3.6-27B. Less private (their gateway sees full request content), but fast and cheap, and useful when I'm doing throwaway work where the privacy posture doesn't matter. I wired it as a slash command (/uomi) that switches the session for one cycle, with a loud banner on every reply reminding me the call is going through a different path. Default never moves; the alternate is one tap away when I want it, and one /reset away when I don't.

Uomi gives free credit on signup and a bit more via referral link ($3.50 free up front, $5 on top through the link). Enough to validate the route without committing.

Hermes can use either Docker or Podman as the runtime for its terminal tool. I went with rootless Podman for three reasons:

1. No root daemon. Docker runs a privileged daemon by default. Podman doesn't. The agent's containers spawn under my user account without any process running as root. If the daemon is the most attractive attack surface in a typical Docker setup, removing it is the right move.

2. User-namespace isolation by default. Rootless Podman maps the container's UID 0 to my user's subuid range, not actual root. A container escape gets you my user, not the host's root. You wanted that anyway, but here it's the default rather than something you opt in to.

3. Drop-in compatibility. Podman accepts the same docker run flag set, so all the security hardening (--cap-drop ALL, --security-opt no-new-privileges, the tmpfs mounts, the pids limit) just works. Switching the runtime didn't change the agent's behavior at all.

Inside the sandbox: only the workspace repo is mounted, read-write, with --userns=keep-id so file ownership comes through correctly. None of ~/.ssh, none of ~/.config, none of my other dev repos. The agent can git status and grep the workspace, but it cannot see the SSH key it'd theoretically need to push, the OAuth tokens for calendar, or any other secret on the host. The container even runs without network for most operations; outbound calls go via specific tools the agent has to explicitly invoke.

The trade-off: the agent can't run git push from inside the sandbox (no SSH key visible). Solved by a small post-turn hook on the host that auto-commits the agent's workspace edits locally; pushes remain a manual action I run. Reversible work the agent does itself, irreversible work I gate.

Hermes ships with two switches I leaned into:

• security.redact_secrets: true scrubs tool output, logs, and chat responses for things that pattern-match as credentials before they leave the agent's view. API keys, JWTs, AWS access keys, anything that looks like a bearer token gets [REDACTED]-ed in the output the model sees on its next turn. Stops the obvious failure mode where a tool's stderr leaks a secret into the conversation, which the model then innocently repeats back to you.

• privacy.redact_pii: true does the same for names, addresses, phone numbers, email addresses in the output stream. Useful when the agent processes a third party's email or a calendar invite. Less useful for my own context (the agent literally needs to know who I am and who my family is to do its job), but the upstream pipeline gets a privacy-clean view either way.

Both default to off; turn them on early.

The workspace repo holds my real life: open projects, partner's name, kids' schedules, what I'm building for work. I push it to Codeberg for backup and cross-device sync, but the parts that matter are encrypted with git-crypt before they leave the box. The remote sees ciphertext for SOUL.md, config.yaml, todos.md, projects/**, the skill files, and the protocol files. Filenames + commit graph are visible (they're inherent to git), but content is not.

The git-crypt key lives in ~/.config/git-crypt/keys/ at chmod 600, backed up to a password manager as a base64 secure note and to an offline encrypted medium. Losing the key means losing the encrypted history with no recovery, so back it up before your first push, not after.

Hermes lets you ship custom skills (instructions the agent loads on demand based on intent match) and lifecycle hooks (Python handlers that fire on events like session-start, session-end, agent-end). I wrote three skills early because the agent kept misrouting project briefs into its memory tool instead of the project files:

• add-project: when I say "open a project on X" or paste a brief, write projects/<slug>.md from a template AND add a section to todos.md. Not memory.

• update-project: edits to status, log entries, new info on an existing project. Same separation.

• list-projects: read projects/ directly, never enumerate memory entries as if they were projects.

And one hook (workspace-git-sync) that does the boring stewardship: pull the workspace on session start, auto-commit on agent end if the agent left anything uncommitted, never push (that stays manual). The result is that the brain stays current across devices without me thinking about it.

• No browser automation. Tempting, but a browsing agent is an attack surface I don't want yet. Skip until I have a specific need.

• No subagents / delegation. Single agent, single thread of execution. Easier to reason about.

• No autonomous skill creation. The agent can suggest skills; I write them. Keeps the trust surface bounded.

• No SimpleX yet. I wanted it (E2EE without metadata leakage), but the v6 daemon's WebSocket subscription pattern didn't play nicely with the Hermes adapter on first attempt. Parked behind Matrix until I can dig in.

• No vector DB. Hermes's built-in memory and full-text session search cover what I need. Adding a vector store is exactly the kind of feature-creep that builds dead infrastructure.

The pattern is: enable when there's a concrete need that the current setup can't meet, not because it's an option in the docs.

1. Write the threat model first. A page is enough. Decide what's acceptable and what isn't BEFORE you start picking tools, because the tool choices are mostly downstream of those decisions.

2. Pick ZDR inference and stick to it. It limits your model menu, but not by much. GLM-5.1 on Wafer or one of the other ZDR routes is enough for any assistant-shaped task.

3. Rootless Podman, not Docker. Same flags, fewer privileged surfaces.

4. Encrypt the brain. git-crypt is twenty lines of setup and it means the contents of your repo are unreadable to the host you're syncing through.

5. Auto-commit, never auto-push. Lets the agent maintain itself without ever doing something you can't quickly undo.

6. Skills over prompt-hacks. When the agent does the wrong thing, write a small skill that targets the intent. Lasts longer than reprompting.

7. Use the alternate-provider escape hatch. Have a fast/cheap route ready for the throwaway work; keep the ZDR route as the default. Don't mix them by accident.

Links one more time:

• nano-gpt (ZDR default route, GLM-5.1:wafer): https://nano-gpt.com/subscription/k7AXDwBb

• uomi (opt-in fast route, Qwen3.6, $3.50 free + $5 via referral): https://uomirouter.uomi.ai/auth/signup?ref=XZG8J3E8

Both referral links if you want to support the writeup. Both products I actually use.

metaend