building jarvis (minus the weaponry)

I’m a tinkerer.

I’ve been using Home Assistant for a while now - and with the recent advances with AI and personal assistants, I started thinking (which is always frustrating for my wife) - can I give my flat a personality? The full blown JARVIS/Tony Stark experience, minus all of the weaponry of course.

I’ve always liked having smart home gadgets, but it always seems a cheap imitation of the movies - I say, “Hey Google, do the thing”, it does the thing, finished, zero thought, zero warmth, zero personality.
No context.
No memory.
Minimal capability.

Home Assistant is incredibly customisable, and I already have a pretty capable AI assistant with personality and persistent memory… Can I connect these things together? The only challenge would be that it could get pretty expensive pretty quick, I’d need to figure out how to manage API costs. So I decided to build a an integration that hooks up local models to Home Assistant with Ollama.

The goal was to run a small always on model (Qwen 3.5 9b) to add a proactive “consciousness” layer that tied into Home Assistants existing event bus, and then a larger, “on demand” model (Qwen 3.5 35b) that adds a personality layer. This combined with a persistent memory architecture, and a 3rd tier (Frontier LLM) for complex reasoning felt like it would give be what I was looking for. A smarthome that’s exactly that, smart. It understands the world state, it builds context, it remembers conversations, it knows who I am and my preferences, it can take certain actions completely proactively.

Does it work? Mostly, it’s a work in progress. In typical fashion, I over engineered the hell out of it, and got distracted by some shiny things along the way. Getting the balance right between the tiers is more challenging than I expected, and it’s nowhere near the sci-fi vision yet. But it’s a promising step in the right direction.

But most importantly, it’s a hell of a lot of fun and I’m learning so much that I always thought I would never have the time for.

Anyone else tinkering with local AI and home automation? I’d love to hear what you’re trying.