It was nearing the middle of summer when i found myself in a high rise apartment overlooking the Muiderzand beach. I had just been welcomed in by the two friendly pensioners living there, cradling a coffee and admiring their lovely view. It was not the jovial inhabitants of this futuristic looking dune settlement that brought me here though, it was their hardware. Specifically the M1 Mac Studio with 32GB of ram they were selling via Marktplaats, the Dutch equivalent of Craigslist. At about €1200 this was the maximum i was willing to invest in my latest fascination. People had been going nuts about their OpenClaws doing their bidding on the internet for a few months already so prices for anything that could run a model locally, even the lowly 32GB M1 were in high demand. While i was pensively inspecting this particular model the current owner of this particular Mac asked if was going to use it for the AI thing. Yes i said. I was indeed going to use it for the AI thing.
Why i want my inference local
When the first squeals of OpenClaw and its brethren started flooding the socials i was mildly interested. But the breathless stories of what it was supposed to be able to do did not seem to match the mess i found when i tried running one on my laptop. So i ended up trying lightweight versions (cleaner but they did not do much) and cloud based versions (useful but now all my personal stuff needed to be in their cloud) but ended up settling on a simple Hermes based setup. Not really a claw but more like an agent with benefits. Through its Telegram integration i could still talk to it remotely and its design did not seem terrible, with some interesting and delightfully simple approaches to learning. But an agent that stops when you close your laptop lid is not really an agent. So that’s how i found myself in Almere Duin on that lovely day in June, eventually walking out of that high-rise building Mac Studio under my arm.
Once set up my Hermes started doing odd jobs for me; mostly working on my many open source projects and keeping things up to date, looking for new issues to fix, searching marktplaats for good deals, asking me if i already did my push-ups that day. Typical helpful assistant stuff. Most of this it was doing using my ChatGPT account and a few Chinese models though umans.ai but i was still dreaming of doing most of this with a local model. If this is a homelab it should be self-sufficient right? It’s looking at all my local stuff so how cool would it be if it did not need some American corporation to do the actual thinking.
At about the same time i also discovered cocore.dev whose founders were dreaming of building some sort of global shared AI resource, where we could all just run models from each others machines and not be dependent on any single billionaire. This seemed like something worthwhile to figure out.
The problem was that while 32GB is more memory than is available in most MacBooks it’s still a rather dinky amount of ram in the world of LLMs. So while some models do run on a Mac this size their brains were peanut size and their output riddled with the sort of errors we now fondly remember GPT 2 making. My experiments trying to figure out which local models would actually run on this hardware were initially focused on models that could do some basic coding. But after a few weeks tinkering with the benchmark, trying a new model, tinkering with the model, and a lot of waiting on tests to fail it finally started to became clear to me that maybe a strong coding model was not in the cards for this setup.
So what does work?
Maybe right out of the gate; there is no world where you run frontier models on your home lab. Even spending €20.000 will ‘only’ get you the latest Mac Studio with 512GB ram (if in stock) and even that will only run a B tier Chinese model. Now these ain’t bad, in fact they are quite marvellous and closely nip the heels of A tier models. But spending that same money on American hosted inference will get you more tokens that you will know what to do with for many many years so that’s not really a deal unless you need the privacy of local at any cost.
A more sane option would be to get something that has 128GB of ram. Still expensive but maybe not eye watering. And this same machine will run Deepseek Flash 4 or 4.1 at acceptable speeds. You could install Dwarfstar for this, which is doing a great job optimizing for this scenario. This is not a bad place to be in, Deepseek is very very good for its size and each new iteration seems to add new magic. For 80% or more of what you’re now doing with AI this will be absolutely fine.
But i don’t have 128GB nor an M5 processor and i haven’t been able to reason myself into buying one yet. So I’m doing the reasonable thing and trying to build a Dwarfstar for poor people called Mei. Mei, like Dwarfstar, is an inference engine for local models fine-tuned for very specific hardware and models. And in my case for a very specific use-case. Since i gave up the local coding dream I’m now focusing on local AI assistant tasks. The updated local model benchmark i use is now focused on tool-use and assistant tasks like reading email, sorting things into folders, and of course asking me how many pushups i did today. This at least seems pretty viable right now and some models are actually quite decent at doing it.
So far I’ve found Qwen 3.6 and Ornith 1.5 to be especially helpful. These models are so called Mixture of Experts models which basically means that unlike ‘dense’ models they only need the currently active experts to be loaded into memory (plus some overhead). This makes it possible for these models to do some actual work and leave some room for the conversation context in memory as well. Pairing that with some trickery for optimized caching, tensors fully aligned for memory optimization and a bunch of other micro optimizations also gets fairly decent speeds of about 20 tokens/second. Making it feel about as fast as a typical ChatGPT conversation.
What’s next?
Now at this point i should probably mention that I’m not actually using it for any agentic work yet. All I’m doing is tweaking and testing basically. And since doing that exhausts all the memory this machine can spare i don’t actually get to run a helpful assistant next to that work. When new interesting models come out that fit the basic shape and size of Qwen 3.6 i also load those up and see if they improve things but all in all this is primo bike shedding.
Still, it’s interesting and fun, and it seemed like something i should write up at some point since I know at least some of you are interested in this stuff.
So if anything go try out Mei and tell me how it breaks or works for you! I’m starving for some actual real world input. And if you think I’m mad and should actually be trying out X or Y instead that is also fun to hear about. The local frontier is here and it’s starting to come into reach of more and more people. But I’m still paying the big labs to bring it closer in the meantime. Perhaps some day a duck tape and elastic band network of affordable computers will spring into existence and I’d love to be there when that happens.