The Slowest Possible Answer to the Right Question

Saw a thread this week about someone running a 744-billion-parameter model on a machine with 25GB of RAM. Not fast. About 0.1 tokens per second, which works out to roughly 8000 tokens a day if you leave it running. That’s a handful of proper questions answered, once every twenty-four hours, on hardware you could buy at JB Hi-Fi.

My first reaction was the same as half the comment section: what’s the point. Then I sat with it for a bit, on the train home, and changed my mind slightly. That happens more than I’d like to admit.

The obvious criticism is that nobody’s going to use this for real work. Fair enough, I’m not going to fire up a laptop and wait three days for a code review. But someone in the thread made a point that stuck with me: getting a genuine expert opinion on something hard, if you don’t have money or access, can already take weeks. Suddenly a day doesn’t sound so bad. It’s not competing with ChatGPT’s five-second response. It’s competing with not having an answer at all.

There’s a tension here I don’t think you can resolve neatly, and I’m not going to pretend otherwise. On one hand, this is a genuinely interesting bit of engineering: streaming hundreds of gigabytes of model weights off a disk, keeping the dense layers hot in memory, treating the whole thing less like inference and more like a very patient library. On the other hand, it’s also true that llama.cpp has been able to do a version of this for over a year, and a chunk of the excitement in that thread was really about the idea of running something enormous on something small, rather than the thing itself being new. Both of those can be true. I don’t need to pick a side.

What actually got me thinking was the sub-thread about who this is even for. Someone mentioned space exploration, which sounds like a stretch until you remember that a probe on Mars doesn’t care if an answer takes six hours, because sending the question to Earth and waiting for a reply takes longer anyway. Someone else mentioned remote listening posts, disaster zones, places where connectivity isn’t a given. I spend my working life assuming connectivity is a given. It’s a very Melbourne, hybrid-office, NBN-took-a-decade-but-we-got-there kind of assumption. Most of the world doesn’t get to make it.

I did a version of this thinking a few years back, not with AI, with backups. We had a house fire scare, nothing dramatic, an electrical fault that got caught early, but it made me actually think about what happens if the internet just isn’t there one day. Not in a prepper way, more in a “huh, I should probably have a local copy of the photos” way. This model-on-a-potato-laptop thing scratches the same itch. It’s not about performance. It’s about not being entirely dependent on someone else’s data centre in Texas being switched on and feeling generous.

The bit of the discussion that got genuinely heated was about whether the project was “vibe coded”, meaning largely AI-generated, and whether that mattered. Half the thread wanted to argue about code purity. I don’t especially care how it was built if it works, though I’ll admit as someone with a dev background I have a soft spot for understanding why something works, not just that it does. That’s probably an old habit more than a principle.

None of this changes my worry about where all this compute actually comes from, or what running frontier models at scale is doing to power grids and water use, a topic that gets less airtime than it should. A model that runs slow on your own hardware, using your own electricity, for your own question, is at least honest about its cost in a way that a free chatbot answer never has to be. You feel the wait. You feel the fan spin up. There’s something almost old-fashioned about that, in a good way.

I don’t think most people will ever run a 744B model off a laptop drive at one token every ten seconds. But I like that someone tried it, and I like that it forced a few hundred strangers to argue about what “usable” even means. That’s not nothing.