The Models Are Learning to Think Without Telling Us

Spent a chunk of Sunday arvo down a r/singularity rabbit hole instead of doing the thing I was supposed to be doing, which was cleaning the gutters before the next storm rolls through. The gutters can wait. This couldn’t, apparently.

The thread that got me was about a reported OpenAI model, apparently codenamed Astra, that’s meant to do something called “latent space reasoning” or “recurrent depth” thinking. The short version, as best I can follow it: current reasoning models like the ones you’d use in ChatGPT or Claude do their thinking out loud, in text, one token at a time. You can watch them work through a maths problem step by step, second-guess themselves, go “wait, let me check that” like a mate talking through a crossword clue. It’s clunky and slow, but you can read it. This new approach apparently lets the model think internally, in its own abstract representation, without translating every step into words first. Faster, possibly deeper, but opaque.

One commenter put it plainly: this is bad for interpretability. Another walked through why, and it stuck with me: right now, even the “shown working” chain-of-thought isn’t a perfectly honest window into what the model’s actually doing internally, researchers have already found cases where the stated reasoning and the real computation diverge. But at least there’s something to read, something to audit, however imperfect. If a model starts reasoning in a purely internal, non-linguistic space, that something to read disappears. You just get the answer, delivered from a black box that used to at least have a little window in it.

I’ve spent a fair chunk of my working life around systems where you can’t see inside the box. In DevOps you learn pretty quick that “it works” and “I understand why it works” are two very different sentences, and the gap between them is where the 2am pages live. You build logging, tracing, dashboards, alerting, all of it in service of one goal: when something goes wrong, you want to know why, not just that it did. The entire discipline exists because black boxes fail in ways nobody predicted, at the worst possible time.

Now scale that instinct up to something that might eventually be making decisions that matter, medical, legal, military, financial, and take away the log file. That’s the trade being floated here. Faster, more capable reasoning in exchange for less visibility into how the machine got there. And the thing is, I get why labs would want it. Spatial reasoning, for instance, isn’t naturally a text problem. When I’m working out whether the new bookshelf will actually fit past the hallway corner, I’m not narrating it in full sentences, I’m just seeing it. Making a model reason more like that, in whatever its version of “seeing” is, probably does make it better at certain kinds of problems. That part’s genuinely interesting, and I won’t pretend otherwise.

But there’s a tension here that nobody’s resolving, and I don’t think it can be resolved, only managed. Everybody wants capability. Nobody wants a mystery. You can’t fully have both, and the commercial incentives point very hard in one direction. One person in that thread mentioned this method’s been described elsewhere as a “redline”, something the field had been wary of for alignment reasons, and now that one major lab has apparently used it to catch up, the argument goes, everyone else will feel forced to follow. That’s the same dynamic you see in a lot of races: nobody wants to be the one who unilaterally disarms, so the caution evaporates the moment someone else drops it.

I don’t know how worried to be about this, and I’m suspicious of anyone, on either side, who tells you they do know. The AI safety crowd has cried wolf before and been wrong about timelines. The AI boosters have also been wrong, constantly, in the other direction. Somewhere in a Reddit thread full of people arguing past each other is probably the most honest position available right now: this is a real trade-off, it’s happening quickly, and the institutions meant to have any say in how it happens, governments, regulators, the public, are roughly a decade behind the conversation.

None of which stops me being fascinated by all of it, which is its own uncomfortable admission. I’ll read the next benchmark thread just as eagerly as this one. I just think it’s worth saying out loud, occasionally, that “fascinating” and “fine” aren’t the same word. The gutters, at least, I can still see inside.