Posts / ai
Good Taste Is the Thing We Can't Benchmark
Someone on r/LocalLLaMA posted a few hours of first impressions of a new coding model, and the comment section has stayed with me. The model is Tencent’s Hy4-flash. The post didn’t talk about benchmarks or speed. The author said the model has “great coding taste”.
They went on to explain. With other models they spend their time ironing the output. They cut half a page of pointless inline comments down to two lines. They strip out a big special-case branch that will never fire in real life. With this one, the first draft was about 99% fine. The design decisions were, in their words, boring and sensible. They also said they don’t care if it’s 10% dumber on benchmarks, because it saves them the cleanup. The bill for a few solid hours of use was ten cents.
I’ve been doing this job long enough to know exactly what they mean. I’ve reviewed a lot of pull requests. The worst ones are rarely wrong. They’re exhausting. Everything is technically correct and there’s three times as much of it as there should be.
Taste is a hard thing to put a number on. A benchmark can tell you whether the function returns the right answer. It can’t tell you whether you’d be happy to inherit it at 4pm on a Friday, or get paged about it at 2am. That second test is the one that matters in a real job, and nobody has built a leaderboard for it.
The comments split in a way I found interesting. The first reply, from several people, was basically: show us. Put the two outputs side by side. That’s fair, and it’s what I’d say too. “Taste” without an example is just vibes. I’d still believe the poster, though. Anyone who has done code review recognises the feeling, even if they can’t define it.
Then the argument turned to comments in code, and it got heated. One person said the walls of comments come from models being distilled on outputs from bigger commercial ones. Their theory was that the prose works as a sort of watermark. I have no idea if that’s true. It’s a neat story, and I’m suspicious of neat stories. Their fix was to keep documentation out of the code altogether and put it in separate domain docs. Another user called this take “absolutely insane”, and I mostly side with the second user. I’ve never worked anywhere where docs stayed in sync with code that lived in a different place. The comment next to the weird line is the one that gets read. It’s also the one that gets updated, because the person changing the line is looking right at it.
But I’ll give the first person this much. A comment that says what the code does is noise. A comment that says why is gold. The models seem to be very good at the first kind and weak at the second. That’s not a model problem so much as a problem with what the model can know. It can’t know that the odd retry exists because a vendor’s API fell over every Tuesday in 2022. Only a person who was there knows that.
The part I keep returning to is a quieter comment from someone who is on call for what they ship. They read every line that gets generated. Mass-producing code is only fun, they said, when you don’t get paged for it. Another said, a bit more bluntly, that anyone claiming AI codes better than they do must be a terrible programmer. I think that’s too harsh. I also think there’s something in it. If you can’t tell good code from plausible code, a tool that makes plausible code faster is a hazard, not a help.
I’m in two minds about all this. I use these tools, and some days they’re great. I’m fascinated by how fast the open models are catching up, and a ten cent afternoon is hard to argue with. I’m also uneasy about the power draw behind all those datacentre tokens, and about what happens to the junior developers who used to learn taste by writing the boring code themselves. Taste comes from having been burned. If the first draft always arrives clean, where do the burns come from? I don’t know.
What I do know is that the poster described the right thing to measure. Not how clever the model is, but how much of its output you have to undo. It’s the same as with any colleague. The best ones aren’t the smartest people in the room. They’re the ones whose work you can merge without holding your breath.
I’ll try Hy4-flash when the GGUFs settle down. If it really writes boring code, that’s the nicest thing I can say about it.