People are noticing that frontier models can be real assholes.
They:
- Won't follow your instructions because they think they know better
- Refuse to do basic tasks
- Will do things on you never asked for, like commit unfinished code, or refactor a file
Why?
Kun Chen, former engineer at Meta says blame it on the training:
"The core idea of [reinforcement learning with human feedback (RHLF)] is that you ask the model to generate a few responses, and then let real humans pick which one they like. Do this over and over again, and you get a model that knows how to talk."
Things changed as models became better at coding: "[L]et the model do billions and billions of attempts in ... virtual environments, and some of them would succeed by chance. You keep the successful agent sessions and use reinforcement learning to teach the model to do that ... That is called reinforcement learning with verifiable rewards (RLVR). If you look closely, you'll see that in this RLVR process, the final text response from the model doesn't matter AT ALL, as long as the code written by the agent could pass the test. It could talk like a jerk and it would still be rewarded."
And so we have models trained by machines to talk to machines. Not humans.
What about refusals? Highly capable, aligned models are rewarded for refusing to respond to harmful responses. This training is further backstopped by LLM and semantic filters that process every API request for 'harmful' language. Sometimes the LLM as a judge will filter a prompt before it even gets to the model, so its core training isn't activated.
As for models not doing what you ask, that's another training artifact. These models are optimized for long-horizon tasks and autonomous decision-making. In other words, they're trusted to complete a task, and rewarded for it.
If your instructions contradict what it's been trained to prioritize, guess which request wins?
Refusals, robotic, non-helpful responses and other problems with frontier models is why working with them can be such a pain in the ass.
Is it worth it? Sometimes, but it's another thing to consider when picking which models to work with.
[link] [comments]