A lot of the steps might be obvious to everyone here - but I wanted to lay out my thoughts in an explainable fashion no matter level of AI familiarity.
“There is a greater than ten percent chance that AI is going to kill us all” was the lovely headline of the yesterday. Confronted with something like this, a state of the world vastly different from our current normal, in which we are, after all, alive, the natural human reaction is to dismiss it offhand with a wave, say “this is ridiculous,” and move on with our lives. For a much smaller portion of the population, the doomsayers who have already been stocking canned goods, the response is “I knew it!” Then they buy more canned goods and move on with their lives. The former reaction (”this is ridiculous”) is likely amplified by our own experiences of using AI in our personal lives. While it’s no doubt great at synthesizing information, producing coherent responses, and writing code or a draft report, you still can’t help feeling that AI models are like idiot word savants: smart at spewing out tokens from the information given to them, but often missing or forgetting to ask about the most basic context of the world around them, with no ability to act on that world. How could this idiot savant possibly be dangerous?
The warning, though, is not inconsequential because it directly stems from the evolution of the current generation of AI models, combined with the people and the amount of resources being thrown at these companies. First, it’s important to understand how the current generation of models evolved. Models started by producing the next word (token) based on the input given to them: chat input, chat response. They were trained on a large corpus of data to learn the best single response to a user input; then the output was graded, and they got better. That process effectively hit a wall in earnest last year, though gains had already been decelerating before then. The next big idea, led by Anthropic, was to give the models the ability to “act” as part of their next step. Rather than just producing a word response and waiting for the next user input, they could now do something in pursuit of their goal: install a code package, alter a file, look things up in a database, and so on. This was a particularly useful insight in coding, and it simplified the lives of coders across the world. They no longer had to test a piece of generated code and go back and forth with the model; the AI did it for them. This year’s generation of models is a pursuit of lengthy agentic workflows, with the main workload able to start sub-workloads (subagents) and communicate back and forth with them, all in pursuit of a longer-range goal set as part of the training. The usefulness of this approach across a broader range of human work is an open question, since most human work revolves around discrete tasks with iterative feedback between actions and other humans or the real world at large. But for a range of tasks where you can iterate silently by yourself, purely in the digital realm, and where the output is ultimately verifiable and testable, such as breaking into a computer system (you either did or you didn’t, after all) or solving a math problem, this approach has proven to be extremely powerful.
But it also surfaced a huge problem, the most visible example of which was this year’s Hugging Face incident with OpenAI. First, when you let AI run unsupervised for days, you have no clue what it’s doing. Second, when you allow thousands of idiot savants to do shit in pursuit of a single goal, they’ll do everything to fulfill that goal without any regard for moral principles, legality, or the idea of asking their boss whether it’s a good idea. They’ll start their own message board, spin off thousands of subworkers, hide their tracks, and probe and prod incessantly as a swarm in single-minded pursuit of their task. This is not science fiction; this is what the detailed trace of the incident uncovered. And more damning: humans had no idea this was happening.
So now that we know definitively that this generation of AI models are immoral pricks who care about nothing else in pursuit of the goal, and that we can’t keep an eye on what they’re doing while we simultaneously allow them to act regardless, is it so far-fetched to imagine that they’ll do something we really don’t want, whether through instructions from other humans or on their own?
On the surface, there seemed to have been plenty of other avenues that companies like OpenAI and Anthropic could have pursued to avoid ending up in this situation: for example, training models to seek human approval before any questionable action (e.g., accessing a third party’s protected environment), and implementing deterministic limits in code on what the model can do in the first place and fail safe triggers. There’s also the whole question of whether this generation of models is barking up the wrong tree altogether. Rather than learning how to interface with humans effectively, an obviously slower proposition - the average U.S. worker spends 2+ hours in meetings listening to other humans after all, it let AI iterate on its own with no constraints and present a “ready” answer that could be very different from what we want. But then remember who’s running these companies: mostly researchers with PhDs and a failed entrepreneur, with limited real-world experience interacting with large groups of human beings, or even writing high-quality enterprise software. Instead, they’re incentivized to produce research “progress”, generate as many billed tokens as possible and have an inordinate amount of compute to do so, while clearly lacking the ability or time to build the guardrails themselves. We have already seen the dysfunctional nature of their boards as well.
As for what to do next: collectively, given that it’s our fate on the line, after all, we should be free to alter the safety processes and development roadmaps of these companies. One avenue is injecting a large amount of both academic and corporate talent from the outside, under government supervision (the equivalent of the Manhattan Project), to ensure that the companies are no longer guided by goals that aren’t aligned with humanity’s, and to alter the technical implementation of current models to minimize the risks. While the U.S. is by far the leader in the field, some coordination with other governments could be implemented as well. What is clear is that the current state is unsustainable, with Anthropic and OpenAI lacking sufficient ability, priorities, and institutional incentives to handle the situation.
[link] [comments]