
Hi there, argmin readers. We continue p(doom) week to celebrate the Anthropic IPO. Today’s post is about that.
Many have convinced themselves that predictions that “come true” signal some secret sorcery on the forecaster's part. But critics who correctly call out inevitable outcomes of power imbalances are called haters. We should not accept this dichotomy. Critiques give premonitory diagnoses instead of forecasts. Failing to heed their etiology makes predictions self-fulfilling.
Along those lines, I periodically revisit a reading list I put together in 2019 to introduce Artificial Intelligence to a wide audience. I made it both to give people who knew nothing about AI a popular introduction and to give new graduate students in AI and related fields counterpoints to the dominant narrative they had learned in classes and online forums as undergrads.
Ted Chiang writes how the AI god can be seen as a deflection from engineers’ unease about power consolidation and capitalism’s unquenchable thirst for growth. Maciej Cegłowski talks about Superintelligence, and every critique he raises of the online AI hivemind is still true. Stephanie Dick describes our history of redefining intelligence with respect to technology. And David Leslie calls out the AI Safety crowd for its scammy fear-mongering cast in the language of science. These were all written before GPT-2. There were no LLMs, and yet the stories were the same, and the critiques remain valid.
These readings are more about the people who make AI and their mythologies. The things that make them tick. You can read the stories in the New York Times and see they are telling themselves the same silly stories as they salivate their way to riches.1
On Tuesday, I wrote about how you should read a lot of the “forecasts” by AI weirdos as design documents. You can find predictions in Cegłowski’s piece that are just as accurate as what’s in AI 2027, even though Mr. Pinboard didn’t want this future to happen. To be fair, these aren’t even Cegłowski’s ideas. He just plainly describes the narrative in Nick Bostrom’s 2014 book Superintelligence. Indeed, in Chapter 6 of Superintelligence, Bostrom lays out what a rogue AI would need to do to break containment. It would need 6 capabilities:
Intelligence amplification
Strategizing
Social manipulation
Hacking
Technology research
Economic productivity
Bostrom even uses the term “Recursive Self-Improvement” to describe the mechanism by which these capabilities would emerge. These are the stories Anthropic sells us. They tell us that they are building a machine to check exactly these boxes. Reading Superintelligence shows how incredibly uncreative they are. As Cegłowski describes it: “If you look at AI believers in Silicon Valley, this is the quasi-sociopathic checklist they themselves seem to be working from.”
My reading list also puts the insider AI clairvoyant forecasters in perspective. AI 2027 is nothing more than Bostrom fan fiction with a decidedly mixed bag of concrete predictions sprinkled in. Those precise predictions are hard to pin down. Many had already happened when the document was written, and many have not come to pass.2
Yes, coding came to pass! Anthropic is in large part responsible for completely remaking software engineering. They might not have built superintelligence, but they have undeniably created Super Stack Overflow. That’s incredibly valuable! But automating coding was in Anthropic’s pitch in 2021. It’s been part of the plan from the get-go. None of the authors I listed above would have expected this to work so well. But if you look at what the people at OpenAI were doing in 2016, it’s clear they didn’t either.
Way back in, um, 2025, I wrote about my 2019 AI reading list:
“Despite the half-decade of impressive chatbot results, these articles hold up remarkably well. They explain the mindset of the AGI technopoly. The AGI god shares the values of the Silicon Valley founder, funder, and engineer. As Chiang and Cegłowski both point out, those values are of resource aggregation, constant optimization, economic productivity, and blind maximization of “reward.” This is Silicon Valley, and they think they are building a more powerful version of themselves.”
And here we are today. The Anthropic people are obscenely rich and dangerously powerful. They are rushing forward to build dirty data centers and wielding their lawyers to steer policy in their favor. They have become their own worst fear, an unaligned mechanical god. And they are about to be handsomely rewarded.
To some extent, the rewards are deserved. Claude Code and the other associated coding agents are amazing technology that makes a lot of computer things easier, better, and more fun. That’s an awesome achievement. But these superintelligence visions remain vile, and while you will be rich, you’re also incredibly unpopular. This doesn’t feel to me like a situation that ends well for any of us.
So congrats to my friends at Anthropic, who now have more money than they’ll ever know what to do with. Now grab some agency with that money. You can now quit and do something you think is not inevitable. Do you want a simple idea that requires zero creativity? Come help the rest of us build truly open AI models so we can all do the safety auditing and science in the public square. But whatever you do, try to imagine a new story. Write up some design documents for a world that everyone truly wants to live in.
This morning, Max Read read Jasmine Sun and had the same thought Ted Chiang had 10 years ago.
One sad standout prediction that didn’t come true in that document: the stock market was supposed to be up 30% this year, led by NVIDIA. J. Huang’s proclivities were not on their radar.
