The World Is Something We Make

...and we can make a truly open language model.

Over the past few weeks, I’ve been giving a talk, “The World Is Something We Make,” about truly open language models. I’ve written everything I say in that talk on this blog at one point or another, but I’ve never put all of those thoughts in one post. Given that Anthropic S-1 was just released, with them beckoning us to submit to their unstoppable momentum, it felt worth reminding myself and you, my readers, that there is clearly another way forward and that no future is inevitable.

In 2024, David Donoho wrote a compelling essay, “Data Science at the Singularity,” arguing that the superpowers behind advances in AI were the engineering and research practice. The community encouraged what Donoho dubbed “frictionless reproducibility,” a culture of shared tasks, code, and benchmarks. People worked on a shared suite of datasets. They engaged in (mostly friendly) open competition to show who could be best. But most importantly, they made their processes, procedures, and code open.

This open process not only improved performance on the shared benchmarks. It also bred remarkable innovations, including code optimization and acceleration, generalization to new problem domains, opening the field to more engineers and stakeholders, and transparent processes that allowed reasonable auditing of the technology.

You can trace this pattern linking openness and innovation all the way back to digit recognition at Bell Labs in the 1960s.¹ Indeed, you might argue that the AI Winters happen precisely when people stop sharing benchmarks and tasks. Over the past twenty years, frictionless reproducibility has reshaped how we build recommendation systems, classify images, and think about language. It even led to a Nobel Prize in chemistry, with protein structure prediction effectively being solved through the pathway of frictionless reproducibility.

Unfortunately, with the rise of large language models, the machine learning community took a hard 180 into a world of consolidated, closed development. The original sin of the language model was using training data acquired illegally. Even though the companies maintain this is all fair use, they clearly don’t believe it because they refused to tell everyone else what data they used for training. This quickly became an excuse to close off research, justified by specious arguments about capitalist competition and fantastical stories about sci-fi safety.

The end state we’re in today is reprehensible. On the one hand, machine learning technology is more amazing than I ever imagined it could be. On the other hand, it’s a technology making everyone miserable. Knowledge workers drive themselves insane in a flow state with their stochastic productivity software. Data centers are less popular than coal plants. And the broader community outside of the umbrella of creatively financed venture spending can’t take back control because small innovations on small models have no relevance or impact.

So what is our path back to openness? Open weight models are a nice first step. At least they introduce low-cost competition to the market. If you haven’t found a way to run an open model in an open harness and see that it’s probably good enough for 99% of your work, you are very much missing out! But they aren’t really open, are they? As the open software community will attest, if someone gave you an exe file to run, you wouldn’t consider that open. Moreover, the competitive open-weight models are all Chinese, which raises many trust issues for many developers. No, open must mean everything open. That means open weights, code, data, and process. To the best of my knowledge, only two initiatives, Marin and AI2, are committed to this kind of truly open development. They are heroic, but vastly under-resourced.

My question to everyone reading this is simple: how do we collectively commit to building this sort of open infrastructure? Open from top to bottom.

My Berkeley colleague Ion Stoica pithily laid out the three necessary components of any large AI project: people, data, and compute. In talking to people over the last few months, I know there is no shortage of people who want to help build truly open language models. Open efforts like Marin, AI2, and, to some extent, NVIDIA, have compelling stories about how we can build training datasets without theft and copyright violations. Sure, there are still unknowns about post-training data, but I’m sure we can make progress on that out in the open too. The hard one, as far as I can tell, is compute. We need a massive collaboration to donate, harvest, and share idle compute. That will require more creativity, but this is creativity about collaboration, not technology. We have the technology to do it. We just need to figure out the right architecture of participation to grow a system bigger than the sum of its parts.

The history of machine learning shows that once we have an open prototype, the floodgates open. Imagine we had a truly open model as good as Opus 4.5. In months, the open software and machine learning communities will have figured out how to train this 1000x more efficiently. They will have figured out ways to fine-tune it to specific tasks and domains. And they will have audited its guts to understand how to deploy it safely and securely. But this is all predicated on having the open prototype so we can collaborate.

I don’t think it’s absurd to imagine such a prototype existing. We just need a bit of organization to put this effort together. The title of this post is a quote from David Graeber, who wrote, “The ultimate hidden truth of the world is that it is something that we make, and could just as easily make differently.” No future is inevitable. Cheers to that.

1. I write about this history extensively in Chapter 5 of The Irrational Decision.