Open and Shut

What should be fair game and fair use for open corpus public intelligence?

I hope the intent of my call on Monday was clear: we should strive for something that any team can build from scratch as long as they have access to the training corpus, the software, and computing resources. We want something open to the public, as the models are a reflection of public intelligence and culture. As Colin Fraser remarked on Bluesky, “there’s no going back to the world before we knew that if you make a language model large enough it appears to become a little guy who sometimes solves open math problems and sometimes makes you insane.” These language models are built upon humanity’s collective, cultural intelligence, and I believe they should thus be open and accessible to everyone.

But what “open” exactly means is tricky. A laudable example of an open corpus model is Olmo from Ai2. Their model report details all of the data used. For pretraining, they use a mix of data from Wikipedia and Wikibooks, web pages extracted by Common Crawl, academic papers sourced from arXiv papers uploaded with LaTeX, code curated from GitHub repos with flexible licenses, and math webpages from FineMath 3. Earlier versions of the Dolma corpus also include Reddit threads, papers from Semantic Scholar, and public-domain books from Project Gutenberg. All of this data is available to everyone.

Now, here’s a question for the purists out there. FineMath 3 is generated using annotations from the Llama LLM. On the one hand, technically speaking, Llama is not an open corpus model. On the other hand, the FineMath dataset is free to download. I’d argue this still counts.

Even if you grant me that one, you’ll find a lot more use of LLMs in the data pipeline in the Olmo tech report. For instance, to generate some of the math training data in the later stages of training, the team uses Qwen. Qwen is not an open-corpus model. For some of the more complex reasoning, they use thinking traces from GPT4. That’s not an open model in any capacity. If you are using a closed-corpus model to generate training data for your open-corpus model, have we ended our game for full openness?

I’m not a purist, but the vague world of distillation is genuinely complicated. Distillation is not a cleanly defined action, but roughly describes the practice of collecting the text outputs from one language model to serve as the training set for another. What we’re allowed to use and not use is incredibly confusing. Even if we just go back to pure text, it’s hard to say what training data is actually legally acceptable under “the law.”

The law is confusing and unsettled. If you buy a physical book, scan it, run OCR, and add it to your training corpus, that counts as “fair use.” If you buy the exact same text as an ebook and add it to your training corpus, that’s violating the e-book licensing agreement. The fact that there is a distinction between these two actions is absurd and stupid. Stupidity is unavoidable in our complex legal code. It doesn’t get less stupid when you try to understand how distillation meshes with the terms of service for use of LLMs.

I’m thinking out loud and consequence free here on the newsletter. I’m happy to admit that it’s complicated. But we’re going to have to change or fight laws that prevent us from just outcomes. If we want to prevent a concentration of power at OpenAI or Anthropic, we need to think about what laws are just and right to prevent that.

Anthropic CEO Dario Amodei, in his typically blindered, insufferable way, chimed into the open models debate on Monday, arguing that the US should “ban industrial-scale distillation” but that he’s not calling for banning open-weight models. His letter is riddled with contradictions like this. I don’t care about his tortured reasoning because it’s still the case that the far more defensible position is banning closed-weight models.

Closed models are indefensible. Sure, we should give Alex Radford, Ilya Sutskever, Sam Altman, and Dario Amodei some credit because I would have never guessed that the models would be as useful as they are today. As Fraser said, there’s no going back to the time before we knew that. However, you don’t have to give them too much credit because you could also argue they should be in jail. If you think that is hyperbole, I’d like you to read the wikipedia page of Aaron Swartz. Or read this in-depth article in the Financial Times about the online libraries used to train our current language machines. The librarians are hunted across the globe by the FBI. The LLM entrepreneurs are gazillionaires and can pay billion-dollar settlements to cover their asses. We can have long, pedantic quibbles about what’s legal and what’s not. We can also ask, “What is right?” and “What is just?” The current situation where the best American models remain closed is deeply wrong.