Open and Shut
What should be fair game and fair use for open corpus public intelligence?
I hope the intent of my call on Monday was clear: we should strive for something that any team can build from scratch as long as they have access to the training corpus, the software, and computing resources. We want something open to the public, as the models are a reflection of public intelligence and culture. As Colin Fraser remarked on Bluesky, “there’s no going back to the world before we knew that if you make a language model large enough it appears to become a little guy who sometimes solves open math problems and sometimes makes you insane.” These language models are built upon humanity’s collective, cultural intelligence, and I believe they should thus be open and accessible to everyone.
But what “open” exactly means is tricky. A laudable example of an open corpus model is Olmo from Ai2. Their model report details all of the data used. For pretraining, they use a mix of data from Wikipedia and Wikibooks, web pages extracted by Common Crawl, academic papers sourced from arXiv papers uploaded with LaTeX, code curated from GitHub repos with flexible licenses, and math webpages from FineMath 3. Earlier versions of the Dolma corpus also include Reddit threads, papers from Semantic Scholar, and public-domain books from Project Gutenberg. All of this data is available to everyone.
Now, here’s a question for the purists out there. FineMath 3 is generated using annotations from the Llama LLM. On the one hand, technically speaking, Llama is not an open corpus model. On the other hand, the FineMath dataset is free to download. I’d argue this still counts.
Even if you grant me that one, you’ll find a lot more use of LLMs in the data pipeline in the Olmo tech report. For instance, to generate some of the math training data in the later stages of training, the team uses Qwen. Qwen is not an open-corpus model. For some of the more complex reasoning, they use thinking traces from GPT4. That’s not an open model in any capacity. If you are using a closed-corpus model to generate training data for your open-corpus model, have we ended our game for full openness?
I’m not a purist, but the vague world of distillation is genuinely complicated. Distillation is not a cleanly defined action, but roughly describes the practice of collecting the text outputs from one language model to serve as the training set for another. What we’re allowed to use and not use is incredibly confusing. Even if we just go back to pure text, it’s hard to say what training data is actually legally acceptable under “the law.”
The law is confusing and unsettled. If you buy a physical book, scan it, run OCR, and add it to your training corpus, that counts as “fair use.” If you buy the exact same text as an ebook and add it to your training corpus, that’s violating the e-book licensing agreement. The fact that there is a distinction between these two actions is absurd and stupid. Stupidity is unavoidable in our complex legal code. It doesn’t get less stupid when you try to understand how distillation meshes with the terms of service for use of LLMs.
I’m thinking out loud and consequence free here on the newsletter. I’m happy to admit that it’s complicated. But we’re going to have to change or fight laws that prevent us from just outcomes. If we want to prevent a concentration of power at OpenAI or Anthropic, we need to think about what laws are just and right to prevent that.
Anthropic CEO Dario Amodei, in his typically blindered, insufferable way, chimed into the open models debate on Monday, arguing that the US should “ban industrial-scale distillation” but that he’s not calling for banning open-weight models. His letter is riddled with contradictions like this. I don’t care about his tortured reasoning because it’s still the case that the far more defensible position is banning closed-weight models.
Closed models are indefensible. Sure, we should give Alex Radford, Ilya Sutskever, Sam Altman, and Dario Amodei some credit because I would have never guessed that the models would be as useful as they are today. As Fraser said, there’s no going back to the time before we knew that. However, you don’t have to give them too much credit because you could also argue they should be in jail. If you think that is hyperbole, I’d like you to read the wikipedia page of Aaron Swartz. Or read this in-depth article in the Financial Times about the online libraries used to train our current language machines. The librarians are hunted across the globe by the FBI. The LLM entrepreneurs are gazillionaires and can pay billion-dollar settlements to cover their asses. We can have long, pedantic quibbles about what’s legal and what’s not. We can also ask, “What is right?” and “What is just?” The current situation where the best American models remain closed is deeply wrong.
The book scanning as fair use is clearly madness. The whole spiritual point of fair use is that you shouldn't make a pain-in-the-ass fuss about people (PEOPLE) engaging with your work- that copying an excerpt for a student or making fun of it in a low-rent collage video shouldn't set the powers of the state delegated to the author by copyright on you. The notion that the largest piles of capital ever assembled by man routinely memorizing those texts and dispensing them for a fee in front of said authors are covered by those provisions is obscene- and I'm a pretty copyleft kind of person! I would, in fact, pirate that hard to find movie only available with seven hours of ads. But I am not a pile of hundreds of billions of dollars. Punch up, not down.
A hypothetical open corpus shouldn't include distillation of closed models, fruit of poison trees and all that, but the bookshredders arguing against distillation is cartoonish. If paying my fee entitles me to the output of a model (that likely represents, practically or morally, theft from authors), you're going to have the gall to suggest the *only* thing I can't use it for is making an LLM? Piss right off with that.
Generally I think the answer should be that an open corpus is unequivocally and enthusiastically open. Vigorous consent. If you can't make a decent question answering machine out of Wikipedia and Project Gutenberg, that's a technical problem to solve, and it's the one everyone needs to solve if they aren't satisfied with what current models can do, because that logarithmic data-to-performance curve means you've already lost.
"We can have long, pedantic quibbles about what’s legal and what’s not. We can also ask, 'What is right?' and 'What is just?'"
If only more people thought like this...
Once the labs have sucked up the entire internet and works of others, they proceed to pay countless dollars to companies like Mercor and Handshake that aggressively lure experts towards the wonderful gig of trying to automate the remainder of their knowledge and skills away. It's much more obviously perfectly legal but all just feels so dystopian and gross.
It's beyond angering when these companies just bulldoze through every legal or moral obstacle with their billions of dollars and then throw a fit about distillation attacks and constantly pretend to have the moral high ground on anything (how many open letters for slowing AI have these near trillion-dollar capex companies signed at this point again?)