Hi there, argmin readers! As the fall semester picks up, posting volume will, too. So I’m going to commit to writing short descriptive headers to help you sort through the different threads. Today’s post is a live blog of Lecture 3 of my graduate seminar “Forecasting: A Critical Retrospective.” A table of contents is here.

Since the Great Depression, US law has required financial management companies that offer products like mutual funds to add a disclaimer to all of their advertising:

The thing is, not a single person believes this. The fund managers don’t believe it, and neither do the customers. Why would you buy a mutual fund if you didn’t think its past performance told you something about how much you’ll have upon retirement? We tend to believe that some investments are riskier than others and some managers are more reputable than others. These beliefs are based upon past observations, and we use them to inform our investment decisions.

So what do we need to do to transform past observations into forecasts? We believe that the past can’t perfectly predict the future. We also believe that a forecaster is only as good as their track record. In today’s class, we’ll link these two together, showing how the evaluation metric for forecast track records leads us to particular forecasting algorithms.

Let’s start with the two main examples from Edmond Halley. We believe the past strongly predicts the future when talking about the motions of celestial bodies. We believe it far less when pricing individual insurance policies.

For his comet, Halley paired three observations together using insights about orbital shapes from Isaac Newton. Given the roughly 76-year gaps between these observations, he predicted we’d see the same object again 76 years later. In 1835, by the time we had seen Halley’s comet twice more, astronomers were uniformly convinced Halley was right and were certain we’d see the comet again in 1910 and 1986 (they were proven correct).

For his life table, Halley grouped people by age and used these cohorts to make demographic forecasts. The proportion of the population aged 25 was 1.668%, and the proportion aged 26 was 1.647%. Therefore, he concluded the odds a 25-year-old would live to see 26 were approximately 80 to 1 in favor.

The move in the demography example to consider odds and chance is interesting, and was something in the air at the time. Proto-demographer John Graunt had made similar calculations of hazard and risk in his tabulations thirty years earlier. Probability applied to casino games had only begun to be formalized forty years earlier. The transmutation of frequencies into risk was intuitive once you started assembling databases. Now we just take it for granted, having created a formal structure that hides the intuition.

In today’s lecture, we’ll work out some of the formalism of this map from rates to risks, deriving the mathematical formulas that encode our assumptions. If you assert that a forecaster will be evaluated on their track record, and if you believe that events are effectively the same, then you bind yourself to making future predictions a deterministic function of the observed rates. The assumptions here are usually implicit. We’re assuming a strong level of interchangeability between past and future events with a particular signature. And we tend to use metrics that beg the question: the common scoring rules always return probabilistic forecasts.

The evaluation ties your hands to making a particular form of forecast. Given a set of knowledge and a statistical score, you are forced to make a constant prediction for all future events. If you allow your predictions to be real-valued, they are suboptimal if they don’t obey the rules of probability. The score itself leads us into a probabilistic mindset. I’ve been calling this metrical determinism, and I find myself inserting some variant of this lecture in every class I teach.

Both the comet example and the life table example can be thought of as scoring track records on average. When you have highly predictable events, a perfect score is possible, but it takes a few hits to convince a skeptic that you really have nailed it down. When events are less predictable, you just want to make sure you’re not losing money on your annuity sales, and maximizing future profits again leads you into a particular form of forecasting.

What’s important here is we don’t have to assume some sort of generative model of randomness to buy into probabilistic prediction. Halley did not have to assume that god was playing dice with who lived and died. Instead, probabilities and odds were simply convenient tools for the actuary to price their products. Probability was the logical consequence of assuming past performance was indicative of future results.

Subscribe now

Hi there, argmin readers! As the fall semester picks up, posting volume will, too. So I’m going to commit to writing short descriptive headers to help you sort through the different threads. Today’s post is a live blog of Lecture 2 of my graduate seminar “Forecasting: A Critical Retrospective.” A table of contents is here.

I’m always looking for how we did science and engineering before the social structures and norms were built up to normalize practice. So for a class on forecasting, let me ask how experts made forecasts before they had proper scoring rules, Bayesian statistics, and Stata.

Though there are plenty of places to start the search, why don’t we today look to the dawn of rationalism in the Enlightenment? Patterns, Predictions, and Actions opens and closes with stories about Edmond Halley. Halley was a master of prediction. He had a keen sense of statistical approximation, knowing how to round and manipulate data to explain the past and predict the future. Halley’s publication record showed a man obsessed with a wide range of forecasting applications.

Halley is best known for his comet, a celestial body that returns to our skies once every 76 years. [footnote: It’s due to return in 2061, but many people’s AGI timelines suggest we won’t be around to see it.] Halley, a strong proponent of Isaac Newton who helped fund the publication of the Principia, wanted to find definitive evidence to prove Newton right.

His proof would come from observations of a comet he had charted in his backyard in 1682. Using Newton’s rules, Halley computed the orbital parameters of the body. He found these parameters neatly matched those of comets observed by Johannes Kepler in 1607. Moreover, they matched those of one seen by German astronomer Petrus Apianus in 1531. Since the gaps between these observations were around 76 years, Halley forecast another observation in 1758. He’d die before its return, but he was right.

This successful celestial prediction is heralded as a crowning achievement of Enlightenment Science. In 1850, Yale astronomer Denison Olmsted, who observed the comet’s return in 1835, wrote, “The Return of Halley’s Comet in exact conformity with the predictions of astronomers established the truth of all those principles by which those predictions were made.” Explaining data we’ve already seen is fine, but there is nothing more convincing to scientists than when theory predicts the future.

The funny thing about this, and a theme we’ll frequently return to this semester, is that this conclusion is completely illogical. It’s a lovely example of affirming the consequent, a fallacy you’ll learn in an introductory logic course.

P implies Q
Q is true
Therefore, P is true.

This syllogism is clearly invalid. (“All the Rationalists live in Berkeley. Ben lives in Berkeley. Therefore, Ben is a Rationalist.” How dare you!) But this is how a lot of science works! Theory predicts a particular outcome. That outcome is observed. This makes scientists feel more convinced their theory is right.

We could go into a long rigmarole about logical positivism at this point, but I don’t want to argue with Bayesian epistemologists today. I just want to point out that even at the inception of the Enlightenment, science was based on the illogic of accurately divining the future. This is one of the things we love about forecasts. When we accurately predict the future, we feel like our internal narrative is true.

Halley’s intuition about forecasting extended far beyond the heavens. He was also a key contributor to modern demography and actuarial science. Protestant pastor Caspar Neumann had collected records of lives, births, and deaths in his hometown of Wroclaw in Poland. Neumann was apparently interested in using this data to disprove the existence of climacterics, where deaths were associated with specific ages like 63. Halley, who came across this data after Leibniz presented it to the Royal Society, had other predictive interests. He churned through Neumann’s data and produced his foundational actuarial life table.

The numbers here represent the counts of people of any given age at a particular snapshot. There were approximately 1000 infants between 0 and 1 (a number, perhaps a bit too convenient), and a total of approximately 34000 individuals in Wroclaw.

In presenting his table to the Royal Society, Halley saw numerous uses for it. He first explained how his table could be used to calculate the number of men draftable into the army. He computed this by counting the number of people aged 18 to 56 and dividing by two.

More relevant to our class, he also pioneered probabilistic forecasting. His second claimed use of the table was calculating the odds that someone might die in a particular time interval. To do this, he counted frequencies and assumed rates of the past were indicative of chance in the future. There were 567 people aged 25, and 560 aged 26. Therefore, the odds a 25-year-old lives to see 26 were 560 to 7 or, simplifying fractions, 80 to 1. Similarly, if you wanted to know the odds that person might live ten years, Halley advocated taking the number alive by age 35, 490, and computing odds: 490 to 77, or approximately 6 to 1.

Halley also used his table to estimate how long people would live by finding the point in a table at which the odds of living dropped below 2 to 1. He called this “The age to which it is an even wager.” Even back in the day before probability, people equated forecasts with fair betting odds.

Not surprisingly, if you could equate mortality forecasts with betting, you could price insurance. This was Halley’s fourth proposed use of his table. Similarly, for a fifth application, Halley worked out more sophisticated calculations and determined a clever scheme to value annuities, a popular means for the crown to raise money. Halley’s calculations set different prices for different ages, based on bets on how long an annuitant might live.

The life table is only one example of Halley’s keen sense that you could make forecasts without physics. Indeed, he seemed to appreciate that the key to forecasting was simply linking past observations with future extrapolations. These extrapolations could be used to confirm physics and create a shared model of reality. They could also guide the pricing of financial instruments wagering on matters of life and death. For Halley, as for us in this class, predicting the future served multiple purposes.

Subscribe now

Hi there, argmin readers! As the fall semester picks up, posting volume will, too. So I’m going to commit to writing short descriptive headers to help you sort through the different threads.

Today’s post is by Jessica Dai, writing up her thoughts on the microconference on “public feedback for AI and beyond” that she ran at UC Berkeley in August. -Ben

Much of the conversation on A(G)I’s “risks and opportunities” centers on as-yet unrealized near- (or far-) future possibilities. Yet AI is already changing the world; its societal impact is already being felt in real time by real people. There’s a yawning gap between “AI elites” — frontier labs and academics and the online AI commentariat — and everyone else, in both beliefs and priorities. I think this is bad — medium- and long-term outcomes will also depend on how society experiences the near-term.1

My vision for dealing with this is something I’ve started calling “public feedback for AI.” The logic goes something like this:

Proposition 1. “The public” has interesting and important things to say about their experiences with AI, but are not typically listened to by decision makers.

Proposition 2. “Evaluations” — and aggregated information, more broadly — are useful, in the sense that they can influence consequential decisions.

Corollary. AI evaluations from public feedback can be a meaningful way to “do something” about the emergent misalignment between those who control AI development and literally everyone else.

I’ve spent the past few years working on this from various angles — trying to articulate why this might be a good idea, methods work for concrete audit instantiations, finding empirical evidence that user-generated content can tell us something useful about AI safety. But the more I think about these problems, the clearer the gaps in my own knowledge; there are many people who are smarter than me, who have worked on related problems for longer than I have, or who have new and different perspectives than those I’m exposed to in my local circles. And while there are a wide range of research questions that I find intellectually interesting, this clearly can’t remain a thought experiment.

Thus, microconference. (You can see a blurb, agenda, and list of attendees at publicfeedback.ai.)

I had a lot of fun at this workshop because while there were some fundamental philosophical questions threaded throughout the two days, everyone was ultimately interested in doing things beyond sending PDF preprints around to one another. And I’m really grateful for everyone’s engagement, because it helped me refine my thinking around each of the foundational ‘propositions’ underpinning the broader vision.

This write-up will be structured around the two ‘propositions’ I outlined above (rather than following the agenda directly). Additionally, I’ll stick closely to what was actually discussed at the workshop, so there is certainly lots of relevant literature that I’m not including here. Editorializing here is mostly mine, and if something is stupid or upsetting, blame me and not whichever speaker I’m referencing!

The first major question I wanted to cover is a core tension that ‘Proposition 1’, as written, glosses over: Who is the public?

While I won’t attempt to do armchair democratic theory here — though if there are any political philosophers reading this, I would love to chat — having clearer and better-motivated answers to this question seems important, because who is considered relevant also directly affects what concerns are substantively important, and also how their views can be understood.

Users?

I started us off on Monday by sharing a few plots from my paper analyzing r/ChatGPT, where we argue that anonymous Reddit data can help us understand broader societal impact; the claim is that the population of “r/ChatGPT posters” can help us simultaneously access a true “population of ChatGPT users” and something about the topics they cared about. I think this was broadly true for the time period we analyzed, but “ChatGPT users” is a very particular choice for defining a “public.”

Nevertheless, if we think AI impacts are largely realized through their users, then usage data is critical to measuring that impact. To this point, the first half of Shayne Longpre’s talk covered the newly launched AI Observatory project, a heroic effort to pull together as many real transcripts as one might reasonably find available on the internet, and to release and analyze them together as a corpus of usage data. Still, there are fundamental challenges with trying to reconstruct proprietary data from the outside; I’m cautiously optimistic about Anthropic’s (very new! Hot off the presses!) data release program.

Study participants or poll respondents?

A common adjective I’ve seen people use to describe chatlogs and other usage data is naturalistic: it reflects the real context in which users were engaging with the AI product. But this data has limitations; it is purely observational, which makes representativeness concerns all the more salient, and cannot provide context beyond the interaction itself (e.g., motivation or outcomes of usage). Instead, the typical approach to “human data collection” in academic contexts is to run more structured studies: recruitstudy participants and ask them to answer survey questions or complete tasks. While this allows researchers the ability to study more detailed questions, these studies have their own limitations; Serina Chang’s talk traced this tradeoff explicitly, with naturalism and control substituting directly for one another.

One issue that came up in discussion was about who these study participants were. Why should we expect that the data they generate is in any way “representative” of the constructs that researchers are trying to measure? A typical academic study promises a pay rate of up to $10/hour (the current standard recruitment platform is Prolific); this is below minimum wage in 30 US states. While $10 USD can be significant in a global context, we were unsure how to reason about the impact of this pay rate on the people who choose to participate.

This is a fundamental problem shared by public opinion polling (which is one reason why Ben hates it), and while it’s probably best not to read into more than one sigfig of poll results, there is reason to believe polls can be useful for coarse-grained judgments of vibe — especially when the vibes are off. Emma Pierson opened her talk with poll results showing the divergence between AI experts/ Claude users, who are generally positive about AI, and a demographically representative sample of the American population, who are generally negative. Jasmine Sun’s datacenter reporting was also motivated in part by national polls showing new datacenter construction polling on par with, or worse than, nuclear and coal. This stood in stark contrast to the SF tendency to dismiss datacenter opposition as merely a problem of misinformation.

Anyone who wants to report?

The approaches described above begin by recruiting people, then gathering their feedback within the explicit context and scope of a study (or poll). An alternative is an inversion of that paradigm: waiting for people to organically share feedback of their own volition, so that ‘the public’ is itself defined by the act of submitting a report. The second half of Shayne’s talk covered a flaw- reporting workflow that lets anyone create a report for an AI-related flaw, vulnerability, or incident, broadly construed, then sends it off to relevant parties. In terms of reporting accessibility, this system rhymes with a public incident reporting portal administered by the California Office of Emergency Services. As Deb Raji explained, this was created as part of SB53’s implementation. While this legislation was mainly directed at companies to self-report severe incidents (more on that later), there is also a provision for enabling public reporting.

Neither of these portals have shared the submissions received thus far, but they have similar design decisions that I suspect also lead to similar challenges (Algorithmic Justice League also has a harm reporting portal, and also hasn’t shared findings, as far as I can tell). Since literally anyone with the link can submit reports, these portals are likely susceptible to ‘low-quality’ submissions or spam. On the other hand, I would guess that overall submission volume is low, since awareness that these platforms exist is also low. I didn’t know about the SB53 portal until Deb’s talk, even though this is kind of my whole thing right now!

Start with an explicitly scoped community in mind, and go talk to them directly.

All three notions of ‘public’ described above involve a very convenient slippage between “public feedback” and “data collection”: they require configuring “the public” as a generic, anonymous mass from which data points can be sampled, and for which summary statistics can be computed. I don’t think this is always a bad thing, but it was striking to me that some of the most compelling case studies for effectively capturing “public feedback” started with an explicitly scoped community, and gathered information by having conversations with individuals within that community.

HCI researcher Samantha Dalal spoke about her work with the Workers’ Algorithm Observatory, which builds tools with gig workers to help identify, measure, and contest algorithmic systems. They bootstrapped worker outreach by collaborating with unions — existing legal entities with corresponding social networks. However, it was still necessary to have face-to-face conversations with workers to build trust and understand their needs.

Humphrey Obuobi shared his work with BLOOM, a nonprofit that is actively facilitating community dialogues in three counties in central Oregon. In this case, while BLOOM develops its own deliberative platform, I see the substance of their work as downstream of the specific, situated collaborations with local civic organizations which — like the unions in Samantha’s work — naturally scope participants to a more specific community.

Finally, Jasmine’s datacenter reporting was focused on understanding the popular backlash; this meant traveling to locations where backlash was visible or brewing. Of course, she talked to local elected officials, who were broadly responsible for decisions at the level of contracts and permitting, as well as activists who were leading more organized opposition. But she also spent a lot of time hanging out at county fairs and beer gardens, talking to people for whom datacenters may not have been the absolutely most salient issue in their lives, but who, nevertheless, turned out to have opinions when asked.

But wait, what about “unknown unknowns”?

I’ve often claimed that the major practical benefit of “public feedback” is the ability to support harm discovery — i.e., to surface unknown unknowns that a centralized decisionmaker (policymaker, company, etc.) might otherwise never think to look for. This seems, to some extent, at odds with the provocation above: doesn’t starting with a specific community in mind inevitably constrain the content of “public feedback” to known unknowns?

One reasonable response is that unknown unknowns exist even within well-scoped communities; prespecifying what groups of people one might be interested in doesn’t necessarily also require committing to particular outcomes or topics, especially if individuals are given the space to express their perspectives in their own voices (e.g., via semi-structured interviews instead of, or in addition to, providing yes/no answers or numerical ratings).

On the other hand, bottom-up community formation can and does already happen. Social media platforms are a natural and well-studied example: networks of like-minded people find each other through a mixture of organic encounters, algorithmic nudges, and moderation choices. For instance, after the initial release of GPT-5, a decentralized campaign to “bring back 4o” emerged on social media (mostly Reddit and Twitter); The Human Line is a research and advocacy group for survivors of AI spirals that emerged from initial encounters on Reddit.

Therefore, if working within communities (rather than a generic abstraction of “users” or “recruited participants”) can be uniquely effective, then perhaps we should try to build systems that explicitly help guide the development of communities based on common concerns. One of the projects covered in Wesley Deng’s talk was an end-user auditing tool called WeAudit, which intentionally included a deliberative and semi-social element that helped individuals contextualize their observations with other user-auditors. A similar motivation is behind an early-stage study shared by Eve Fleisig, where participants see other people’s experiences and can explicitly compare them to their own. In the universe of these projects, the community itself might emerge as an unknown unknown.

The other thing about “community” is that it’s likely of some intrinsic value to a participant — a very different source of motivation from the $10/h paid by Prolific. I’ll write more on the question of “what someone might get out of participating” in the next post.

Subscribe now

I more or less said what I was going to do in my tongue-in-cheek, cryptic post on Tuesday, but let me dive into the full details of how I’m planning on structuring this grad seminar. I’m going to log the course content here, which now lists a rough schedule for the class sessions. Taking inspiration from Matt Jones and Chris Wiggins, I’m going to do a weekly split between culture and engineering. Focusing on a particular application domain each week, we’ll spend one session discussing the purpose of forecasts in that domain and the other on the methods.

I arranged things into a thematic arc, starting with the weather—forecasting’s biggest success story—then moving to shakier ground in seismology and epidemiology, and ending with, well, millenarianism. As we move across timescales, our ability to predict nature dissipates: we can make accurate ten-day forecasts, but predicting large-scale climate disruptions is far more qualitative. We can predict immediate earthquake impacts at a distance once an earthquake has happened, but we can’t nail down precisely when a big one will happen. What we do with precise, short-term forecasts is completely different from what we do with imprecise long-term forecasts. Short-term forecasts dictate actions; long-term forecasts of discrete shocks inform risk management, preparedness, and rapid-response policies. I’m hoping that by the end of the semester I can better articulate what long-term forecasts of the end of the world do.

We’ll look at forecasts in governance and how they influence and shape policy. I’m particularly interested in discussing the rise of cost-benefit analysis in US governance. Cost-benefit analysis puts a specific number on something completely unknowable, but is now mandatory for any bill or law to pass. I want to trace how we became so reliant on a particular set of methods for guessing costs and benefits. I’m also interested in how economists became convinced you could forecast “the economy.” This required inventing something called “the economy” that could be forecast in the first place. How our system of government got so tied to a particular style of economic prediction will occupy several weeks of the class.

We’ll also get into why we want to predict “the public.” I want to examine how opinion polls went from a question of legibility to one of prediction. How did we get obsessed with using surveys to predict outcomes like elections? In parallel, we’ll look at the history of attempts to simulate the public. Simulation is nice because you don’t have to talk to people, right? We’ll look at the many misses over the history of human simulation in policy scenarios, and dig into the current obsessions with using LLMs to predict what people might do.

Finally, we’ll get into the weird culture of competitive forecasting. We’ll engage with ideas from superforecasting and prediction markets and ask why people think these are useful information-processing systems. We’ll talk about punditry and how it’s not always interested in minimizing a Brier Score. We have to talk about the relationship between forecasts and gambling. And we’ll try to piece together why putting odds on outcomes makes people feel better about the future.

For the methods, I did my topic matching so you could extract a logically ordered half-semester course on forecasting from those lectures alone. This is not a class on how to be a rational forecaster. I want to problematize those methods more than tell you how to implement them. You can ask your friend Claude if you need an honest, load-bearing implementation.

Our methods survey starts with a refresher on my idiosyncratic views of machine learning as optimization-driven algorithmic pattern recognition. This will lead to a lot of discussion of the optimization problems themselves and why people like them. We’ll cover scoring rules, calibration, maximum likelihood, and utility maximization. We will discuss the role of models and look at probabilistic recurrence models, dynamical system models, differential equations, and other simulation-based tools. We’ll spend time on uncertainty quantification and how people come up with error bars (part of being a good forecaster is plausible deniability). Then we’ll look at offline and online optimization methods that let you fill in predictions based on your cost functions and modeling assumptions. I’m interested in highlighting the metrical determinism. The cost functions and models more or less tie your hands algorithmically, and most of the cleverness goes into how you evaluate.

Hopefully this arc will feel coherent as we go. I’m not into predictions, so don’t get mad if the story changes as I go. I’ll blog through it, and then we can reflect on where we land at the end of the semester.

Enrolled students (and those dedicated to following along at home) have an important first assignment: pick something to forecast. I don’t care what it is. Throughout the course, the goal is to learn the practical techniques by making predictions. Every week I’ll ask you to try to apply the tools to your problem. Or at least find how other people have applied those same tools to your problem. At the end of the semester, we’ll present our full findings and see how accurate we can be.

Subscribe now

Update: I haven’t joined Anthropic or taken leave from the university.

That means I’ll be teaching as usual this fall, and you can look forward to your regular installment of live lecture blogs. Unlike past years, I’m assigned to my grad seminar in the fall, not the spring. Next semester I’ll be teaching our undergraduate probability class. That is assuming that the University is still here in the spring and hasn’t been put out of business by some new AI super tutor released by my friends across the bay. I mean, it would be the year 2027, and popular forecasts suggest AIs will be able to do everything taught in a CS degree by late 2026. By the end of the spring semester, those same prognosticators predict those AIs will go rogue, and we’ll find ourselves in the reality forecast by James Cameron in his 1984 prophecy, Terminator. Why should I bother dusting off my copy of Bertsekas and Tsitsiklis when the forecasts tell me I should work harder at the gym to prepare myself for robot enslavement in the salt mines?

Actually, you know what would be a good way to prep for the robot apocalypse? Why don’t we spend a semester talking about forecasting and why people are obsessed with being certain about the future?

Answering that question could potentially be a great way to shape a graduate course. We could spend a semester digging into not only how people forecast, but why they forecast. We could split our time in half, looking at the particularities of different domains where people make forecasts, and then looking into the tools they have settled on as mathematical culture.

If you look at the places where forecasts are common, they all have different purposes. A local weather report is very different from a prediction of the end of the world. The former tells you if you should pack an umbrella on the way to work. The latter tells you whether you need to lobby your government to radically change its planned energy buildout. We all believe the evidence supporting forecasts of rain is far more certain and reliable than forecasts of nature’s end. But the costs of being wrong couldn’t be more different.

What impact do these forecasts have? Why do we forecast the weather, and what hidden technology is needed to make these forecasts accurate? Why does Congress demand that we forecast future budgetary consequences of proposed laws, even though we know we can’t predict the actual structural shocks that will render those forecasts moot? Why are people so obsessed with predicting the rise of superintelligent robots? Is it more than a way to justify their greed and obsessive 996 work conditions?

We’d learn a lot from a comparative study. I’d like to look at astronomy, meteorology, climate science, seismology, macroeconomics, government, epidemiology, public opinion research, and millenarianism to see what they have in common and how they differ. Forecasts let people externalize their beliefs about the likelihoods and consequences of various scenarios. Some forecasts are made for mundane planning. Some forecasts are made to literally gamble. Some forecasts communicate possible futures that others might not be considering. Some forecasts are made to be self-fulfilling, to manifest a change in the world the forecaster desires. Some are made to be self-negating, to convince people to act to avoid worst-case scenarios. I’m interested in understanding the threads that link all of these different purposes together.

Though I’m much more interested in the why, the how has some fun tidbits too. The how is about creating certainty about the future by quantifying it. Uncertainty becomes certain once we turn it into an interval, right? In talking about the how-of-forecasting, I could tie together a lot of loose ends in methods that I’ve blogged about over the years.

We might determine the conditions under which pattern recognition (aka machine learning aka AI) becomes a forecast. We could look at how forecasts are evaluated post-hoc with scoring rules and calibration tests and why people think those are good evaluations. We could look at methods for cost-benefit analysis and uncertainty quantification, and how people justify their modeling assumptions to make decisions. We could learn about tools from dynamical systems that move from simple moving averages to complex simulations. We could examine how statistical tools can be applied to extrapolate from the present to the future. And we could see how these sorts of metrics and models tie your hands algorithmically into unsurprising answers.

This sounds like a fun class to me. I predict I’ll teach this class this fall and live blog it here, starting this Thursday. If you’re a Berkeley graduate student whose research depends on forecasts, email me if you’d like to join the course.[footnote: If you do email, please send me a description of your background and why you’re interested.] If you’re not local, I’ll post a syllabus and webpage this week, and I’ll do my best to keep all of the material public. I predict it will be fun.

Subscribe now

Thanks to everyone for the constructive feedback on Monday’s microconferences post. I wanted to take a beat to engage with two salient themes in the replies: runbooks and homophily. I’ll start with runbooks today, and tackle homophily next week.

No two microconferences are alike, and we shouldn’t impose hard-and-fast rules on their structure. One of the fun things about small conferences is you can tailor them to particular goals and dreams. And there are so many ways to do this well.

However, I think it will be useful to compile a runbook of agreements, strategies, and rules that you can use to help modularly assemble your ideal microconference. Today, I’ll run down a bunch of disorganized examples. You tell me your favorite ideas in the comments. I’ll assemble these more coherently into a document that I’ll widely share.

We can learn a lot from existing institutions, and many people spoke fondly of places I should have shouted out in the first post, e.g., BIRS, Oberwolfach, Dagstuhl.1 I also adore the quirkiness of the American Institute of Mathematics, which has a very particular and very fun way to run a microconference, disallowing canned and prepared talks.2 I am inspired by unconferences that bring together dozens of people to spontaneously generate many microconferences. All of their best practices should be part of the runbook.

I’ve found an easy model for microconferences is a bundle of short, 5-10 minute talks with adjoined group discussions led by the speakers. The past four microconferences I’ve attended have run this way. Short talks are fun because they force people to think hard about messaging and concision, and lead to a lot of interesting back-and-forth with the right serendipitous assignments. I was somewhat randomly assigned to a panel with Dan Wang and Dan Davies on cybernetics, and it was probably the most fun and rewarding 90 minutes I’ve ever had at a conference. That single brief session reshaped the narrative arc of The Irrational Decision.

Johan Ugander raised several good ideas in his comment. About archiving and proceedings, he wrote: “The non-proceedings nature of workshops is key to drawing in a diverse set of people.” I agree. We should think broadly about what should count as “proceedings” or “archiving.” I liked Johan’s suggestion of simply inviting participants to submit to a special issue. You could consider this series of blogs ([1], [2], [3], [4]) and this youtube video the “proceedings” of the Cultural AI workshop organized by Leif Weatherby and Tyler Shoemaker this spring. I just want to suggest that we explore creative ways to archive, evaluate, and credit microconferences in the broader academic ecosystem. I’d add to my list of core microconference values that “archiving should encourage, not discourage dissemination and broader engagement.”

As Johan wrote:

“ACM EC has a forward-to-journal mechanism, which gets Computer Science, Operations Research, and Economics folks together for a coherent conference but lets them still go harvest their respective tokens. The EC reviews get passed to the journal. And EC also organizes a ‘Highlights beyond EC’ session, which lets people request to present work recently/already published “elsewhere” but relevant to the community. Both of these mechanisms help keep the ‘conference’ and ‘publication’ goals separate.”

Endorsed!

Anna Gilbert raised the idea of “bump sessions”:

“The Dagstuhl and Oberwolfach type conferences in TCS used to also have “bump sessions” (maybe they were called rump!) where people proposed open problems, noodled over difficulties, etc. Sometimes these micro workshops even “published” open problems from these bump sessions. They were quite useful and engaging!”

Anna also raised one of my favorite ideas, which I’m looking for an opportunity to try:

“Another model I’d like to advocate for is to have members of a PC present the papers they selected (see my rant about paper reviewing and big conferences) and then an audience discussion. Like what the statisticians do but in person and with a publication resulting for the authors.”

I call this the “Not-so-royal Society.” The conference or session would center on a single paper written before the conference is organized. The organizers invite discussants to compose a response/review of this paper. The meeting could start with a presentation of the paper by the author. Many papers have multiple authors, and the presentation can be as collaborative as the writing. The author’s presentation is followed by commentary from the discussants. The remainder of the session is a conversation that invites commentary from the rest of the workshop participants.

After the workshop, the author can revise the paper, the discussants can write formal commentaries, and the author can write a rejoinder if they choose. All of this writing could be posted to arXiv as refereed conference proceedings and linked from the microconference proceedings webpage. This format might be the most legible in the current regime of bean counting and could perhaps serve as a “gentle introduction” to the microconference format. If you have ideas of papers we should use as testbeds for this format, reach out! I’d love to help set something like this up.

Finally, I want to give a shout out to Henry Farrell and Cosma Shalizi, who have been experimenting with clever microconference formats for years. Cosma wrote up the idea of having people present others’ work. Everyone writes a talk or slides, but then the presentation is assigned to someone else. They found that doing this at the start of the workshop creates a shared comprehension, allowing for a lot of multidisciplinary crosstalk. And they also found that speakers had to work extra hard to make their points comprehensible.

This is by no means an exhaustive list of ideas. And it shouldn’t be. My next major to-do is posting a working document of this runbook somewhere. But before I jump in and commit myself to a design, I’m looking for pointers to good online institutional memories that allow for edits and revisions. I love the minimalism of PMLR and bactra.org, but I also want to host a living mission statement and runbook on the same site. Let me know what platforms I should look into. Tech tips would be most appreciated!

Subscribe now

Part of the reason I brought up microconferences on Monday is that I was attending a great one this week on public feedback for AI, organized by Jessica Dai here at Berkeley. In the spirit of creating an archival footprint, Jess and I will have a lot to say about the workshop over the next few days. I’ll kick things off by describing what I talked about.

I opened with a provocation about the trap that policy-minded computer scientists and social scientists so easily fall into. Longtime argmin readers will recognize the pattern. If we want to raise the concerns of a public, we need to convincingly present “evidence” supporting those concerns to policymakers. “Evidence” means cold, hard quantifiable facts that are easy to explain to policymakers, not just anecdotes of harm. I’m sympathetic. If you want to advance an agenda you care about in a complex world, you have to make it simple for those in power to understand. Too many things are happening at once, there’s far too much nuance and ambiguity, policymakers can only keep so many in their heads, and only so many laws can be drafted and passed at any given time.

It makes sense then that people concerned with technocratic solutions spend so much of their time designing architectures of legibility. They focus on the right ways to summarize data, weigh competing interests, and write compelling reports. They build computational frameworks to compile complicated, singular events into useful statistical summaries. A nice chart is worth a thousand testimonies.

These architectures of legibility are what enable the inevitable quantification trap. To make things legible to decision makers, we quantify them. Since we agreed on transparent procedures, the quantified must be objective. Numbers are always objective, right? Objectivity buys analyses authority. And expert authority then becomes a tool of power.

The quantification trap has been a pervasive and mimetic signature of the information age. And it has become progressively invisible as computation has miniaturized and sublimated into ubiquity. Every moment of our lives is now surveilled and quantified, ready to be summarized into new systems of control.

I am not against quantification of social systems. I just want to consistently raise awareness of its hegemony. There are clear benefits to quantification. It buys us a level of intersubjectivity, as anyone can trace the path from evidence to summary statistic. This shared, standardized language lets us collaboratively govern complex societies.

On the other hand, the quantification trap removes discretion, forcing us to abide by rigid rules. Quantification erases individuals in its bucketing and summarization. And quantification enables structural violence, forcing citizens to constantly make themselves legible to those with power to avoid being punished.

The question I always get from technocrats after presenting these critiques of quantification and architectures of legibility is “What else could you do?” My answer is to think about an alternative type of social architecture, architectures of participation. Tim O’Reilly coined this term in the early aughts to describe what makes participatory culture work on the internet. Why do some software systems take off as collaborative efforts? Tim noted that many software and communication systems are designed for contribution. Open source software has countless success stories. We also have the legacy of internet communication, message boards, and the World Wide Web. We have the astounding body of knowledge that is Wikipedia. And we have a powerful public challenge to copyright that was Napster. This last example highlights that not every architecture of participation brings unambiguous good to every stakeholder. The lens of participation helps us think about what elevates voices and values directly through individual actions.

Architecture of participation subsumes many different perspectives on the design of social systems. It includes standard economic mechanism design, the rules of games we play, and the decision-making agreements established by anarchist groups. What distinguishes these systems from architectures of legibility is they aim to cultivate broad expertise from a broad group of people. They are ugly and organic by nature. They are structured agreements for interaction and discussion. They do not suppose a benevolent set of experts at the top will make decisions based on what they see. Mods often write explicit rules, but community patterns are encouraged to be emergent and reflexive. They let the community work together, in small spaces and large ones.

Architectures of participation focus on designing flexible agreements for flexible ends. They accept that there is no clear metric to maximize. Sure, you can build legibility dashboards to make sure your system isn’t crashing. Again, I’m not against quantification. However, the focus of participatory design is not quantification, but broadening engagement and diversifying served ends.

So what does this have to do with AI and public feedback? It is very weird that our contemporary AI took all of human knowledge, made something very interesting and very powerful, and then gave all of the rewards to a small group of people who whine all the time about how they have no power. This is a perversion of participation. Built on the labor and love of individuals, generative AI technology concentrated power. And then those in power convinced themselves they are powerless. San Francisco wants to abdicate all human agency and reduce us to making ourselves legible to an artificial bureaucratic god.

On the other hand, the data center protests are inspirational. Here you have a lot of people who are really upset for a complicated set of reasons about AI. It’s hard to say why they are so mad, and why this issue is so salient, but it’s undeniable that they are driving policy conversations. Their protests and organization have made them not legible, but unignorable.

AI doesn’t have to be exclusionary. We could build an AI that’s a public good. One that invites participation. One that involves known training data, participatory training data. We now know that if you train next-token predictors on collective intelligence, you end up with a quirky piece of software that speaks in natural language, solves impossible math problems, and does all of your coding. We can’t unsee that. But we don’t have to let a small group of whiny, weird people own it. We can use this insight to build a public infrastructure for collective, participatory intelligence.

Subscribe now

I ended my last post with a call for action. I stated, without evidence, that if we’re unhappy with the status quo, we can choose to do something else. I’m hoping this semester I can instead lead by doing (sorry that’s corny) and be a weird freak out in public on this blog. Let me start today by sharing something I’ve been working on: a plan to create alternative systems for generating, evaluating, and sharing knowledge.

While academia has exploded in size in the post-war period, the ideal size of a conference has remained constant. Small workshops with one or two dozen participants have consistently launched big ideas, forged multidisciplinary connections, and set new standards across fields. Look at the history of artificial intelligence (a marketing term I continue to loathe with every ounce of my being but one that is unfortunately useful given its current cultural dominance). The field was conceived at a small conference held at Dartmouth. Societal implications and multidisciplinary connections were hashed out at the Macy Conferences. The ICML conference began as a small workshop of AI researchers at CMU, a splinter group that reclaimed and rebranded the term machine learning. NeurIPS began as small workshops organized at Caltech, called “Hopfest,” where John Hopfield brought together neuroscientists, physicists, and computer engineers to rethink information processing. These workshops each had 10 to 30 people, were cross-disciplinary, aimed at bridging disciplinary boundaries, and focused on brewing creative, new ideas.

Machine learning conferences like ICML and NeurIPS now draw tens of thousands of participants. Somewhere along the way, computer scientists convinced themselves that conference proceedings should be valued as highly as journal publications in other disciplines. A combination of industrial interest and metric-maximizing left AI conferences as an undignified mix of job fairs and “vetting” of “publications” for CV padding. The evaluation is indistinguishable from coin tossing, and no particular result ever stands out. It’s more likely that a mishap at an industry party will get attention than a paper will change the field. This situation is unfortunate and untenable.

Of course, the small workshops—mocking scale, let’s call them microconferences—live on. A professor with a little initiative can put together an ad hoc meeting. There are non-profit institutions dedicated to hosting these sorts of events, including mathematics research institutes like the Institute for Pure and Applied Mathematics at UCLA, computing research institutes like the Simons Institute at UC Berkeley, and institutes that grew to define an entire field like the Santa Fe Institute. However, microconferences are considered luxury goods, something everyone wants to do all the time, but is only allowed to do after their paper writing, teaching, reviewing, and committee work. For some reason, in the modern calculus of the curriculum vitae, a refereed conference paper is more valuable than inspiring a new research direction at a small meeting. This is absurd.

The funniest part about the quantification trap is how we forget that the quantities were ad hoc and artificial in the first place. What makes a Science paper count as prestigious is nothing more than the salivation of a bunch of annoying academics who want Science papers for themselves. If enough of us just want “well-curated microconference proceedings,” then we can value these just as highly.

How could we go about increasing the value and worth of these meetings? I don’t propose constructing an elaborate counting mechanism to have microconferences register as yet another notch on a publication list. And I know we can’t break down our current system of bureaucratic accounting overnight. Deans will still rely on stodgy metrics, scholars will need to produce legible artifacts to appease said deans, and historical institutional prestige will still signal more than it should. Nonetheless, I’d like to offer an initial path for fields to return to the days when small-scale but intense evaluations were valued more than large-scale but cursory selection processes.

Valuing the outputs of small workshops wouldn’t be a departure from our academic traditions. Most of our publication system traces back to reading papers before the Royal Society of London. A paper was only considered published after it had been read aloud and debated in person. Oftentimes, people read other people’s papers and debated these. Lots of our modern systems try to carry on this legacy of debate. In statistics, a main contributed paper is accompanied by “discussions” by other authors and a rejoinder by the contributed paper’s authors. In economics, the working paper serves as a way to gather feedback before official publication, which can take years. Many fields use “special issues” to compile talks from their conferences into proceedings. These are all great ideas! We should draw from the best of these traditions.

One of the hardest things to avoid is making hard-and-fast rules. Mission statements are always forgotten in giant bureaucratic systems, leaving behind a fossilized set of rules and conventions that people assume have been there since time immemorial. These systems then invite the same old gaming and goal displacement. Fully aware that this is the case, let me first state some values I’ve compiled in discussions with friends and colleagues. I’ll offer a few guidelines after.

1. What: Microconferences are where the best ideas are debated. Some things do not and cannot scale.

2. Who: Microconferences should bring together people who didn’t know each other before. You can invite friends, but you can’t only invite your friends. Sometimes you need to get different people in the room and see what happens.

3. When: The turnaround time from conception to microconference should be as short as possible.

4. Organization: While a parent institution for these microconferences can streamline logistics for microconference organizers, it should not require a permanent home or dedicated staff. Its leadership should be minimal and rotate over time. It should be easy to organize a microconference.

5. Output: Not all outputs of a microconference need to be a publication, but sharing findings and insights with others is part of what academics do. While not everything needs to be written down, and ephemeral conversation is important, participants should err on the side of more documentation.

For this final bullet, I envision a documentation structure similar to what organizations like the aforementioned IPAM and Simons Institute currently maintain: a webpage. The Proceedings of Machine Learning Research (PMLR) provides an ideal model. Inspired by the minimal elegance of the Journal for Machine Learning Research, Neil Lawrence cooked up a GitHub repo to compile and curate conference proceedings. It does exactly what is needed and nothing more. The main page will cleanly list past events, and you’ll be able to search by title or participant. Individual microconference webpages could include a list of participants, a photo of the gathering, and a documentary record that links to slides, videos, or papers. What is archived and how it is archived would be up to the organizers.

In the style of arXiv overlay journals, these pages could signify the “blessing” of a particular version of arXiv papers by the participants, demarcating that a paper has been legitimately peer-reviewed. People might dismiss such peer review as cronyism, but if you see the names of participants, you can judge for yourself what credence you want to give to the review.

I am not claiming that microconferences solve all problems. One glaring example is the widespread belief that computer science graduate students need papers. What is a PhD if not a bunch of papers stapled together? Yikes. The obsession with using conferences as a proxy for grad student education is a mistake we are paying for dearly. This proposal does not fix the problem. That said, having microconferences all over the place would be nothing but a boon for graduate students. There will always be one near their university that they can apply to attend. And nothing in this model prevents them from organizing their own microconferences.

Whatever the case, the current situation is untenable. Conferences are impossibly overrun, and their publications fully depreciated. A framework for microconferences offers something else. It is not a replacement. It is a new path to credentialing and cultivating expertise. Will it also end up a bureaucratic mess in a few years? Probably so! But I hope you’ll join me in trying it out before declaring defeat.

Subscribe now

I’ve started and stopped four posts this week and couldn’t figure out why I was stuck. Finally this morning, in a delightful retelling of an Alan Moore comic, Henry Farrell struck the nail in my head.

Henry describes Moore’s society where a supergenius benevolent dictator named Abelard Snazz designs giant police robots to fight crime. The robots get too zealous and start bothering ordinary citizens. So Snazz deploys criminal robots to keep the police busy. The cycle continues until people are forced to flee the planet.

Henry, always keen to show us what sci-fi tells us about our lived experience, describes how this automation is playing out in our lives, with people heading to chatbots to deal with bureaucracies, and governments planning new AI systems to deal with the onslaught of ChatGPT protestations. We know where this ends. And this week has seen way too many stories of this abandonment of agency and succumbing to AI.

I drafted a blog about our annual complain-fest about the academic wreckage that is the NeurIPS conference. Everyone is taking to social media to complain that people are shirking their reviewing responsibilities. They scoff at new limits of 25 papers per author per conference. They lament that AI papers and referee reports are now better than those submitted by people. They say no one reads any of these papers. And yet, everyone continues to submit papers, review papers, and be area chairs.

It is completely unclear what everyone wants from this process anymore. With every purported fix, the process gets more onerous, and the problems only get worse. More papers are submitted, more hair is torn out in frustration. Every day I’m reminded of what Kevin Baker wrote about this acceleration of publishing and perishing: “Systems can persist in dysfunction indefinitely, and absurdity is not self-correcting.” But Kevin’s next sentence is more damning. “Whether the acceleration produces collapse or adaptation or simply more of the same is not a question about the technology, and it won’t be answered by debates about capabilities.“

I drafted a blog post about open letters from employees of tech companies. They warn of all of the potential harms of the technology they are building. They beg for the government to regulate them. They decry their helplessness as they collect their 7-figure remuneration. Apparently they can’t see a path to do anything themselves.

These people want us to trust them, but then they keep admitting they are committing crimes. They write about their software as a person, trying to deflect from their own incompetence in security standards. They demand strict testing of open models while bragging about their laughably sloppy practices. They demand that the government come in and regulate them, because they can’t get out of their own way.

Even in esoteric places, we see a deferral to the AI. The lore laundering machines are now good enough to prove actual theorems, finding the key clever steps needed to take the existing literature and close “major” open problems.1 There’s a lot to say about what’s happening here. It’s still lore laundering, and mathematicians are rightfully angry that the models remain horrible at proper attribution. But the new models are generating much more clever stitches to glue the proofs together.

OpenAI has been running a distributed denial-of-service attack on unsolved math problems, and their internal tooling has finally been able to solve some very hard ones. Only three people in the world know what a non-sofic group is and why it’s important,2 but the new OpenAI Astra model has managed to construct one. The mathematical progress has created an arms race in mathematics, where people are telling mathematicians that they need to do all their work with ChatGPT now or be left behind. Or even worse, that there’s no need for mathematicians anymore. What this means for mathematics and applied mathematics more generally remains to be seen. But I guarantee you that people are going to quickly tire of theory papers on arXiv that are exclusively transcripts of professors having conversations with ChatGPT.

As Henry writes, all of these cases are arms races. As with all arms races, the momentum can feel insurmountable. I’m surrounded by people throwing up their hands as if they have no agency. But sometimes it’s important to step back for a second and see that where we’re heading is not inevitable. We do have choices here.

Researchers don’t have to send papers by the tens of thousands to conferences where no one reads anything. They can collectively design better systems to generate, evaluate, and share knowledge. Employees can use the power of their labor to steer their companies, rather than pleading for someone else to step in. Mathematicians can continue to think, understand, and teach. We all have the power to do something else.

Subscribe now

I hope the intent of my call on Monday was clear: we should strive for something that any team can build from scratch as long as they have access to the training corpus, the software, and computing resources. We want something open to the public, as the models are a reflection of public intelligence and culture. As Colin Fraser remarked on Bluesky, “there’s no going back to the world before we knew that if you make a language model large enough it appears to become a little guy who sometimes solves open math problems and sometimes makes you insane.” These language models are built upon humanity’s collective, cultural intelligence, and I believe they should thus be open and accessible to everyone.

But what “open” exactly means is tricky. A laudable example of an open corpus model is Olmo from Ai2. Their model report details all of the data used. For pretraining, they use a mix of data from Wikipedia and Wikibooks, web pages extracted by Common Crawl, academic papers sourced from arXiv papers uploaded with LaTeX, code curated from GitHub repos with flexible licenses, and math webpages from FineMath 3. Earlier versions of the Dolma corpus also include Reddit threads, papers from Semantic Scholar, and public-domain books from Project Gutenberg. All of this data is available to everyone.

Now, here’s a question for the purists out there. FineMath 3 is generated using annotations from the Llama LLM. On the one hand, technically speaking, Llama is not an open corpus model. On the other hand, the FineMath dataset is free to download. I’d argue this still counts.

Even if you grant me that one, you’ll find a lot more use of LLMs in the data pipeline in the Olmo tech report. For instance, to generate some of the math training data in the later stages of training, the team uses Qwen. Qwen is not an open-corpus model. For some of the more complex reasoning, they use thinking traces from GPT4. That’s not an open model in any capacity. If you are using a closed-corpus model to generate training data for your open-corpus model, have we ended our game for full openness?

I’m not a purist, but the vague world of distillation is genuinely complicated. Distillation is not a cleanly defined action, but roughly describes the practice of collecting the text outputs from one language model to serve as the training set for another. What we’re allowed to use and not use is incredibly confusing. Even if we just go back to pure text, it’s hard to say what training data is actually legally acceptable under “the law.”

The law is confusing and unsettled. If you buy a physical book, scan it, run OCR, and add it to your training corpus, that counts as “fair use.” If you buy the exact same text as an ebook and add it to your training corpus, that’s violating the e-book licensing agreement. The fact that there is a distinction between these two actions is absurd and stupid. Stupidity is unavoidable in our complex legal code. It doesn’t get less stupid when you try to understand how distillation meshes with the terms of service for use of LLMs.

I’m thinking out loud and consequence free here on the newsletter. I’m happy to admit that it’s complicated. But we’re going to have to change or fight laws that prevent us from just outcomes. If we want to prevent a concentration of power at OpenAI or Anthropic, we need to think about what laws are just and right to prevent that.

Anthropic CEO Dario Amodei, in his typically blindered, insufferable way, chimed into the open models debate on Monday, arguing that the US should “ban industrial-scale distillation” but that he’s not calling for banning open-weight models. His letter is riddled with contradictions like this. I don’t care about his tortured reasoning because it’s still the case that the far more defensible position is banning closed-weight models.

Closed models are indefensible. Sure, we should give Alex Radford, Ilya Sutskever, Sam Altman, and Dario Amodei some credit because I would have never guessed that the models would be as useful as they are today. As Fraser said, there’s no going back to the time before we knew that. However, you don’t have to give them too much credit because you could also argue they should be in jail. If you think that is hyperbole, I’d like you to read the wikipedia page of Aaron Swartz. Or read this in-depth article in the Financial Times about the online libraries used to train our current language machines. The librarians are hunted across the globe by the FBI. The LLM entrepreneurs are gazillionaires and can pay billion-dollar settlements to cover their asses. We can have long, pedantic quibbles about what’s legal and what’s not. We can also ask, “What is right?” and “What is just?” The current situation where the best American models remain closed is deeply wrong.

Subscribe now

I applaud Jensen Huang and industry leaders for coming out in support of open large language models. The entire tech sector minus Anthropic has now signed on. Now that the movement has momentum, I urge the signatories to endorse something even more radical that would truly separate us in our economic competition with China: The USA should heavily invest in models not just with open weights, but with open source and open corpus.

Let me back up in case you missed it. Last Friday, Huang pasted a short letter on Twitter calling for American investment in “a strong, open ecosystem” around open source artificial intelligence. “Open source…created a shared foundation of knowledge on which generations of American engineers and entrepreneurs built their institutional sovereignty,” Huang predicts that a similar thing will happen with the embrace of open source AI.

Part of this letter was a plea to the Trump administration, which had been sending signals about banning open source software in a protectionist move to squash insurgent Chinese language machines. Seeing that this would benefit Anthropic and OpenAI and no one else, the CEOs of all of the big and small tech firms quickly signed on to Huang’s letter on Friday.

This is an amazing, positive development, and I want to air some points that Huang left out. I’ve been passionately calling for a deeper investment in open source language models on this blog for years. In fact, this post from last July makes a bunch of points about what I think is necessary for this to happen, and I don’t think any of it has changed.

You should go read that post, but let me summarize what I said for the present moment. What is needed to build a good language model is people, compute, and data. It’s not clear how much of each of these you need, but the answer seems to be a lot. We’ve always had the people, though the promise of impossible riches has lured many smart young minds away from more ethical paths. But now that we have the backing of the entire tech industry, I think this base is covered.

What about compute? Hyperscalers and their economic bedfellows always want you to believe that their massive datacenter buildout is a moat from us open source plebes. But the Chinese models have complicated this story and shown you can get by on the cheap. No one has precise estimates, but models from Chinese companies DeepSeek and Moonshot were arguably trained using under ten million dollars. Dozens of AI startup “NeoLabs” with few concrete ideas are currently being given 10x that by venture firms. Good VCs could make a small bet on the open ecosystem to juice their other investments. Compute for open models will require a massive industrial consortium, but it would cost each member of the consortium a minuscule fraction of their war chests. Let’s work together to build computing agreements for open source development.

Once these models are built out in the open, there is no doubt that they will only get dramatically more efficient. We’ve seen time and time again that machine learning is a field that innovates through “frictionless reproducibility.” Research, code, and data out in the open, evaluated by competitive benchmarking, rapidly improve machine learning systems. It also makes them more efficient. In last year’s post, I wrote about how high-quality ImageNet models went from something only trainable at Google to something you could build on a desktop in less than a year. These sorts of efficiency gains happened throughout the 2010s. The secrecy of labs in the 2020s has harmed the broader engineering field, even though the artifacts produced have been beyond impressive. Put everything out in the open, and we’ll figure out how to make it faster and more efficient. It’s guaranteed.

That brings us to the actual hard part. The data. As my friend and colleague Alyosha Efros loves to remind us, “It’s all about the data,” and language models are trained on unfathomable amounts of it. Discovery in lawsuits has revealed that companies trained these models on pirated libraries of books, academic papers, and copyrighted imagery. The models are trained on collaborative knowledge bases like Wikipedia, countless volunteer forums like Reddit, and all of the public code on GitHub. They are trained on the transcript of every video you post to YouTube. All of this collective work by human society gets slurried into proprietary software so a few zealots can get rich. This is a bad outcome!

The companies are all up front about this use. They have been found liable in court. They have admitted it in papers they have written. They claim this is all fair use because the training was transformative of the texts. Fine, if that’s the case, then it’s fair use to take the outputs of their models and build new ones. This is called “distillation.” Chatbot terms of service agreements do not negate my argument. That the Trump administration is flirting with banning distillation is a travesty.

But I want something bigger. The biggest step to making competitive open source models is allowing the broader community fair use access to the same material the companies used. This is the open corpus. If closed models can exist, then open models should be allowed a level playing field. This will require a long overdue rethinking of intellectual property and what we owe the people who create it.

I think all of these challenges are surmountable. We’re going to have to (a) get young people to care about open source instead of becoming impossibly rich. (b) get billionaires to collaborate in a non-winner-take-all fashion, and (c) have a long hard conversation about copyright law, intellectual property, and fair use standards. But we’ve thrived in an open ecosystem before ChatGPT. It hasn’t even been half a decade of closedness. With the blessing of the entire tech industry, it’s time to open things up again.

Subscribe now

The title of this post is a coinage of Kevin Kelly. His dedication also inspired me to write this post.

I’ve been obsessed with playing guitar and making music since I was thirteen. Obviously, as a giant nerd, I approach this in the nerdiest way possible. So much of my playing is thinking about the mathematical relationships in patterns of intervals and shapes. I love challenging abrasive music that involves odd counting or microtones. I geek out on music technology, be it software or hardware. But in the decades I’ve been doing this, I’ve never come across someone who thinks I’d be able to get better at music through science.

Why are there no science-based guitar lessons? There are countless books with varied methods on the fastest way to learn or the best way to get better. There’s beautiful math you can nerd out on. Just like in weightlifting, people sell their lessons online on Instagram. But none of the musicians I follow ever put up PubMed screenshots. Instead, you find a psychopathic Spaniard who learns to play impossibly hard drum parts with one arm tied behind his back.

Don’t the laws of adaptation apply to music? It’s certainly the case that the more you practice, the better you get. Much of music practice is just sensorimotor learning. You are adapting both neural pathways and strengthening muscles. There’s psychophysical adaptation in syncing your limbs up with a click track. There are fine motor skills involved in hyperefficient picking. It’s also clear that if you practice too much, you’ll make your hands bleed or develop tendonitis. Surely Selye’s General Adaptive Syndrome applies to woodshedding.

I mean, the concept of progressive overload clearly applies to practice. The way you learn a song is to set your metronome to half time and work your way through the notes. Once you have it down, you increase the beats per minute by one. Each time you come back, you play faster than last time. Next thing you know, you are shredding Chopin.

And it’s not like there’s one perfect way to learn how to be the best at your instrument. There are wide disagreements about the best auxiliary exercises to improve your chops. There is an infinite collection of tutorials and courses designed to improve your skills, telling you which rhythms to practice, which strings to skip, which positions to memorize. And yet, no one thinks you need to run RCTs on these books to crown the best one.

You might say music exists more for art than for competition, and I’d applaud your idealism. But the music industry is a rough one, and it gets pretty cutthroat when you try to make a career out of it.

I write all this to ask what it is about the gym that makes us think that we can science it. I don’t have a good answer to this yet. I have a few partial answers, but none are particularly satisfying to me.

First, there is a clear connection to medicine, and for a wide variety of reasons we’ve decided that medicine needs to be based in science. Physical therapy can only argue for legitimacy in the healthcare sphere if it can be proven a cost-efficient therapy. That means it must be based in our post-modern science of efficiency, and hence we are forced to run RCTs.

The medical reasoning breaks down when it comes to weight lifting. A lot of the biggest names in the science-based lifting community are obsessed with hypertrophy, the fancy scientific word for increased muscle size. While it’s true that to be a strength athlete you need to have big muscles, most of these guys are selling programs for aesthetics. Dudes want to grow their muscles because looking jacked builds their self-esteem. The sport of hypertrophy, bodybuilding, is competitive body dysmorphia. Bodybuilding is an astonishingly unhealthy sport. It involves taking ungodly doses of performance-enhancing drugs, alternating binge eating with starvation, and walking around at dangerously low body fat levels. The pro athletes in this sport die at rates far higher than football players. It’s a mess. There is no good health-related reason for you to look like a modern bodybuilder.

But perhaps the universal desire of perfect aesthetics is a core part of why science is so alluring. Not everyone wants to be able to play 10 over 11 polyrhythms, but everyone wants to look hot. And since it’s so universal, the nerds want a way to stick it to the jocks and claim a masculine domain that hadn’t originally been theirs. The authority of science gives the nerds an institutional leg up in a world where all men want to stake their claim. Science-based training is to the gym as the analytics department is to professional sports teams.

However, when it comes to the strength sports, the science takes you about as far as the analytics takes a football team. Strength sports are not as quantitative as they look. There’s no optimal answer for any person to follow to become a championship Olympic weightlifter, even though the goal of that sport is to get the sum of two numbers as high as possible. Working hard, resting, and eating get you 80% of the way there. Science can’t fill in the rest. If you want to do heavy clean and jerks and snatches, you have to show up and do them.

Subscribe now

Last week in Zócalo Public Square, I wrote a piece on optimizing diets, adapted from the second chapter of The Irrational Decision.1 It’s one of my favorite stories in the book. Though it predates computers by several years, it’s a microcosm of the computer age and fits in seamlessly with today’s oddly dominant online culture of wellness optimizers. In a spat with USDA nutritionist Hazel Stiebeling about what sorts of recommendations are acceptable for the government to publish, the prickly economist George Stigler solved a complicated tableau by hand to find a rather unpalatable “minimum cost subsistence diet” of wheat flour and navy beans. Go read the essay, and then come back here and read a few fun addenda.

I tell the story of the diet problem in most of my book talks, and I always receive fun feedback. Jeff Linderoth sent me a hilarious reflection by George Dantzig, the inventor of linear programming, on his apparently futile attempt to find his own optimal diet to lose weight. Though Dantzig knew Stigler’s optimization problem had nothing but absurd and disgusting solutions, he figured he was adept enough at building linear programming models to patch Stigler’s simplistic assumptions with appropriate shaping of objectives and constraints. He used spare cycles of the IBM 701 at the RAND Corporation to churn out ever more innovative meal plans. But he kept getting bizarre recommendations, like drinking gallons of vinegar or consuming mass quantities of bouillon. He diligently refined the diet over the course of a week before his wife, Anne, got fed up:

Speaking firmly so that I would know who was boss, she said, “I have been studying the various menus the computer has been generating. There are some good ideas there that I can use. I’ll put you on MY diet. She did and I lost 22 pounds.

Steve Stigler—not only an amazing statistician and historian but also George’s son—attended my talk at the University of Chicago. Steve told me how this paper made its way out of academia and into national newspapers, ruffling feathers from coast to coast. George would receive angry letters scolding him about how “this is no way to feed growing boys.” I thought it was pretty clear from reading the original paper that George didn’t think anyone should try to eat his diet. He was trying to prove a point about the impossibility of optimal diets and the paternalistic nature of government recommendations. But people ended up taking him literally. Papers that start as sardonic jokes can surprisingly take on a life of their own.

Stigler’s paper is part of a broader conversation about the scope of government policy. The idea of a computable government was central to economic discussions during the Great Depression and throughout the Second World War. Experts and government officials argued about what is optimal, what can be planned centrally, what individuals should be allowed to navigate for themselves, and what sort of information is beneficial and which is coercive. These debates strongly influenced von Neumann, as you can see in his and Morgenstern’s engagement with contemporary economic debate in the introduction to their revolutionary book on game theory. For another fun example of the people building computers closely interacting with the people designing policy, here’s a 1958 photograph of a sharply dressed Claude Shannon at the Center for Advanced Study in the Behavioral Sciences at Stanford, taken by George Stigler on Steve Stigler’s camera.2

The diet debate also raises the uncomfortable central theme in Elizabeth Popp Berman’s book, Thinking Like an Economist. Everyone across the political spectrum is arguing about efficiency, as if that’s the only thing the government should think about. Left-wing technocrats (aka the Democrats) apply this sort of economic thinking to the utility of the population. Right-wing policymakers (aka Republicans) apply economic thinking to the utility of the individuals in that population. No matter their politics, everyone is thinking like an economist. The valence of the arguments and the parties making those arguments remain uncannily similar today.

Given the grand scale and ambition of the federal government, USDA dietary guidelines should be a fourth-order concern. But there’s something about worrying about what we should eat that galvanizes the popular imagination. It’s fun to walk through the original arguments about what should be in the food pyramid, especially given the weird steak-centric geometry being pushed by RFK’s cuckoo version of HHS. A steak every day sure sounds more appealing than a bean pie.

Subscribe now

In the noisy chaos of science-based wellness on social media, there are a few reasonable voices. One of my favorites1 is Greg Nuckols, a champion powerlifter who has been blogging about the science of strength training for well over a decade. Over the weekend, I read his 2015 books The Art of Lifting and The Science of Lifting, and they hammer home how a simple topic can be made overly complicated when we decide to write scientific papers about it.

Nuckols’ two-volume series is less than two hundred pages long and written in an engaging, bloggy voice. Though the books link out to systematic reviews and randomized trial reports, he doesn’t get bogged down in that material. Instead, he focuses on one simple modeling principle and its consequences: adaptation.

Though the human body is impossibly complex, its many interconnected systems deal with stressors in a surprisingly uniform way. Stressor here means all sorts of things: wounds, infections, temperature changes, environmental changes, activity changes. Something out of the ordinary. The model of homeostasis posits that bodies actively work to stay the same as much as possible. Most of our daily activities don’t stress the body, and we remain in a pleasant, steady state. However, the body changes and adapts if it gets pushed far enough out of its comfort zone. If you apply enough stress, the body initiates a response to prevent harm or death. There are limits to this response, and too much stress will cause injury or death. But in the right Goldilocks range, the body will reconfigure itself to resist the stress next time it sees it.

Exercise can be modeled as a stressor. It’s certainly stressful. It increases muscle tension and body temperature. It increases demand for oxygen and nutrients. It lights up your sympathetic nervous system and triggers the release of adrenaline. Your body responds to exercise similarly to how it fights off other stressors like injuries and infections. It panics and then marshals resources to make sure it sucks less the next time you go to the gym.

There are many models for the temporal behavior of the stress reaction. The most common mathematical model is the fitness fatigue model developed by Tom Calvert, Eric Banister, and collaborators in the 1970s. Notably, Calvert was an electrical engineer who introduced his physiology colleagues to cybernetics and impulse responses. The fitness-fatigue model is a parametric model of adaptation: stressors introduce bad effects (fatigue) from which the body works hard to recover quickly. It also introduces positive training effects, the adaptations you were chasing. The fitness-fatigue model posits that the positive effects decay more slowly than the negative ones. For the dynamical systems nerds out there: they model this with a simple second-order linear system. Hence, if you repeat this process of enough but not too much stress over time, you nudge your body to handle more and more stress.

This feels science-y, no? The general adaptation syndrome is a well-tested, clean model that certainly helps guide how to train athletes and implement physical therapy. This model hasn’t changed since 1980. It prescribes working hard and continuing to do so over time to increase adaptation. It predicts that training harder than last time is beneficial. These two together form the principle of progressive overload, a very simple concept that influencers try to make absurdly complicated. Moreover, this model of adaptation tells you that you need to recover to hasten the elimination of fatigue. It even suggests implementing tapering periods before competition, a common practice.

OK great, now let’s try to apply this model to practice. What is the right shape of the actual stress to give you the results you want? If you want to get stronger, which exercises should you do? How frequently should you do them? What weights should you use? How many repetitions should you do for each exercise?

Unlike most of the charlatans in the science-based fitness space, Nuckols is brutally honest about this: the answer to all of these questions is that we don’t really know. Or at least, we at best only have partial answers. You probably need to go above some nominal effort to induce adaptation. Because there is so much intersubject variability, the exact perfect amount is hard to pin down. But if you work hard and make sure to rest and eat, you’ll get stronger over time.

Nuckols thinks the broader scientific research on sports training tells us more than I think it does. From my reading of too many studies in this space, the ns are always really small, the data are always poor, and the assessments are always ill defined. Do we need thousands of studies to tell us that we should “keep it simple, stupid?” People obviously can go to the gym and get stronger.

But what’s interesting to me about how Nuckols writes is that he doesn’t consider “science” to be some combination of biomolecular pathways and randomized trials. Instead, his scientific view is one of simple predictive models and their limitations. For an athlete like Nuckols, the scientific view helps systematize what matters in training. Working hard matters. Working harder over time matters. But resting and recovery are also critical to good performance. The right balance might not be easy to quantify, but the qualitative arc and predictions based on principles of adaptation are helpful. The general adaptation model for training tells us that training isn’t that complicated if you think carefully about what you are going to measure, and how you are going to track progress. This cybernetic view of biomedical science—one closer to what I would call engineering—feels like a promising path for many other aspects of medicine.

Subscribe now

Of the many cursed segments of vertical video, few annoy me more than the cottage industry of lunatics ranting over screenshots of PubMed pages to claim authority for whatever therapy, training program, or peptide they are selling. These people are all full of shit. They promise to cure your pain, get you thin, and make you stronger. All through SCIENCE.

These influencers will tell you that the scientific is better than the not scientific, and might even go so far as to say that the remainder is pseudoscience or quackery. But we unfortunately live in a narrow-minded world where scientific too often means “proven efficacious in a systematic review of randomized controlled trials.” You’ll be hard-pressed to find anything in the realm of treating musculoskeletal limitations that fits the bill.

It’s hard to shoehorn these sorts of therapies into the starting requirements for randomized trials. We’d need to start with clean definitions of an intervention and an outcome. What would these be for the management of pain by physical therapy?

Let’s start with the intervention. If we are being dogmatic evidence-based practitioners, the intervention in any physical therapy trial is the invitation to have therapy. According to the intention-to-treat principle, the invitation is the only thing that can be randomized. We can’t look only at the people who comply with every instruction and make it to the end of the rigorous therapy regimen. The model patients who diligently comply might differ from those who don’t, and our statistical signal will be biased if we include only the former. The only way to avoid these selection biases is to count everyone who was randomized, regardless of what happens between randomization and the final assessment. Both committed patients and no-shows contribute to the measured average efficacy.1

But what does it even mean for patients to follow the protocol? Physical therapy is far more complex than taking a drug. There’s no simple unit of treatment applied. Each interaction with a physical therapist involves a conversation about how things have been going, a plan for moving forward, some sort of interactive intervention in the office, and a discussion about what to do once the patient goes home. Each step here introduces a new branch in a deep decision tree. And every PT I’ve interacted with has been different, even when performing similar range-of-motion tests or manual therapies. Moreover, the treatment of any session depends on the entire history of the treatment so far. On top of this, every physical therapist I’ve seen has assigned daily exercises to do between sessions. This is part of the treatment, too! There is no way to perfectly isolate and randomize a single component of these complex treatment protocols.

A multi-stage protocol is an exponentially large collection of interventions. I made this point on the blog a few weeks ago in the context of anticoagulant trials for heart disease: “If you want to compare the effect of three different timings and three different dosages of a single drug, you need nine arms in your trial. If you want to additionally see if a second drug is helpful, you need 18.” Physical therapy is arguably much more complex.2

What about the outcome? In drug trials, we might get grim, unambiguous, objective outcomes like mortality. In vaccine trials, we might get unambiguous outcomes, such as a PCR diagnosis. Unfortunately, in pain management, the outcome is necessarily subjective. You can measure changes in range of pain-free motion, but there is too much heterogeneity to definitively stake out what a good outcome would be. Instead, pain therapies are most commonly evaluated based on improvements on the Numeric Ranking Scale. Studies ask participants at admission how their pain is on a scale of 0 (no pain) to 10 (worst pain imaginable). They ask them again at the follow-up. Statistical protocol then dictates computing the mean of the differences in treatment and control and running a t-test. You can try to remove the heterogeneity in how people respond to these questions, but these adjustments are based on subjective clinician calls. No matter what you do, pain is hard to mathematize. Doing statistics on these “numbers” and coming away with strong conclusions is a fool’s errand.

Beyond the treatment and outcome, all sorts of investigator biases make randomized trials even messier. You can blind the patients and the clinicians who assess outcomes, but you can’t blind the people applying physical therapy. It’s impossible to say what effect this sort of bias has on the scientific record. Even when well-intentioned, a clinician who believes in PT can subtly give away the secret assignment to their patients during a session.

I’ve never read a single study in this space that’s been compelling, and I don’t know why we hope that a narrow view of therapy can help us out of it. This fuels the fire of debate with people using studies to attack each other’s practices. There are countless articles and videos castigating stretching, massage, or cupping as not backed by evidence. These are all denounced as pseudoscience by a medical establishment that prescribed OxyContin like candy for two decades. Boy do I have some bad news for people who think there is great evidence that opioids work for pain management.

If we want to understand best practices for “wellness,” we need a different language around it. Maybe this language will need to lean on biomechanical plausibility or biochemical pathways. That certainly wouldn’t hurt. But more importantly, the language will have to prioritize discussions of craft, practice, and the cultivation of expertise. Whatever the case, the narrow definition of evidence-based needs to be reimagined. Healthcare is far more than a collection of unambiguous interventions with unambiguous outcomes.

Subscribe now

Monday’s post highlighted how physical therapy sits at the heart of the tension between populations and individuals in modern scientific medicine. The field addresses one of the core problems people come to their doctors with: musculoskeletal pain or restriction. More often than not, these ailments can’t be treated by simple prescriptions of rest, drugs, or surgery, and primary care physicians don’t have solutions for these problems. So instead, they are often referred to physical therapy, which combines an uncomfortable mix of science and craft that eludes clean, evidence-based evaluation.

I’m a bit of a PT fanatic, and have had the privilege of working with five excellent physical therapists since I got a bit too overly serious about strength and conditioning in my forties. They were all very different, but all of them applied the same core principles. These principles highlight key aspects of what it means for a therapy to be individualized.

Every PT consultation begins with an assessment. Because there are so many moving parts and everything is so interconnected, it’s often hard to identify a single cause of a particular pain or restriction. Weakness in one area is balanced by strength in another. Pain in your foot might be caused by limited mobility in your hip. While everyone’s musculoskeletal system is connected in the same way, we have wide variability in the sizes, shapes, and positioning of our muscles and bones. Once we combine this with the variability of people’s physical interactions with reality, we find there’s no single simple answer for everyone. Each person is a weird biomechanical puzzle, and not every puzzle is particularly easy to solve. However, there is a set of therapeutic principles for making people better.

A therapist will test out the range of the different muscles and joints that could be connected to your symptoms. They try to find how far you can move without pain, which is a mix of qualitative (pain) and quantitative (degrees of rotation). Once they find which specific movements are restricted, they have a general process to fix all of them.

First, the therapist will open the restricted range using some sort of manual therapy. This is done in a single session and might involve stretching, deep-tissue massage, or even cupping (more on that later). Next, they give you exercises to gain control over that newly opened range. This is done between sessions with the homework exercises they assign you. After you can confidently access this new range, you load it, adding strength-bearing exercises to further strengthen the mobility you aim to increase. All of these are progressed over time, adding more intense manual therapy, more difficult exercises, and more weight.

Progression is the core principle, whether you’re treating tennis elbow or a sprained hip (I’ve had both). The difficulty of your physical therapy exercises will gradually increase with each session. It’s sort of obvious when you say it that way, no? This is all there is to the buzzy principle of progressive overload. It works because your body adapts to stress. When you add enough stress, it overcompensates and resists the stress more strongly the next time. Cascading this reaction over a long period in a controlled, thoughtful way leads to a clear and measurable outcome.

The complication is that bodies adapt more slowly than most of us would like. It might take weeks to start to see progress. Because progressive overload is slow, lots of people walk away from physical therapy convinced that their injury would have just gotten better anyway. They might be right, and I can’t argue their concerns away. But that’s why I’ve found it helpful to track progress over time. It might not be the PT prescription that lets me access that extra range or strength, but I feel more in touch with whatever it is that my body is doing. I’ve found that it’s worth targeting something even more ambitious than just “back to normal.” I now aim to come out of physical therapy more capable than I was before my symptoms started.

For many people, PT doesn’t work. I like it, but I might be particularly well suited for it. I love rituals. I love obsessing about niche minutiae. I love tracking and journaling. But recovering from injury and managing pain are not easy. Compliance with physical therapy might be a greater challenge than with other treatments. It’s a much bigger commitment than “take two pills a day with meals.” If you don’t like to spend 30 minutes a day on weird stretches, PT might not work for you. But that doesn’t mean it doesn’t work for someone else.

And this is why physical therapy is so easy to attack on scientific grounds. How do you run a controlled trial on a practice this complex and individualized aimed to treat subjective symptoms like pain? I’ll dig into the impossibility in the next post.

Subscribe now

Last week in the Washington Post, fitness columnist Gretchen Reynolds wrote about a “new way to treat sprains.” The common, catchy advice of RICE (Rest, Ice, Compression, and Elevation) was out, and new science had come to light: Doctors and scientists now concur with youth sports coaches that you should “walk it off.”

I’m barely being glib. Reynolds quoted research suggesting that reducing inflammation impairs the body’s natural healing mechanisms. By damping inflammation, scientists conjecture you are damping regeneration. This means that anti-inflammatory drugs like ibuprofen are also counterproductive, as they suppress key pathways that aid the healing process. Moreover, resting leads to muscular atrophy and loss of strength in tendons and ligaments. Putting it all together, rest and anti-inflammatory therapies compound to impede healing.

Reynolds cites expert studies on this topic (there are always studies), but to someone immersed in the culture of bro-science, I was surprised this advice was considered new and novel. Since getting a bit too obsessed with strength sports, I have injured myself multiple times, and I have never visited a physical therapist or watched a fitness YouTuber who recommends resting after an injury. Instead, most tell you to move as much as possible and to reduce inflammation only if it’s getting in the way of that movement. If you get injured, they advise you not to panic, to find what motion gets you moving, and to exercise around the limitations brought on by the injury. The place you sprained will be mobility-restricted, but you should access as much range as you can and keep things light, slowly adding work back as you go.

The working model that coaches and PTs share is that weight training and injury recovery are both processes for getting stronger. An injury is just a disadvantaged starting point. In both cases, the way to increase strength is to regularly stress the system with increasingly harder tasks. Every time, you hope to access more range or lift more weight. This is the principle of progressive overload.

Though now the backbone of most sports training and fitness programs, progressive overload originates in physical therapy. The common prescription of “three sets of ten” originates in the practice of physician Thomas DeLorme, who worked to rehab injured soldiers during World War II. His plan to heal, borne out by diligent clinical practice, became the foundation of the modern paradigm for getting stronger.

Though DeLorme’s progressive overload principle worked wonders in patients and athletes alike, standard medical practice unfortunately always needs a stamp of “science” to back it up and make it real. The practice of coaches, trainers, and physical therapists everywhere can’t justify a technique. That’s bro-science. You need some obscure microbiological pathway or a randomized experiment.

The problem is, the science of physiology is much murkier than you might expect. For years, I have been looking for a clean mechanistic description of why bodies adapt to training. Why do we get stronger and faster by progressively working harder at the gym? Reynolds’ column quotes UC Davis Professor Keith Baar applying his research to his own ankle sprain. Baar has long studied the molecular biological pathways that explain why training techniques like progressive overload work and why bodies adapt to stress. He co-authored one of my favorite surveys on adaptation to training, which lays out the myriad biological pathways and how different sports modalities shape them. My main takeaway from the survey is that we still don’t have a clear, comprehensive description of how the body adapts to exercise. The experiments all agree that if you progressively overload, then you get better. There are lots of things you can measure in a body under stress that point to something mechanistic going on. But they can’t pin down a precise mechanism that drives adaptation, and they certainly can’t figure out what an optimal training protocol would look like. In sports medicine, we don’t follow the science. Science follows the practice.

Coming back to inflammation, we can look at the evidence that ibuprofen inhibits muscle growth in strength training. Common gym wisdom is that ibuprofen, those sugar-coated orange pills often consumed like candy by gym addicts, are actually killing your gains. Bros tell you that you have to tough out the delayed onset muscle soreness associated with training. This was also the conclusion conveyed by the scientists quoted in Reynolds’ article.

Well, the science is of course far less certain than the YouTube advice. I could only find one study showing that taking ibuprofen inhibits muscle hypertrophy when strength training. In this study of 31 people aged 18-35, the treatment group took 1.2 grams of ibuprofen every day for eight weeks, and the researchers observed that they had less muscle growth than the control group. 1.2 grams is the maximum recommended over-the-counter dosage! This is a bit extreme. And yet, despite multiple attempts, no study has ever reproduced these results. There’s a fun fight between one of the failed replication teams and the original authors if you’re into those sorts of academic spats.

So who knows, team? It’s possible ibuprofen doesn’t affect training performance at all. I personally wouldn’t recommend eating it like candy. But what do I know? I’m not a scientist.

Subscribe now

My argmin.md file compels me to write a blog about blogging every July 1st. It’s my official Substack anniversary, and we’ve made it through year three. The process here is, for better or worse, set in stone and could be written as a skill file for an LLM agent. I listed the rules last year, and they haven’t changed since. I don’t expect them to change this year either.

In fact, going back over my posts, it seems like the most novel change was adding a new kind of content that got decidedly mixed reviews: football blogging. Of all the topics I’ve dabbled with on this Substack, there is none more polarizing. Some people really liked it! Others, let’s say, did not. I’m waiting until training camp gets going before I decide whether I’ll commit to another season of yelling about the absurdity of win probability.

The football blogging is related to one of the trickier things I haven’t been able to figure out in three years writing this blog. Argmin is a periodical with different authors. Those authors just all happen to be me. I realize you receive a notification from me whether I’m writing about a topic you care about or not. This is a problem I don’t really know how to address, but I suppose it is an issue with all Substack subscriptions. I don’t want to take on the bother of splitting this into four different Substacks, which I think would annoy you even more than it would annoy me. I hope the disclaimers at the top of the posts help you decide if you want to keep reading. I would love feedback on how to make that signposting clearer.

Relatedly, commenters are increasingly asking questions about things I’ve already talked about. This is totally expected, as I know I write more than anyone likely cares to read, but I worry about repeating myself—thus alienating a different collection of readers—when addressing them. The balance between repetition and clarity is delicate. Apoorva Lal said my blog has its own private dialect. If I don’t repeat myself somewhat, filling in the context of what my brain thinks obvious, the content will be even less approachable, and no one wants that.

Moreover, sometimes the only way to iron out a nonobvious point is by repeating it until it’s clear. I wrote in the afterword of The Irrational Decision that I used this blog “as a sketch pad, where I first workshopped much of the text that you read here.” This newsletter remains a place for me to play and write out first drafts. I get a lot out of putting ideas out there, even when they are way underbaked. “Talks and blogs are ephemeral, but interactions with the audience helped to shape and hone my arguments. All of those early interlocutors, in person and online, helped shape this final archival form. I’m thankful to all of you.”

In the spirit of learning from the audience, we can use this year’s reflection post as a chance for you to tell me what you’d change about the argmin magazine. Do you want more of any topic? Would you prefer a different format or cadence of posts? You can email me if you’d prefer not to comment (my berkeley.edu address is fine for that).

In the meantime, I can give you a preview of what to expect over the next year. There will definitely be more live course blogging. In the fall, I am planning an unhinged new class on forecasting, the tools we use to do it, and why we’re so obsessed with it. That should be a fun one. In the Spring, I’m apparently signed up to teach undergrad probability. This is for engineering students and based on the excellent text of Bertsekas and Tsitsiklis. Since it’s about engineering, we’ll have to grapple with some of the complicated metaphysics of what we think these probabilities mean in the context of building things. You, me, and the undergrads will all get confused together.

As for the remainder of this summer, I want to finish this series Against Optimaxxing. I’ve been talking about writing this forever, and I need to put in the effort to finish it. It would be an attempt at writing what Henry Farrell calls speculative nonfiction, seeking a reasonable language to imagine a future free from the vulgar positivism that runs our culture, politics, and industry. This blog series will try to establish a necessary vocabulary. My draft writing on this particular topic might lead nowhere. It might remain a private dialect. But I want to try to write it down.

Subscribe now

Yesterday on Good Morning America, NFL Hall of Famer Chris Johnson announced that he had ALS. Johnson, nicknamed CJ2K for his 2006 rushing yards in 2009, was one of the most electrifying players I’ve ever seen. He was impossibly fast. Every one of his carries was must-watch football, as you’d never know when he’d break a run for a touchdown. Johnson still holds the record for the most yards from scrimmage in a season, with his total of 2509 a comfortable 80 yards in front of second place.

Johnson retired in 2018, and, like most football players, disappeared from the national spotlight. So it was beyond heartbreaking to see him yesterday, at 40 years old, speaking to Michael Strahan with an eye-tracking device. ALS attacks the body’s nerve cells, progressively inhibiting a person’s ability to move their limbs, speak, and breathe. Though highly variable, the progression can be shockingly rapid. Johnson told GMA that he had been diagnosed less than a year ago. He wanted to share his story to bring more attention to ALS and the search for radical new treatments.

In 2023, Gideon Lewis-Kraus wrote a moving investigation into this search for The New Yorker. People with ALS and their families are, for quite obvious reasons, passionately dedicated to finding any reasonable treatments for the disease. The hope is to find therapies that at least slow disease progression and improve the patient’s quality of life. The challenge is this tends to require drugs, and ALS lies outside the sweet spot of modern drug discovery.

ALS is a rare but devastating condition, with around 5,000 new cases diagnosed in the United States every year. There is no clear established cause for ALS. Though it typically manifests in people’s late fifties or early sixties, Johnson’s case exemplifies how unpredictable its onset can be. Sometimes ALS progression takes years; sometimes it rapidly advances in months. There are no clear genetic signals for ALS, and family history isn’t predictive. There is a biomarker present in the nerve cells of almost all ALS patients, but current technology only lets us detect this biomarker after death.

Without access to a marker, measuring progression or reversal is a subjective challenge. Current ALS surrogates numerically assess bodily functions like speech, swallowing, breathing, and motor skills, and combine these different tests into a single number. For example, the ALSFRS-R score is the sum of 12 different assessments, each scored on a scale from 0 to 4, where 0 indicates total loss of function, and 4 indicates no loss of function.

All of this adds up to make drug testing a major challenge. It’s hard to define reasonable endpoints, and most trials aim to find a statistically significant difference between treatment and control in ordinal assessments made by clinicians. Though patients and their families urgently seek new possible therapies, there are not enough ALS patients to meet the harsh statistical standards of clinical trials. Moreover, drugs require trials, and the ethics of potentially denying the control group something beneficial are complex and fraught.1

Lewis-Kraus draws connections to the disease that set the standard for patient advocacy against institutionalized medicine: AIDS. AIDS activists in the 1980s were not only vocal in spreading awareness of the disease and its various treatments, but they actively engaged in scientific research and drug policy. It’s a remarkable example of a citizen group not only challenging, but shaping scientific expertise. To this day, AIDS activists remain go-to consultants for helping people and families affected by rare diseases to organize and advocate for the afflicted.

In hindsight, HIV treatment is similar to the treatment of other progressive diseases like ALS. None of the treatments cured an HIV infection, but they allowed patients to manage their infection. Today, HIV can be treated pharmaceutically with an excellent long-term prognosis. Though initially a terrifying death sentence, people with HIV infections can live a normal life with access to the proper medication.

The differences between AIDS and ALS highlight what makes a progressive, terminal condition treatable. AIDS was caused by an infection with the virus HIV. The first effective HIV treatment, AZT, targeted the virus by fooling it to bind to a byproduct of the drug rather than the person’s DNA manufacturing system. This eliminated HIV’s ability to replicate. Moreover, AIDS had an easily measurable surrogate marker to monitor progress. HIV killed a particular type of white blood cell, and counting these cells was a reasonable way to test if the drug was having an effect.

Sadly, while researchers are hopeful that we’re making rapid progress, we’re still missing many crucial causal pieces to the ALS story that might illuminate an appropriate target for the pharmaceutical industry. Johnson hopes that raising awareness of ALS might help us find the missing link in understanding and treating the disease. He wants to keep fighting. My heart goes out to CJ2K, his family, and everyone else suffering from this terrible condition. And I hope his inspiration will help us find the missing gap for a breakthrough treatment.

Subscribe now

My teacher and friend Dimitri Bertsekas passed away earlier this month. I just learned about this over the weekend, and I’m still processing my thoughts.

Dimitri was a hero of mine for so many reasons. I took my first and only class on optimization with him, and I was riveted by his clean presentation of convex analysis and mastery of the overhead projector. He took a liking to me, and I made it a point to find him for a chat whenever I’d visit MIT. He was always generous with his time and excited to exchange ideas, but he wouldn’t hesitate to harshly scold me when I’d present some result new to me that he had in fact written about twenty years earlier.

This was unavoidable because Dimitri did foundational work across mathematical optimization, penning landmark results in stochastic gradient descent, convex optimization, distributed optimization, dynamic programming, and reinforcement learning.

His work in reinforcement learning remains underrated. His collaboration with John Tsitsiklis, compiled in their book Neurodynamic Programming, was the first to show that most reinforcement learning algorithms were effectively approximating dynamic programming. Our contemporary, model-free mindset, rooted in Markov decision processes, derives from these initial insights. As far as actual practice goes, their book is more important to the modern way we implement reinforcement learning than Sutton and Barto’s.

Dimitri was also a passionate mathematical communicator. I own more of his books than any other mathematics researcher, but he wrote more than any other mathematics researcher in my field. Is there too much material? You could make the case! However, that he has definitive texts covering a broad range of optimization theory is remarkable.

In many ways, Dimitri was an original math blogger. He wrote exactly what he wanted to write, and he wrote frequently. He got fed up with publishers getting in the way of his process, so he started his own publishing company, Athena Scientific, shipping stacks of books out of his garage in Belmont, Massachusetts. This allowed him to write countless new editions and revisions of his work, reflecting trends in practice and incorporating new insights and simplified arguments. Though you’ll see plenty of repetition across his volumes, he knew that no book was a final draft. Each volume was a step towards broader understanding.

A decade ago, when he retired from MIT, I wrote a post of appreciation for his scholarly and pedagogical work. I’m going to reprint it here today, as it includes one of my favorite passages in his books about the tension between theory and practice. Right now, in our frenzied agentic era, we’re leaning heavily on only practice and vulgar empiricism to push out as many papers and products as we can before the bubble pops. Is there a role for theory at all anymore?

Dimitri argued that optimization theory is always a mix of qualitative and quantitative. The qualitative helps us understand what we know and don’t know. By gathering feedback from practice and constructing a working narrative, theorists can help engineers develop a language to describe what is possible and what can be improved. Theory provides a narrative scaffolding that helps us understand what to build and what to attend to when things break. Building stories of connections helps us streamline our processes and try things we might not have thought of.

There’s an ebb and flow between the theory and practice. I will revisit this ten years from now and see where the theories land.

Until then, rest in peace, Dimitri. Know you changed the way I think.

The following initially appeared as The Role of Convergence Analysis on June 10, 2016 at the old argmin blog.

This year marks the retirement of Dimitri Bertsekas from MIT. Dimitri is an idol of mine, having literally written the book on every facet of optimization. His seminal works on distributed optimization, dynamic programming, and Lagrangian methods remain the best references available. I had the privilege of taking Dimitri’s convex analysis course in grad school, and he would frequently burst into class beaming because he had stayed up until 2AM the night before simplifying an argument of Rockafellar’s down to elementary calculus.

My last post on Lagrangians was based on Chapter 3 of Dimitri’s Nonlinear Programming Book. Chapter 2 also happens to feature one of my favorite passages about the delicate balance between theory and practice in optimization. One of the trickiest parts about optimization (and a point I intend to repeatedly hammer on this blog) is realizing how many of the theorems are “qualitative” rather than “quantitative.” I wanted to just quote Dimitri’s text in full here, as I don’t think I could write it better. Best wishes to you in retirement!

The Role of Convergence Analysis by Dimitris Bertsekas

The following subsection gives a number of mathematical propositions relating to the convergence properties of gradient methods. The meaning of these propositions is usually quite intuitive but their statement often requires complicated mathematical assumptions. Furthermore, their proof often involves tedious ϵ−δ arguments, so at first sight students may wonder whether “we really have to go through all this.”

When Euclid was faced with a similar question from King Ptolemy of Alexandria, he replied that “there is no royal road to geometry.” In our case, however, the answer is not so simple because we are not dealing with a pure subject such as geometry that may be developed without regard for its practical application. In the eyes of most people, the value of an analysis or algorithm in nonlinear programming is judged primarily by its practical impact in solving various types of problems. It is therefore important to give some thought to the interface between convergence analysis and its practical application. To this end it is useful to consider two extreme viewpoints; most workers in the field find themselves somewhere between the two.

In the first viewpoint, convergence analysis is considered primarily a mathematical subject. The properties of an algorithm are quantified to the extent possible through mathematical statements. General and broadly applicable assertions, and simple and elegant proofs are at a premium here. The rationale is that simple statements and proofs are more readily understood, and general statements apply not only to the problems at hand but also to other problems that are likely to appear in the future. On the negative side, one may remark that simplicity is not always compatible with relevance, and broad applicability is often achieved through assumptions that are hard to verify or appreciate.

The second viewpoint largely rejects the role of mathematical analysis. The rationale here is that the validity and the properties of an algorithm for a given class of problems must be verified through practical experimentation anyway, so if an algorithm looks promising on intuitive grounds, why bother with a convergence analysis. Furthermore, there are a number of important practical questions that are hard to address analytically, such as roundoff error, multiple local minima, and a variety of finite termination and approximation issues. The main criticism of this viewpoint is that mathematical analysis often reveals (and explains) fundamental flaws of algorithms that experimentation may miss. These flaws often point the way to better algorithms or modified algorithms that are tailored to the type of practical problem at hand. Similarly, analysis may be more effective than experimentation in delineating the types of problems for which particular algorithms are well-suited.

Our own mathematical approach is tempered by practical concerns, but we note that the balance between theory and practice in nonlinear programming is particularly delicate, subjective, and problem dependent.

Subscribe now