Hi there, argmin readers! As the fall semester picks up, posting volume will, too. So I’m going to commit to writing short descriptive headers to help you sort through the different threads.

Many readers have asked me to write about AI companies’ conquest of mathematics. Today’s post is a first, but by no means final, attempt at grappling with our new mathematical condition.

Early in my career, I was fortunate to get caught up in a fascinating research frenzy at the intersection of pure and applied math, the compressed sensing gold rush. Compressed sensing asked whether signals could be compressed at the time of measurement. Rather than sampling an image with a high-resolution camera and then compressing it to a JPEG, could we collect a number of samples equal to the number of bytes in the JPEG? Compressed sensing rested on deep mathematics from geometric functional analysis, convex geometry, and probability theory. It yielded multiple engineering artifacts, from faster MRI capture times to better systems for content recommendation.

Though we can see the influence of the field across many applied domains, the math of compressed sensing was never decidedly prescriptive. The theorems always assumed things about reality that couldn’t be verified or required measurement systems that were too costly or impractical. Yet the math of compressed sensing helped us focus on a shared narrative of design principles. It helped us design new algorithms. It helped us construct new measurement schemes that were robust to noise. It helped us map out which other system structures were amenable to compressive techniques. Pure math gave us a frame to see what was possible.

While this mathematical formalism was unreasonably effective, it came with a decidedly unhealthy downside. Shahar Mendelson best described this general problem of applied pure mathematics in a talk he gave at COLT 2014. Applied mathematicians often need to build a giant scaffolding of mathematical modeling to solve a problem. This scaffolding creates new mathematical puzzles that aren’t directly connected to the original problem of interest, but that entice problem solvers. You’ll then see dozens of follow-up papers solving the puzzles but forgetting the problem we cared about in the first place.

This is open problem culture, and it’s corrosive. It leads to trophy hunting, where people race to scoop each other, consult expert friends for secret insights, or steamroll each other with ever more complicated math.

This fetishization of puzzle-solving as genius has long been a destructive tendency in mathematics more broadly. It’s easy to get caught up in the thrill of it. Mathematics is arguably the most meritocratic academic discipline. There are set problems, and the people who solve them are the smart ones. Everyone forgets that the only reason problems confer status is that (a) they are currently unsolved and (b) enough mathematicians have decided these are worth solving. That (b) part is not meritocratic.

This is why many are confused and angry at the practicing mathematicians who try to explain that the discipline of mathematics is about understanding, not proving stuff. To many observers, even those who strive to become mathematicians, math seems set up as a competition from the get-go. It’s rote testing all the way up through college. Ace the SAT as a 7-year-old. Win the IMO gold as a 14-year-old. Max the Putnam Exam as a 19-year-old.

Your reward is the permission to work on whatever puzzles you want, without questions, for the rest of your life. There is no requirement for the winners to explain anything. Maybe they have to teach calculus, but they don’t have to do a good job at it.

From the outside, you can see why people think mathematics is just about winning those competitions and proving what is true. Math doesn’t send many outward signals that “understanding” is a core part of the pursuit. Most people see math as a quiz show culture. Math culture is ruthlessly competitive, and it makes a lot of people feel stupid.

The actions of many notable mathematicians have only lent credibility to their critics. Wars over credit and who gets there first have now ruined two of Clay’s Millennium Problems. This will have to change in light of recent events with AI companies solving math problems few thought they’d be able to. When computers do something we think they wouldn’t, the reaction should not be writing insanely long posts about how Eliezer Yudkowsky was right and the machines are going to kill everyone. Instead, we have to adjust our reference narrative about what we thought was true.

Indeed, I didn’t learn anything about fluid dynamics from OpenAI’s proposed solution to the Clay Millennium Prize Navier-Stokes problem. This problem is exactly the sort of puzzle artifact that I lamented above. The resolution of the Navier-Stokes problem itself tells us nothing about the dynamics of fluids that the equations attempt to model.

That said, I’ve learned a lot from the supposed resolution. I learned that the jump from rote IMO solving to the Millennium Prizes was much shorter than I expected. If you build an algorithm that’s good at solving IMO problems, and you present it with the right ingredients and computational resources, you can solve hard math problems too. That is, a lot of mathematics is training people to benchmaxx. We have already created a battery of tests, carefully tuned with the best psychometrics to find mathematical genius. Training computers to maximize those benchmarks ends up solving the benchmarks. What are millennium problems other than humanity’s final math exam?

This unfortunately makes a lot of sense with the benefit of hindsight!

If this is the lesson, there’s a funny takeaway. While it feels like you need to be an IMO prodigy to set foot in the mathematical arena, being a great IMO solver doesn’t mean you’ll become a great mathematician. For that, you need to bring other talents to bear. Despite the efforts of many smart and caring people, those talents remain ineffable. They certainly aren’t benchmarkable.

In an age of the decidedly anti-intellectual culture of artificial intelligence, mathematicians, both pure and applied, need to keep working to articulate what on earth those talents are. The statements so far, describing how mathematical programs are more than the truth values of their associated theorems, are a good start even if they are not met with universal acclaim. More need to chime in with stories about how mathematics, even the very pure variety, is valuable for scientists, engineers, and everyone else.

I can describe my own experience. Though I’m much less concerned with proving theorems than I was earlier in my career, I still consider myself an applied pure mathematician. Applied mathematics is a formal language that bridges two unbridgeable worlds. Mathematics is a deductive practice that combines axioms via a set of well-specified rules to generate lemmas, theorems, and corollaries. Empirical science and engineering are inductive. We confirm theories when they make correct predictions, willfully committing the logical fallacy of affirming the consequent. This does not make science wrong. It just means, as David Hume told us three hundred years ago, that mathematics can’t justify science.1

Applied mathematics is thus a logical language for describing inductive processes. It’s, um, unreasonably effective at this task. As captured above in my discussion of compressed sensing, it can never perfectly specify what you should do in practice. Instead, it acts as a form of linguistic technical drawing, allowing communities of scientists to build complex theories and engineers to build complex systems. Pure mathematics gives applied mathematicians new pens and brushes for those drawings.

This is why I like (and have been using throughout) Jordan Ellenberg’s term applied pure mathematics. Applied mathematics often just means the mathematics of partial differential equations. Applied pure mathematics is any application of any mathematics to anything outside of the closed world of mathematics itself. You never know which weird corner of the vast libraries of “apparently useless” mathematics will help you make sense of reality.

Let me give an example of unexpected brushwork from my time in the compressed sensing gold rush. Did I need to learn p-adic analysis as an undergrad? Maybe not, but it fixed a set of regularities and patterns in my head. I remembered Bochner’s theorem on locally compact abelian groups when Ali Rahimi and I were trying to make sense of our code generating random features. This turned into a very cool paper with a lot of practical impact. The web of facts I had gathered sitting through weird courses and reading esoteric math books shaped how I saw this applied machine learning problem. AI could likely make that connection today, but my personal education is still needed to create the prompt.

In the first lecture of my first college math course, the legendary Chicago Professor Paul Sally (IYKYK) barked that he wasn’t there to teach us facts, but to fix our brains. Sally dedicated his career to mathematics education, passionately broadening the conception of who could be a mathematician. Math wasn’t a competition for Sally. It was a way of seeing. It still can be, even if our computers now outcompete us.

Subscribe now

Hi there, argmin readers! As the fall semester picks up, posting volume will, too. So I’m going to commit to writing short descriptive headers to help you sort through the different threads. Today’s post is a live blog of Class 7 of my graduate seminar “Forecasting: A Critical Retrospective.” The syllabus and list of past posts is here.

In Kathryn Schulz’s New Yorker article, “The Really Big One,” she often cites figures about the chances of earthquakes.

“[T]he odds of the big Cascadia earthquake happening in the next fifty years are roughly one in three. The odds of the very big one are roughly one in ten.”

She described how these numbers arose by counting historical events and turning them into probabilities of the future. In the first week of class, we discussed this alchemy for clear discrete events. Coin flips, free throws, or elections have known times at which they occur. Only their outcome is uncertain. When we try to predict time, we need a new protocol: survival analysis.

Survival analysis is something I learned (and I think most people learn) in medical statistics. The name kind of gives it away: who do you think is surviving here other than patients in medical studies? In medicine, survival analysis captures the proportion of individuals still alive after a potentially life-extending treatment. Or, just as commonly, we flip this around and ask what proportion of subjects have not experienced a bad event yet.

In a randomized clinical trial, all patients start their timers at the same time—when they are randomized. The trialists gather the times from randomization until the bad events, and then estimate a probability distribution on the time until an event occurs. This distribution is over times, and is specified by a curve that models the chance the time to a bad event is greater than T. For example, this could be a distribution of how long it takes for cancer to progress under some new treatment. Or it could be how long until someone contracts an infection in a vaccine study. A survival curve lets you make probabilistic forecasts. For every time, you can look at the estimated proportion of individuals who have not yet experienced a bad event and call that the prognosis. “90% of patients experience no bad outcomes in the year following treatment.”

Here’s the most famous survival curve of all time. Who remembers this one?

These curves are estimated using a nonparametric method called the Kaplan-Meier estimator. The Kaplan-Meier curves give a rough shape of the survival distributions and let clinicians compare the relative effectiveness of treatments. When the treatment and control curves are far apart, it suggests something meaningful differs between the treatment and control conditions.

We can apply the same survival analysis ideas to other time-to-event forecasts. Let’s caricature how we might do it for earthquakes. For a single fault, you can imagine a major earthquake as a “reset” of the tension in the earth. Each earthquake gives you a time to start counting until the next one. If we assume every earthquake follows identical geodynamics, we can treat each earthquake like a patient in a trial, now estimating a survival curve for the time to the next earthquake. This is a crude model, as it assumes a total reset of conditions, but it’s a starting point.

Once we have this model, our historical record gives us a path to estimate the survival curve and forecast the probability of an earthquake in the next T years. Although we could build a Kaplan-Meier curve here, we could also pose an explicit model of how the probability changes over time and fit the parameters.

Any probability distribution over nonnegative numbers can serve as a model for the time to event and thus be turned into a survival curve. A common distribution in earthquakes is the exponential distribution. That is, the model is that the probability that a new major earthquake happens within T years after the first one is:

The nice thing about the exponential distribution is that it only has one parameter to estimate from data, and the maximum likelihood estimate is super simple. It’s

This formula gives us a straightforward program. Look at the historical record and compute the average waiting time between events, W. If you want probability forecasts over time windows, treat this average time as the inverse of the parameter of the exponential distribution. In this model, the probability that there will be a new event in T years is

This model is too simplistic, but it’s the first back-of-the-envelope calculation people do, and it’s where all the figures in Schulz’s New Yorker article come from.

This exponential survival model is the same as modeling earthquakes as a Poisson process. You can get fancy and make your model more sophisticated to capture more physical reality, specializing parameters to the particulars of each fault. You can model the survival function with some other distribution, be it log-normal, Weibull, or whatever. However, every modeling assumption you make adds more parameters to fit from data, and earthquakes don’t occur frequently enough to fit that many parameters to reasonable precision. If you have only forty events, you should probably estimate only one parameter.

Whatever modeling you do, survival analysis gives us another apparatus for turning counts into chances. How precise you think those chances are now rests on a whole lot of untestable modeling assumptions. What is the chance those assumptions are wrong?

Subscribe now

Hi there, argmin readers! As the fall semester picks up, posting volume will, too. So I’m going to commit to writing short descriptive headers to help you sort through the different threads.

Today’s post is by Jessica Dai, writing her second post on the microconference on “public feedback for AI and beyond” that she ran at UC Berkeley in August. -Ben

Today, I’ll continue blogging readouts for the AI & public feedback workshop. As a reminder, here’s some of my motivating logic:

Proposition 1. “The public” has interesting and important things to say about their experiences with AI, but are not typically listened to by decision makers.

Proposition 2. “Evaluations” — and aggregated information, more broadly — are useful, in the sense that they can influence consequential decisions.

Corollary. AI evaluations from public feedback can be a meaningful way to “do something” about the emergent misalignment between those who control AI development and literally everyone else.

In the first post about the workshop, I wrote about who, or what, “the public” refers to. Today, I’ll continue with “Proposition 1,” and try to reason through some of the challenges in the process of actually providing feedback.

Easing the burdens of “participation.”

What does a participant experience in the process of sharing information? As an economist might say, engagement is ‘costly’ — this is why, for instance, human subjects studies typically compensate participants, and why the “representativeness” of the people who self-select to participate in light of these costs remains a central challenge (I discussed some of these issues in the prior post).

But costs can manifest concretely in ways that are difficult to quantify by economic measures. Moreover, they are not only about the initial decision to participate; the process of participation itself entails challenges that can affect the outcomes of data collection. For example, Samantha Dalal, the researcher with WAO, emphasized the importance of understanding the barriers to participation that might be specific to the relevant “slice of the public.” If data will be collected via a mobile app, it should be available on a variety of platforms (including older versions of Android and iOS), and small enough to be feasibly downloaded to a phone with limited storage or on limited cellular data; online forms should be readable on mobile browsers. While these factors may seem like basic design fundamentals, they are also a reminder of the friction inherent to collecting “real” data.

Some of this friction also involves active support from facilitators that shapes the feedback itself. Some of the preliminary findings in Humphrey Obuobi’s presentation about BLOOM’s work in central Oregon were results from a Polis-like platform, where participants could provide statements of their own positions on various topics as well as engage with previously-written statements (for example, by indicating support or disagreement). Some workshop attendees noticed that these statements varied widely not just in content, but style, with some statements being noticeably longer, more detailed, and having more complex sentence structure. Humphrey explained that BLOOM uses a mix of participant-generated and facilitator-written statements in the deliberation process; the latter can synthesize existing participant-generated positions, while also help support participants to develop finer-grained perspectives.

Both of these are examples of ways that facilitators can be actively involved in the process of collecting feedback, rather than passively waiting for data to arrive; they also suggest that this involvement can ultimately result in higher-quality feedback. Another lens for thinking about these examples is legibility. If data collection platforms are poorly designed, then there will be members of the public who are “illegible” to facilitators; meanwhile, the facilitator-written statements are directly increasing the “legibility” of participants’ original statements.

Pursuing and enabling legibility in this way feels important; why? Ben’s talk provides one conceptual answer. He spoke about the quantification trap, wherein the demand for legibility is the first in a series of dominoes that ultimately requires power to be enacted only through “objective” numbers, and conversely, endows numbers (“objective,” or otherwise) with power. I’ll discuss the latter part of this statement in a later post, but I want to highlight one of Ben’s arguments (really, Graeber’s) that his post glosses over: One vector through which people experience “structural violence” is that they must work to make themselves legible to the decision makers who hold power over their lives. Any illegibility, or irregularity, excludes them from bureaucratic accounting; in Graeber’s account, this can be dangerous and, in the worst case, subject them to material harm.1

When viewed from the perspective of legibility, therefore, the various ways in which facilitators can make participants’ lives easier is also a way to prevent their exclusion from the evaluation, and whatever future conversation this evaluation enables. Some people might find it otherwise difficult to make themselves or their feedback legible, and while their resulting exclusion might not result in anything as dramatic as material danger, it still feels worthwhile for facilitators to ease whatever burdens participants face to make themselves heard.

The costs of legibility.

It is not lost on me that there are also costs to legibility. For instance, many of the projects that involve analyzing transcripts can be fairly invasive from a privacy perspective, even when transcripts are donated voluntarily, and analysis does not rise to the level of privacy violations. Similarly, any “monitoring” approach, whether it’s of product usage or of social media (as in my r/ChatGPT paper), also essentially amounts to surveillance that subjects users to a level of scrutiny they may not feel entirely comfortable with. One of the other recent California bills that Deb Raji discussed was SB243, which requires chatbot providers to track and disclose the number of conversations that mention suicide or self-harm; while requirements for such disclosures seem important for accountability and transparency, there are obvious privacy tradeoffs as well.

Perceived discomfort seems to matter. Evi Micha, from USC, has done a lot of theoretical and algorithmic work on ensuring representativeness in citizens’ assemblies (e.g., demographic, regional, etc.). Representativeness might reasonably be thought of as a necessary precondition for such assemblies to produce high-quality discussions that can be effective proxies for viewpoints of the population as a whole. In her talk, she shared findings from recent work with a different methodological perspective: how do participants actually perceive “representativeness” in assembly selection? Perhaps unsurprisingly, people generally prioritized representation in the sense of political or issue-based agreement; they would be happiest to be represented by someone who shared their views on the topics to be discussed.

What was more surprising was the degree to which people seemed to hate the idea of representativeness measured via demographic attributes. Evi shared some of the free-text commentary from study participants, and I can’t over-emphasize how strongly negative this feedback was. Participants seemed to take offense at the very idea that demographics might have any correlation, and therefore relevance, to their views on substantive issues. While we know that demographics and political views often do actually correlate on the population level, it seems like it’s worth considering how it feels to an individual for their worthiness as an assembly participant to be reduced to immutable demographic characteristics.

There’s of course a bit of a chicken-and-egg problem, in the sense that it would be difficult for any facilitator to select assemblies that were fairly representative of issue-level perspectives before even knowing what those perspectives might be or how they might be distributed across the population. It’s therefore understandable that demographics, which are easily “legible” a priori, become the fallback mechanism for ensuring “representativeness,” but this cheap legibility might be exactly what participants are chafing against.

The necessity of legibility.

In some cases, it might be necessary to explicitly impose the burden of legibility on participants, especially when the goal is to shift power to them. Ira Globus-Harris spoke about their work designing a “bias bounties” mechanism for auditing a deployed machine learning model. Specifically, the mechanism allows any population subgroup that perceives the deployed model to be inaccurate on them to submit a “bounty”; this brings the model developers’ attention to performance on that particular subgroup, allowing developers to iteratively improve the model in the future.

This framework is compelling, because it effectively identifies the power in the public as due to knowledge about their own experiences (i.e., groups of people can determine when the model is inaccurate on them specifically), and leverages that knowledge while also recognizing that only model developers have the power to actually change the model itself. The catch is that, for any given subgroup, model developers can only improve the model for that subgroup if it is statistically possible to do so. (One major takeaway from the fair machine learning research of the late 2010s is that what often appears as “bias” is often actually due to “variance” — it can be inherently harder to predict Y from X in some subgroups — so for a fixed measure of performance there may be subgroups for which that measure can never improve, no matter how complex the underlying model.)

Ira’s mechanism therefore requires that a submitted “bounty” for a given subgroup includes not just a statement that the model performs poorly on that subgroup, but also evidence that it is even possible to do better on that subgroup. This makes sense, because no model developer, no matter how benign, can do better than what is statistically achievable; on the flip side, as long as improvement is possible, “bounties” of this form allow the model developer a straightforward algorithmic approach to incorporate the reported information to implement the improvement. On the other hand, this also asks a lot of potential “bounty hunters”, including, perhaps, collecting their own data and training their own model — a requirement that might well be practically infeasible.

In this case, the mechanism specifies exactly what it means to be legible, without explicitly providing a pathway for participants to meet those criteria. Even so, I want to emphasize that the specification itself is, already, an invitation to the public. Since the goal is to change the deployed model, participants must share feedback in a way that can be legible to the model developer. The specification of what counts as “legible” is a starting point for helping participants to be seen the way they want to be seen — and ultimately, for making it clear that it is their voices we want to hear in the first place.

Subscribe now

Hi there, argmin readers! As the fall semester picks up, posting volume will, too. So I’m going to commit to writing short descriptive headers to help you sort through the different threads. Today’s post is a live blog of Class 6 of my graduate seminar “Forecasting: A Critical Retrospective.” A table of contents is here.

Given the week’s events, it’s a bit unfortunate that I scheduled our discussion of p(doom) for the last week of class. I predict AI won’t have killed us by then, and the real question is whether we’ll all be bored to tears discussing the topic in November. But the agenda for today, earthquakes, is a good preview for the challenges associated with quantifying uncertainties about catastrophe. Seismologists don’t think an earthquake will lead to human extinction, but it can cause massive casualties and damage. How do we quantify our predictions of whether an earthquake will happen? And then what do we do about it?

Most experts agree that predicting the exact time and location of earthquakes on long time horizons is impossible. The dynamics of the Earth moving, building up stress, and slipping are far too complicated to predict with any reasonable granularity using differential equation models. Earthquake forecasting couldn’t be further removed from weather forecasting in that regard.

At best, we can make coarse predictions based on a mix of temporal and spatial localization. Earthquakes tend to occur near fault lines. Fault lines have a history of previous ruptures of different sizes. Using these data, we can estimate rough statistical models. You might naively estimate an exponential recurrence time: the rate at which earthquakes occur is just the count divided by the observation window. In an exponential model, the expected time to the next earthquake would be the inverse of this number. A slightly more complicated formula then gives you the chance of an earthquake in the next decade.

chance = 1 - np.exp( - rate * time )

Such primitive models are not precise, but they are helpful. What do you do with these probabilities? You can turn them into general warnings. If you expect a certain frequency of shaking, you should build infrastructure that can withstand it and teach people how to prepare for the disruption the next one will cause. If you know big earthquakes occur every few decades, that’s enough to inform planning and insurance.

But nailing down the probability of an earthquake, even to one decimal place, is a fool’s errand. Our first reading of the week, Freedman and Stark’s classic paper “What is the Chance of an Earthquake?”, highlights the futility of precise probability models. If you want to validate a probabilistic forecast, you need a lot of events. The law of large numbers needs a lot of numbers! Large earthquakes are rare. Probabilistic models can’t be tested on human time scales. Moreover, when you add more geological reality to your model, you introduce a variety of hard-to-estimate parameters and researcher degrees of freedom into the equations. Every new modeling assumption introduces new unidentifiable parameters. More realistic doesn’t mean better estimates.

If you want to predict really big earthquakes, like those with magnitudes greater than 8.5, then we have an even sparser record. The old-fashioned AI chatbot, Wikipedia, has dozens of tables listing earthquakes by all sorts of characteristics. It lists only 17 of these in the past hundred years. Scientists have developed techniques to infer the occurrence of giant earthquakes thousands of years in the past. These tend to give noisier estimates of recurrence times, but sometimes they yield very ominous predictions.

One of the most ominous is in this week’s reading, “The Really Big One,” a riveting 2015 New Yorker article by Kathryn Schulz. Schulz reports on the Cascadia subduction zone, a thousand-mile fault that runs from Northern California to Vancouver Island. Combining oral history, Japanese tsunami records, and tree rings, seismologists determined that a massive earthquake, with a magnitude pinned between 8.7 and 9.2 on the Richter scale, happened on this fault on the evening of January 26, 1700. It killed coastal forests of the Pacific Northwest and created a massive tsunami in Japan. Oral histories from First Nations tell of entire communities vanishing. Scientists have gone back to geological samples and counted 41 major earthquakes on this fault in the last ten thousand years. Using the rough rule of thumb, we should expect a major, destructive earthquake once every 243 years. It’s been 326 years since the last one.

Now, you could try to guess the probability that an earthquake occurs on this fault before 2050, but that number doesn’t really do much of anything for you. We don’t know when it will occur, but we know an earthquake is inevitable here, and we know it will be catastrophic.

Shulz details some predictive horror stories of what will happen when the next big one hits the Cascadia Subduction Zone. It does seem like a bad idea to put millions of people near such a seismically volatile region. But this is the problem with our slow ape brains. As Shutz writes, “[forty] years ago, no one knew that the Cascadia subduction zone had ever produced a major earthquake. [Fifty-five] years ago, no one even knew it existed.” In 1970, Seattle was already a major city with over half a million people.

So the question is, what do we do now? The low end of state estimates of fatalities from the next major earthquake is in the tens of thousands. One answer would be to move millions of people away from the danger zone. No one is proposing this. The other is to build as much infrastructure as possible to handle the incoming crisis through seismic retrofitting and social infrastructure for tsunami evacuation protocols and earthquake preparedness. The work involves building systems to keep damage as small as possible, even though the damage will be unavoidably large. As Freedman and Stark say, “probabilities are a distraction.”

Subscribe now

Hi there, argmin readers! As the fall semester picks up, posting volume will, too. So I’m going to commit to writing short descriptive headers to help you sort through the different threads. Today’s post is a live blog of Class 5 of my graduate seminar “Forecasting: A Critical Retrospective.” A table of contents is here.

The fun thing about teaching a new class is realizing in week three how your sequencing is already off. I do this every year, so I’m no longer surprised. At this point in my career, I’d put the probability that I’ll be disappointed with my syllabus before the official add-course deadline at 95%.

How did I get to that number?

So yes, though it’s not directly about forecasting, the first mathematical lecture of this class should have been about probability. Because probability is unavoidable in forecasting, and it is used far too casually for my tastes. We saw this in the weather example where outputs of data assimilation were interpreted as “probability distributions” over the state of the atmosphere. Samples from this distribution were used to create a sample of future possibilities. Frequencies of these future possibilities were turned into beliefs about whether it will rain on Sunday.

All of these probabilities are sort of different, right? Some are about counts, some are about beliefs, and somehow we move between the two as if there is a well-specified set of formal rules for doing so.

I think every class on applied probability, including the undergrad ones, should call out this slippery transmutation between frequency and belief, and I’m particularly fond of the development in Paul Meehl’s course on Philosophical Psychology.1 Meehl follows Rudolf Carnap, who explicitly distinguishes the two kinds of probability.

Probability 1, sometimes called logical probability, relates propositions with beliefs. It quantifies how much credence we should give a hypothesis based on the observations and facts laid before us. Probability 1 represents the certainty of a logical proposition being true, the doubt that something in the past happened, the likelihood that future events will occur, or a person’s internal beliefs about the world.

Probability 2 concerns frequencies of events. It more or less just amounts to counting, measuring the relative frequency of some property in a set of objects. Probability 2 might quantify the relative frequencies of occurrences that currently exist in the world, but it can also describe the relative frequencies of hypothetical infinite populations.

Probability 2 is usually the one we start with and teach and never problematize, and then we casually jump to using Probability 1 in our thinking, writing, and experimenting without realizing it. This is because they both obey the same axioms of probability.

Let’s say I have a bunch of marbles in an urn. I want to describe the frequencies with which certain properties of those marbles hold. The following things are true for any subset of marbles:

1. Any subset has proportion greater than or equal to 0

2. If I take the entire set, the proportion equals 1.

3. If I take two nonoverlapping subsets, the proportion of their union is equal to the sum of their proportions.

These are Kolmogorov’s axioms of probability. Item 1 is nonnegativity, Item 2 is unit measure, Item 3 is additivity.

Probability 2 obviously obeys Kolmogorov’s axioms, but what about Probability 1? Well, we can sort of force it to have those properties too.

1. Any syntactically valid statement has probability greater than or equal to zero.

2. Any statement that is certain or tautological has probability 1.

3. If two statements describe mutually exclusive outcomes, then the probability of either one or both of the outcomes is equal to the sum of the individual probabilities.

Lo and behold, statements of belief seem to also obey Kolmogorov’s Axioms. These assertions might feel a bit less obvious and certain than marble counting. You can’t get a tangible grip on why beliefs should obey Kolmogorov’s axioms because beliefs live in your head. You can check the properties of Probability 2 on a set of marbles. You can never check whether the axioms work for Probability 1.

But there are strategic reasons for using probabilities when quantifying beliefs. In last Thursday’s lecture, we saw that if you are scored using the Brier Score, then when your forecasts don’t obey the axioms of probability, there is always another set of forecasts that achieves a higher place on the leaderboard. If instead of being a forecaster, you’re a degenerate gambler, there are arguments about dealing with bookies that motivate the same probabilistic rules. As Dennis Lindley put it, probability is inevitable once you try to quantitatively evaluate belief.

I think we should also distinguish a third kind of probability, Probability 0, to describe the formal language of mathematical probability. This is probability whose referent is neither frequencies nor beliefs but mathematics itself. In this case, we’d consider the following to all be Probability 0:

1. Any finite list of numbers that is nonnegative and sums to one

2. Any infinite list of numbers that is nonnegative and whose infinite sum converges to one

3. Any nonnegative function on the unit interval whose integral is equal to one

These sorts of mathematical structures arise a lot when we’re doing calculations. And whenever people find a convenient mathematical application of Probability 0, they tend to find a convenient application of that structure in Probability 1 or Probability 2. We see Probability 0 everywhere. It only becomes metaphysical when we attach some meaning to it off the chalkboard.

When we build computational forecasts, we have to play with all three kinds of probability. We saw this last time: We need Probability 2 because the best predictions correspond to the rates of outcomes in similar future events. We need Probability 1, or else our forecasts are incoherent. We need Probability 0 to write and reason about algorithms that analyze noisy data. But how we tie the three together is not given. There is no god-given algorithm of prognostication that we can derive from Kolmogorov’s axioms, and techniques and conventions vary between disciplines. Which scoring rule are we using? Are we insisting on building calibrated forecasts? What do our forecasts do? These questions determine how we work with probability.

Subscribe now

Hi there, argmin readers! As the fall semester picks up, posting volume will, too. So I’m going to commit to writing short descriptive headers to help you sort through the different threads. Today’s post is a live blog of Class 4 of my graduate seminar “Forecasting: A Critical Retrospective.” A table of contents is here.

Our first modern case study is weather forecasting. This is one of forecasting’s greatest success stories, and it’s worth pulling apart how it came to be. Max Raginsky pointed me to a fun passage in Game-Theoretic Foundations for Probability and Finance by Shafer and Vovk, noting that in the 19th century weather forecasters were called “weather prophets,” and only in the 20th century did they change their name to claim an air of scientific authority. Forecast sounds more authoritative than prophecy or prediction.

Indeed, the main turn in weather forecasting in the 20th century was toward physics. American Cleveland Abbe and Norwegian Vilhelm Bjerknes proposed using the laws of physics to forecast the weather, just like astronomers do to forecast the future positions of planets. After all, at its core, the atmosphere was just a giant ensemble of gas and fluid. If we could measure the initial state of all of the particles, we could run Newton’s Laws forward in time and exactly predict the weather for the rest of time.

Now, though they were strong believers in determinism, Abbe and Bjerknes were not that naive. They knew that they’d need to lean on thermodynamics and fluid dynamics. And they knew that even at these higher levels of abstraction, solving the differential equations by hand was out of the question. But they proposed a reasonable program: (a) measure the current state of the atmosphere to as high a precision as possible, (b) use whatever computational means possible to run a physics model forward in time. This is what we still do today.

Obviously, computation is key. From the beginning of modern computing, weather forecasting has been one of the driving frontier applications. It is an ideal application for hyperscaling because computers are never big enough to give us the global precision needed to predict whether I’ll need an umbrella two weeks from now. Weather forecasting was one of von Neumann’s favorite application problems for computers, and some of the earliest modern forecasts were demonstrated on the ENIAC in 1950. With each generation of new computers, our forecast horizon improves, to the point where 3-day forecasts are now remarkably prophetic.

Here’s a chart of the current skill of high-resolution weather forecasting. The y-axis is the “Anomaly Correlation Coefficient”, which measures the correlation between a forecast atmospheric condition and the measured deviation from the seasonal average. From 1985 to 2020, we gained about one day of forecast accuracy every 10 years. This is a remarkable success story of computational scale. In 100 years, with multiple doublings of computer power, we turned a curious scientific pipe dream into a global predictive infrastructure.

However, those gains look like they’re plateauing. Part of what makes weather forecasting so interesting is that we can predict out a few days with striking accuracy using global-scale measurement infrastructure and supercomputers. But it might only be predictable to a certain point.

Lorenz famously demonstrated that simplified weather models were chaotic, meaning two nearby trajectories diverge exponentially quickly over time. Weather models empirically display a similar property, and any small initial measurement uncertainty means that there will eventually be huge forecast uncertainty. The exact time scale of what is predictable isn’t clear, but progress on 10-day forecasts does look a bit stuck in the graph above.

One thing I find fascinating is how this uncertainty from a deterministic equation becomes probabilistic. Chaos is not randomness. Completely deterministic equations exhibit the “diverging trajectories” phenomenon. You can run fun simulations with the logistic map:

x[n+1] = 3.9 * x[n] * (1-x[n])

The output sequence will look like random noise, and two close initial conditions quickly end up in completely different places after a few steps. No random number generators are required.

So where does the “chance of rain” come from? It’s a multi-step process. Probability enters because, despite global investment in measurement, we can’t perfectly nail down an initial condition to start our weather simulation. Measurement quality is better where population density is higher, as populated areas are where it’s easiest to put weather stations. However, you need high resolution everywhere to have a perfect model, and there’s still very uneven coverage. And the measurements themselves of course have inaccuracies. Backing out the state of the atmosphere from the measurements we have is not an exact formula.

So weather forecasters use estimation algorithms, including Kalman filtering techniques, to compute probability distributions of the current state of the weather from the best available measurements. I haven’t found a good discussion of why these probabilistic methods are preferred or how we should interpret the associated probabilities, but this is where probabilities enter the forecast: they use probabilistic tools to translate measurement and modeling uncertainty into a generative probability distribution of initial conditions. They can sample from this distribution and run several simulations simultaneously, thus producing a few dozen candidate “samples” of what the future will look like.

With these samples, forecasters can then count frequencies of events in the sample. If it rains in 40 out of 50 samples, they say “the chance of rain is 80%.” Is this a valid probability? Not really, because the modeling assumptions introduce all sorts of biases. So weather agencies adjust probabilities based on past events so forecasts are calibrated.

We’ll talk more about calibration on Thursday. For forecasters, they’d like you to interpret this roughly as “in all of the historical records when the atmospheric conditions were like this, precipitation was observed 80% of the time.” A forecast is calibrated if the rates in the historical forecast match the rates in the historical observations. A calibrated forecast means that it rained on 80% of the days when the forecast chance of rain was 80%. Similarly, it only rained on 20% of the days when the forecast chance of rain was 20%. Calibration is much weaker than the forecast skill plotted above. If it rains on days starting with T and you always predict a 28.6% chance of rain, your forecast is calibrated but missing the forest for the trees. Still, calibration is a nice thing to have in a weather forecast because it pins down what the forecaster means by chance of rain. Whether this interpretation of probability has any profound effect on your life is uncertain.

Subscribe now

Hi there, argmin readers! As the fall semester picks up, posting volume will, too. So I’m going to commit to writing short descriptive headers to help you sort through the different threads. Today’s post is a live blog of Lecture 3 of my graduate seminar “Forecasting: A Critical Retrospective.” A table of contents is here.

Since the Great Depression, US law has required financial management companies that offer products like mutual funds to add a disclaimer to all of their advertising:

The thing is, not a single person believes this. The fund managers don’t believe it, and neither do the customers. Why would you buy a mutual fund if you didn’t think its past performance told you something about how much you’ll have upon retirement? We tend to believe that some investments are riskier than others and some managers are more reputable than others. These beliefs are based upon past observations, and we use them to inform our investment decisions.

So what do we need to do to transform past observations into forecasts? We believe that the past can’t perfectly predict the future. We also believe that a forecaster is only as good as their track record. In today’s class, we’ll link these two together, showing how the evaluation metric for forecast track records leads us to particular forecasting algorithms.

Let’s start with the two main examples from Edmond Halley. We believe the past strongly predicts the future when talking about the motions of celestial bodies. We believe it far less when pricing individual insurance policies.

For his comet, Halley paired three observations together using insights about orbital shapes from Isaac Newton. Given the roughly 76-year gaps between these observations, he predicted we’d see the same object again 76 years later. In 1835, by the time we had seen Halley’s comet twice more, astronomers were uniformly convinced Halley was right and were certain we’d see the comet again in 1910 and 1986 (they were proven correct).

For his life table, Halley grouped people by age and used these cohorts to make demographic forecasts. The proportion of the population aged 25 was 1.668%, and the proportion aged 26 was 1.647%. Therefore, he concluded the odds a 25-year-old would live to see 26 were approximately 80 to 1 in favor.

The move in the demography example to consider odds and chance is interesting, and was something in the air at the time. Proto-demographer John Graunt had made similar calculations of hazard and risk in his tabulations thirty years earlier. Probability applied to casino games had only begun to be formalized forty years earlier. The transmutation of frequencies into risk was intuitive once you started assembling databases. Now we just take it for granted, having created a formal structure that hides the intuition.

In today’s lecture, we’ll work out some of the formalism of this map from rates to risks, deriving the mathematical formulas that encode our assumptions. If you assert that a forecaster will be evaluated on their track record, and if you believe that events are effectively the same, then you bind yourself to making future predictions a deterministic function of the observed rates. The assumptions here are usually implicit. We’re assuming a strong level of interchangeability between past and future events with a particular signature. And we tend to use metrics that beg the question: the common scoring rules always return probabilistic forecasts.

The evaluation ties your hands to making a particular form of forecast. Given a set of knowledge and a statistical score, you are forced to make a constant prediction for all future events. If you allow your predictions to be real-valued, they are suboptimal if they don’t obey the rules of probability. The score itself leads us into a probabilistic mindset. I’ve been calling this metrical determinism, and I find myself inserting some variant of this lecture in every class I teach.

Both the comet example and the life table example can be thought of as scoring track records on average. When you have highly predictable events, a perfect score is possible, but it takes a few hits to convince a skeptic that you really have nailed it down. When events are less predictable, you just want to make sure you’re not losing money on your annuity sales, and maximizing future profits again leads you into a particular form of forecasting.

What’s important here is we don’t have to assume some sort of generative model of randomness to buy into probabilistic prediction. Halley did not have to assume that god was playing dice with who lived and died. Instead, probabilities and odds were simply convenient tools for the actuary to price their products. Probability was the logical consequence of assuming past performance was indicative of future results.

Subscribe now

Hi there, argmin readers! As the fall semester picks up, posting volume will, too. So I’m going to commit to writing short descriptive headers to help you sort through the different threads. Today’s post is a live blog of Lecture 2 of my graduate seminar “Forecasting: A Critical Retrospective.” A table of contents is here.

I’m always looking for how we did science and engineering before the social structures and norms were built up to normalize practice. So for a class on forecasting, let me ask how experts made forecasts before they had proper scoring rules, Bayesian statistics, and Stata.

Though there are plenty of places to start the search, why don’t we today look to the dawn of rationalism in the Enlightenment? Patterns, Predictions, and Actions opens and closes with stories about Edmond Halley. Halley was a master of prediction. He had a keen sense of statistical approximation, knowing how to round and manipulate data to explain the past and predict the future. Halley’s publication record showed a man obsessed with a wide range of forecasting applications.

Halley is best known for his comet, a celestial body that returns to our skies once every 76 years. [footnote: It’s due to return in 2061, but many people’s AGI timelines suggest we won’t be around to see it.] Halley, a strong proponent of Isaac Newton who helped fund the publication of the Principia, wanted to find definitive evidence to prove Newton right.

His proof would come from observations of a comet he had charted in his backyard in 1682. Using Newton’s rules, Halley computed the orbital parameters of the body. He found these parameters neatly matched those of comets observed by Johannes Kepler in 1607. Moreover, they matched those of one seen by German astronomer Petrus Apianus in 1531. Since the gaps between these observations were around 76 years, Halley forecast another observation in 1758. He’d die before its return, but he was right.

This successful celestial prediction is heralded as a crowning achievement of Enlightenment Science. In 1850, Yale astronomer Denison Olmsted, who observed the comet’s return in 1835, wrote, “The Return of Halley’s Comet in exact conformity with the predictions of astronomers established the truth of all those principles by which those predictions were made.” Explaining data we’ve already seen is fine, but there is nothing more convincing to scientists than when theory predicts the future.

The funny thing about this, and a theme we’ll frequently return to this semester, is that this conclusion is completely illogical. It’s a lovely example of affirming the consequent, a fallacy you’ll learn in an introductory logic course.

P implies Q
Q is true
Therefore, P is true.

This syllogism is clearly invalid. (“All the Rationalists live in Berkeley. Ben lives in Berkeley. Therefore, Ben is a Rationalist.” How dare you!) But this is how a lot of science works! Theory predicts a particular outcome. That outcome is observed. This makes scientists feel more convinced their theory is right.

We could go into a long rigmarole about logical positivism at this point, but I don’t want to argue with Bayesian epistemologists today. I just want to point out that even at the inception of the Enlightenment, science was based on the illogic of accurately divining the future. This is one of the things we love about forecasts. When we accurately predict the future, we feel like our internal narrative is true.

Halley’s intuition about forecasting extended far beyond the heavens. He was also a key contributor to modern demography and actuarial science. Protestant pastor Caspar Neumann had collected records of lives, births, and deaths in his hometown of Wroclaw in Poland. Neumann was apparently interested in using this data to disprove the existence of climacterics, where deaths were associated with specific ages like 63. Halley, who came across this data after Leibniz presented it to the Royal Society, had other predictive interests. He churned through Neumann’s data and produced his foundational actuarial life table.

The numbers here represent the counts of people of any given age at a particular snapshot. There were approximately 1000 infants between 0 and 1 (a number, perhaps a bit too convenient), and a total of approximately 34000 individuals in Wroclaw.

In presenting his table to the Royal Society, Halley saw numerous uses for it. He first explained how his table could be used to calculate the number of men draftable into the army. He computed this by counting the number of people aged 18 to 56 and dividing by two.

More relevant to our class, he also pioneered probabilistic forecasting. His second claimed use of the table was calculating the odds that someone might die in a particular time interval. To do this, he counted frequencies and assumed rates of the past were indicative of chance in the future. There were 567 people aged 25, and 560 aged 26. Therefore, the odds a 25-year-old lives to see 26 were 560 to 7 or, simplifying fractions, 80 to 1. Similarly, if you wanted to know the odds that person might live ten years, Halley advocated taking the number alive by age 35, 490, and computing odds: 490 to 77, or approximately 6 to 1.

Halley also used his table to estimate how long people would live by finding the point in a table at which the odds of living dropped below 2 to 1. He called this “The age to which it is an even wager.” Even back in the day before probability, people equated forecasts with fair betting odds.

Not surprisingly, if you could equate mortality forecasts with betting, you could price insurance. This was Halley’s fourth proposed use of his table. Similarly, for a fifth application, Halley worked out more sophisticated calculations and determined a clever scheme to value annuities, a popular means for the crown to raise money. Halley’s calculations set different prices for different ages, based on bets on how long an annuitant might live.

The life table is only one example of Halley’s keen sense that you could make forecasts without physics. Indeed, he seemed to appreciate that the key to forecasting was simply linking past observations with future extrapolations. These extrapolations could be used to confirm physics and create a shared model of reality. They could also guide the pricing of financial instruments wagering on matters of life and death. For Halley, as for us in this class, predicting the future served multiple purposes.

Subscribe now

Hi there, argmin readers! As the fall semester picks up, posting volume will, too. So I’m going to commit to writing short descriptive headers to help you sort through the different threads.

Today’s post is by Jessica Dai, writing up her thoughts on the microconference on “public feedback for AI and beyond” that she ran at UC Berkeley in August. -Ben

Much of the conversation on A(G)I’s “risks and opportunities” centers on as-yet unrealized near- (or far-) future possibilities. Yet AI is already changing the world; its societal impact is already being felt in real time by real people. There’s a yawning gap between “AI elites” — frontier labs and academics and the online AI commentariat — and everyone else, in both beliefs and priorities. I think this is bad — medium- and long-term outcomes will also depend on how society experiences the near-term.1

My vision for dealing with this is something I’ve started calling “public feedback for AI.” The logic goes something like this:

Proposition 1. “The public” has interesting and important things to say about their experiences with AI, but are not typically listened to by decision makers.

Proposition 2. “Evaluations” — and aggregated information, more broadly — are useful, in the sense that they can influence consequential decisions.

Corollary. AI evaluations from public feedback can be a meaningful way to “do something” about the emergent misalignment between those who control AI development and literally everyone else.

I’ve spent the past few years working on this from various angles — trying to articulate why this might be a good idea, methods work for concrete audit instantiations, finding empirical evidence that user-generated content can tell us something useful about AI safety. But the more I think about these problems, the clearer the gaps in my own knowledge; there are many people who are smarter than me, who have worked on related problems for longer than I have, or who have new and different perspectives than those I’m exposed to in my local circles. And while there are a wide range of research questions that I find intellectually interesting, this clearly can’t remain a thought experiment.

Thus, microconference. (You can see a blurb, agenda, and list of attendees at publicfeedback.ai.)

I had a lot of fun at this workshop because while there were some fundamental philosophical questions threaded throughout the two days, everyone was ultimately interested in doing things beyond sending PDF preprints around to one another. And I’m really grateful for everyone’s engagement, because it helped me refine my thinking around each of the foundational ‘propositions’ underpinning the broader vision.

This write-up will be structured around the two ‘propositions’ I outlined above (rather than following the agenda directly). Additionally, I’ll stick closely to what was actually discussed at the workshop, so there is certainly lots of relevant literature that I’m not including here. Editorializing here is mostly mine, and if something is stupid or upsetting, blame me and not whichever speaker I’m referencing!

The first major question I wanted to cover is a core tension that ‘Proposition 1’, as written, glosses over: Who is the public?

While I won’t attempt to do armchair democratic theory here — though if there are any political philosophers reading this, I would love to chat — having clearer and better-motivated answers to this question seems important, because who is considered relevant also directly affects what concerns are substantively important, and also how their views can be understood.

Users?

I started us off on Monday by sharing a few plots from my paper analyzing r/ChatGPT, where we argue that anonymous Reddit data can help us understand broader societal impact; the claim is that the population of “r/ChatGPT posters” can help us simultaneously access a true “population of ChatGPT users” and something about the topics they cared about. I think this was broadly true for the time period we analyzed, but “ChatGPT users” is a very particular choice for defining a “public.”

Nevertheless, if we think AI impacts are largely realized through their users, then usage data is critical to measuring that impact. To this point, the first half of Shayne Longpre’s talk covered the newly launched AI Observatory project, a heroic effort to pull together as many real transcripts as one might reasonably find available on the internet, and to release and analyze them together as a corpus of usage data. Still, there are fundamental challenges with trying to reconstruct proprietary data from the outside; I’m cautiously optimistic about Anthropic’s (very new! Hot off the presses!) data release program.

Study participants or poll respondents?

A common adjective I’ve seen people use to describe chatlogs and other usage data is naturalistic: it reflects the real context in which users were engaging with the AI product. But this data has limitations; it is purely observational, which makes representativeness concerns all the more salient, and cannot provide context beyond the interaction itself (e.g., motivation or outcomes of usage). Instead, the typical approach to “human data collection” in academic contexts is to run more structured studies: recruitstudy participants and ask them to answer survey questions or complete tasks. While this allows researchers the ability to study more detailed questions, these studies have their own limitations; Serina Chang’s talk traced this tradeoff explicitly, with naturalism and control substituting directly for one another.

One issue that came up in discussion was about who these study participants were. Why should we expect that the data they generate is in any way “representative” of the constructs that researchers are trying to measure? A typical academic study promises a pay rate of up to $10/hour (the current standard recruitment platform is Prolific); this is below minimum wage in 30 US states. While $10 USD can be significant in a global context, we were unsure how to reason about the impact of this pay rate on the people who choose to participate.

This is a fundamental problem shared by public opinion polling (which is one reason why Ben hates it), and while it’s probably best not to read into more than one sigfig of poll results, there is reason to believe polls can be useful for coarse-grained judgments of vibe — especially when the vibes are off. Emma Pierson opened her talk with poll results showing the divergence between AI experts/ Claude users, who are generally positive about AI, and a demographically representative sample of the American population, who are generally negative. Jasmine Sun’s datacenter reporting was also motivated in part by national polls showing new datacenter construction polling on par with, or worse than, nuclear and coal. This stood in stark contrast to the SF tendency to dismiss datacenter opposition as merely a problem of misinformation.

Anyone who wants to report?

The approaches described above begin by recruiting people, then gathering their feedback within the explicit context and scope of a study (or poll). An alternative is an inversion of that paradigm: waiting for people to organically share feedback of their own volition, so that ‘the public’ is itself defined by the act of submitting a report. The second half of Shayne’s talk covered a flaw- reporting workflow that lets anyone create a report for an AI-related flaw, vulnerability, or incident, broadly construed, then sends it off to relevant parties. In terms of reporting accessibility, this system rhymes with a public incident reporting portal administered by the California Office of Emergency Services. As Deb Raji explained, this was created as part of SB53’s implementation. While this legislation was mainly directed at companies to self-report severe incidents (more on that later), there is also a provision for enabling public reporting.

Neither of these portals have shared the submissions received thus far, but they have similar design decisions that I suspect also lead to similar challenges (Algorithmic Justice League also has a harm reporting portal, and also hasn’t shared findings, as far as I can tell). Since literally anyone with the link can submit reports, these portals are likely susceptible to ‘low-quality’ submissions or spam. On the other hand, I would guess that overall submission volume is low, since awareness that these platforms exist is also low. I didn’t know about the SB53 portal until Deb’s talk, even though this is kind of my whole thing right now!

Start with an explicitly scoped community in mind, and go talk to them directly.

All three notions of ‘public’ described above involve a very convenient slippage between “public feedback” and “data collection”: they require configuring “the public” as a generic, anonymous mass from which data points can be sampled, and for which summary statistics can be computed. I don’t think this is always a bad thing, but it was striking to me that some of the most compelling case studies for effectively capturing “public feedback” started with an explicitly scoped community, and gathered information by having conversations with individuals within that community.

HCI researcher Samantha Dalal spoke about her work with the Workers’ Algorithm Observatory, which builds tools with gig workers to help identify, measure, and contest algorithmic systems. They bootstrapped worker outreach by collaborating with unions — existing legal entities with corresponding social networks. However, it was still necessary to have face-to-face conversations with workers to build trust and understand their needs.

Humphrey Obuobi shared his work with BLOOM, a nonprofit that is actively facilitating community dialogues in three counties in central Oregon. In this case, while BLOOM develops its own deliberative platform, I see the substance of their work as downstream of the specific, situated collaborations with local civic organizations which — like the unions in Samantha’s work — naturally scope participants to a more specific community.

Finally, Jasmine’s datacenter reporting was focused on understanding the popular backlash; this meant traveling to locations where backlash was visible or brewing. Of course, she talked to local elected officials, who were broadly responsible for decisions at the level of contracts and permitting, as well as activists who were leading more organized opposition. But she also spent a lot of time hanging out at county fairs and beer gardens, talking to people for whom datacenters may not have been the absolutely most salient issue in their lives, but who, nevertheless, turned out to have opinions when asked.

But wait, what about “unknown unknowns”?

I’ve often claimed that the major practical benefit of “public feedback” is the ability to support harm discovery — i.e., to surface unknown unknowns that a centralized decisionmaker (policymaker, company, etc.) might otherwise never think to look for. This seems, to some extent, at odds with the provocation above: doesn’t starting with a specific community in mind inevitably constrain the content of “public feedback” to known unknowns?

One reasonable response is that unknown unknowns exist even within well-scoped communities; prespecifying what groups of people one might be interested in doesn’t necessarily also require committing to particular outcomes or topics, especially if individuals are given the space to express their perspectives in their own voices (e.g., via semi-structured interviews instead of, or in addition to, providing yes/no answers or numerical ratings).

On the other hand, bottom-up community formation can and does already happen. Social media platforms are a natural and well-studied example: networks of like-minded people find each other through a mixture of organic encounters, algorithmic nudges, and moderation choices. For instance, after the initial release of GPT-5, a decentralized campaign to “bring back 4o” emerged on social media (mostly Reddit and Twitter); The Human Line is a research and advocacy group for survivors of AI spirals that emerged from initial encounters on Reddit.

Therefore, if working within communities (rather than a generic abstraction of “users” or “recruited participants”) can be uniquely effective, then perhaps we should try to build systems that explicitly help guide the development of communities based on common concerns. One of the projects covered in Wesley Deng’s talk was an end-user auditing tool called WeAudit, which intentionally included a deliberative and semi-social element that helped individuals contextualize their observations with other user-auditors. A similar motivation is behind an early-stage study shared by Eve Fleisig, where participants see other people’s experiences and can explicitly compare them to their own. In the universe of these projects, the community itself might emerge as an unknown unknown.

The other thing about “community” is that it’s likely of some intrinsic value to a participant — a very different source of motivation from the $10/h paid by Prolific. I’ll write more on the question of “what someone might get out of participating” in the next post.

Subscribe now

I more or less said what I was going to do in my tongue-in-cheek, cryptic post on Tuesday, but let me dive into the full details of how I’m planning on structuring this grad seminar. I’m going to log the course content here, which now lists a rough schedule for the class sessions. Taking inspiration from Matt Jones and Chris Wiggins, I’m going to do a weekly split between culture and engineering. Focusing on a particular application domain each week, we’ll spend one session discussing the purpose of forecasts in that domain and the other on the methods.

I arranged things into a thematic arc, starting with the weather—forecasting’s biggest success story—then moving to shakier ground in seismology and epidemiology, and ending with, well, millenarianism. As we move across timescales, our ability to predict nature dissipates: we can make accurate ten-day forecasts, but predicting large-scale climate disruptions is far more qualitative. We can predict immediate earthquake impacts at a distance once an earthquake has happened, but we can’t nail down precisely when a big one will happen. What we do with precise, short-term forecasts is completely different from what we do with imprecise long-term forecasts. Short-term forecasts dictate actions; long-term forecasts of discrete shocks inform risk management, preparedness, and rapid-response policies. I’m hoping that by the end of the semester I can better articulate what long-term forecasts of the end of the world do.

We’ll look at forecasts in governance and how they influence and shape policy. I’m particularly interested in discussing the rise of cost-benefit analysis in US governance. Cost-benefit analysis puts a specific number on something completely unknowable, but is now mandatory for any bill or law to pass. I want to trace how we became so reliant on a particular set of methods for guessing costs and benefits. I’m also interested in how economists became convinced you could forecast “the economy.” This required inventing something called “the economy” that could be forecast in the first place. How our system of government got so tied to a particular style of economic prediction will occupy several weeks of the class.

We’ll also get into why we want to predict “the public.” I want to examine how opinion polls went from a question of legibility to one of prediction. How did we get obsessed with using surveys to predict outcomes like elections? In parallel, we’ll look at the history of attempts to simulate the public. Simulation is nice because you don’t have to talk to people, right? We’ll look at the many misses over the history of human simulation in policy scenarios, and dig into the current obsessions with using LLMs to predict what people might do.

Finally, we’ll get into the weird culture of competitive forecasting. We’ll engage with ideas from superforecasting and prediction markets and ask why people think these are useful information-processing systems. We’ll talk about punditry and how it’s not always interested in minimizing a Brier Score. We have to talk about the relationship between forecasts and gambling. And we’ll try to piece together why putting odds on outcomes makes people feel better about the future.

For the methods, I did my topic matching so you could extract a logically ordered half-semester course on forecasting from those lectures alone. This is not a class on how to be a rational forecaster. I want to problematize those methods more than tell you how to implement them. You can ask your friend Claude if you need an honest, load-bearing implementation.

Our methods survey starts with a refresher on my idiosyncratic views of machine learning as optimization-driven algorithmic pattern recognition. This will lead to a lot of discussion of the optimization problems themselves and why people like them. We’ll cover scoring rules, calibration, maximum likelihood, and utility maximization. We will discuss the role of models and look at probabilistic recurrence models, dynamical system models, differential equations, and other simulation-based tools. We’ll spend time on uncertainty quantification and how people come up with error bars (part of being a good forecaster is plausible deniability). Then we’ll look at offline and online optimization methods that let you fill in predictions based on your cost functions and modeling assumptions. I’m interested in highlighting the metrical determinism. The cost functions and models more or less tie your hands algorithmically, and most of the cleverness goes into how you evaluate.

Hopefully this arc will feel coherent as we go. I’m not into predictions, so don’t get mad if the story changes as I go. I’ll blog through it, and then we can reflect on where we land at the end of the semester.

Enrolled students (and those dedicated to following along at home) have an important first assignment: pick something to forecast. I don’t care what it is. Throughout the course, the goal is to learn the practical techniques by making predictions. Every week I’ll ask you to try to apply the tools to your problem. Or at least find how other people have applied those same tools to your problem. At the end of the semester, we’ll present our full findings and see how accurate we can be.

Subscribe now

Update: I haven’t joined Anthropic or taken leave from the university.

That means I’ll be teaching as usual this fall, and you can look forward to your regular installment of live lecture blogs. Unlike past years, I’m assigned to my grad seminar in the fall, not the spring. Next semester I’ll be teaching our undergraduate probability class. That is assuming that the University is still here in the spring and hasn’t been put out of business by some new AI super tutor released by my friends across the bay. I mean, it would be the year 2027, and popular forecasts suggest AIs will be able to do everything taught in a CS degree by late 2026. By the end of the spring semester, those same prognosticators predict those AIs will go rogue, and we’ll find ourselves in the reality forecast by James Cameron in his 1984 prophecy, Terminator. Why should I bother dusting off my copy of Bertsekas and Tsitsiklis when the forecasts tell me I should work harder at the gym to prepare myself for robot enslavement in the salt mines?

Actually, you know what would be a good way to prep for the robot apocalypse? Why don’t we spend a semester talking about forecasting and why people are obsessed with being certain about the future?

Answering that question could potentially be a great way to shape a graduate course. We could spend a semester digging into not only how people forecast, but why they forecast. We could split our time in half, looking at the particularities of different domains where people make forecasts, and then looking into the tools they have settled on as mathematical culture.

If you look at the places where forecasts are common, they all have different purposes. A local weather report is very different from a prediction of the end of the world. The former tells you if you should pack an umbrella on the way to work. The latter tells you whether you need to lobby your government to radically change its planned energy buildout. We all believe the evidence supporting forecasts of rain is far more certain and reliable than forecasts of nature’s end. But the costs of being wrong couldn’t be more different.

What impact do these forecasts have? Why do we forecast the weather, and what hidden technology is needed to make these forecasts accurate? Why does Congress demand that we forecast future budgetary consequences of proposed laws, even though we know we can’t predict the actual structural shocks that will render those forecasts moot? Why are people so obsessed with predicting the rise of superintelligent robots? Is it more than a way to justify their greed and obsessive 996 work conditions?

We’d learn a lot from a comparative study. I’d like to look at astronomy, meteorology, climate science, seismology, macroeconomics, government, epidemiology, public opinion research, and millenarianism to see what they have in common and how they differ. Forecasts let people externalize their beliefs about the likelihoods and consequences of various scenarios. Some forecasts are made for mundane planning. Some forecasts are made to literally gamble. Some forecasts communicate possible futures that others might not be considering. Some forecasts are made to be self-fulfilling, to manifest a change in the world the forecaster desires. Some are made to be self-negating, to convince people to act to avoid worst-case scenarios. I’m interested in understanding the threads that link all of these different purposes together.

Though I’m much more interested in the why, the how has some fun tidbits too. The how is about creating certainty about the future by quantifying it. Uncertainty becomes certain once we turn it into an interval, right? In talking about the how-of-forecasting, I could tie together a lot of loose ends in methods that I’ve blogged about over the years.

We might determine the conditions under which pattern recognition (aka machine learning aka AI) becomes a forecast. We could look at how forecasts are evaluated post-hoc with scoring rules and calibration tests and why people think those are good evaluations. We could look at methods for cost-benefit analysis and uncertainty quantification, and how people justify their modeling assumptions to make decisions. We could learn about tools from dynamical systems that move from simple moving averages to complex simulations. We could examine how statistical tools can be applied to extrapolate from the present to the future. And we could see how these sorts of metrics and models tie your hands algorithmically into unsurprising answers.

This sounds like a fun class to me. I predict I’ll teach this class this fall and live blog it here, starting this Thursday. If you’re a Berkeley graduate student whose research depends on forecasts, email me if you’d like to join the course.[footnote: If you do email, please send me a description of your background and why you’re interested.] If you’re not local, I’ll post a syllabus and webpage this week, and I’ll do my best to keep all of the material public. I predict it will be fun.

Subscribe now

Thanks to everyone for the constructive feedback on Monday’s microconferences post. I wanted to take a beat to engage with two salient themes in the replies: runbooks and homophily. I’ll start with runbooks today, and tackle homophily next week.

No two microconferences are alike, and we shouldn’t impose hard-and-fast rules on their structure. One of the fun things about small conferences is you can tailor them to particular goals and dreams. And there are so many ways to do this well.

However, I think it will be useful to compile a runbook of agreements, strategies, and rules that you can use to help modularly assemble your ideal microconference. Today, I’ll run down a bunch of disorganized examples. You tell me your favorite ideas in the comments. I’ll assemble these more coherently into a document that I’ll widely share.

We can learn a lot from existing institutions, and many people spoke fondly of places I should have shouted out in the first post, e.g., BIRS, Oberwolfach, Dagstuhl.1 I also adore the quirkiness of the American Institute of Mathematics, which has a very particular and very fun way to run a microconference, disallowing canned and prepared talks.2 I am inspired by unconferences that bring together dozens of people to spontaneously generate many microconferences. All of their best practices should be part of the runbook.

I’ve found an easy model for microconferences is a bundle of short, 5-10 minute talks with adjoined group discussions led by the speakers. The past four microconferences I’ve attended have run this way. Short talks are fun because they force people to think hard about messaging and concision, and lead to a lot of interesting back-and-forth with the right serendipitous assignments. I was somewhat randomly assigned to a panel with Dan Wang and Dan Davies on cybernetics, and it was probably the most fun and rewarding 90 minutes I’ve ever had at a conference. That single brief session reshaped the narrative arc of The Irrational Decision.

Johan Ugander raised several good ideas in his comment. About archiving and proceedings, he wrote: “The non-proceedings nature of workshops is key to drawing in a diverse set of people.” I agree. We should think broadly about what should count as “proceedings” or “archiving.” I liked Johan’s suggestion of simply inviting participants to submit to a special issue. You could consider this series of blogs ([1], [2], [3], [4]) and this youtube video the “proceedings” of the Cultural AI workshop organized by Leif Weatherby and Tyler Shoemaker this spring. I just want to suggest that we explore creative ways to archive, evaluate, and credit microconferences in the broader academic ecosystem. I’d add to my list of core microconference values that “archiving should encourage, not discourage dissemination and broader engagement.”

As Johan wrote:

“ACM EC has a forward-to-journal mechanism, which gets Computer Science, Operations Research, and Economics folks together for a coherent conference but lets them still go harvest their respective tokens. The EC reviews get passed to the journal. And EC also organizes a ‘Highlights beyond EC’ session, which lets people request to present work recently/already published “elsewhere” but relevant to the community. Both of these mechanisms help keep the ‘conference’ and ‘publication’ goals separate.”

Endorsed!

Anna Gilbert raised the idea of “bump sessions”:

“The Dagstuhl and Oberwolfach type conferences in TCS used to also have “bump sessions” (maybe they were called rump!) where people proposed open problems, noodled over difficulties, etc. Sometimes these micro workshops even “published” open problems from these bump sessions. They were quite useful and engaging!”

Anna also raised one of my favorite ideas, which I’m looking for an opportunity to try:

“Another model I’d like to advocate for is to have members of a PC present the papers they selected (see my rant about paper reviewing and big conferences) and then an audience discussion. Like what the statisticians do but in person and with a publication resulting for the authors.”

I call this the “Not-so-royal Society.” The conference or session would center on a single paper written before the conference is organized. The organizers invite discussants to compose a response/review of this paper. The meeting could start with a presentation of the paper by the author. Many papers have multiple authors, and the presentation can be as collaborative as the writing. The author’s presentation is followed by commentary from the discussants. The remainder of the session is a conversation that invites commentary from the rest of the workshop participants.

After the workshop, the author can revise the paper, the discussants can write formal commentaries, and the author can write a rejoinder if they choose. All of this writing could be posted to arXiv as refereed conference proceedings and linked from the microconference proceedings webpage. This format might be the most legible in the current regime of bean counting and could perhaps serve as a “gentle introduction” to the microconference format. If you have ideas of papers we should use as testbeds for this format, reach out! I’d love to help set something like this up.

Finally, I want to give a shout out to Henry Farrell and Cosma Shalizi, who have been experimenting with clever microconference formats for years. Cosma wrote up the idea of having people present others’ work. Everyone writes a talk or slides, but then the presentation is assigned to someone else. They found that doing this at the start of the workshop creates a shared comprehension, allowing for a lot of multidisciplinary crosstalk. And they also found that speakers had to work extra hard to make their points comprehensible.

This is by no means an exhaustive list of ideas. And it shouldn’t be. My next major to-do is posting a working document of this runbook somewhere. But before I jump in and commit myself to a design, I’m looking for pointers to good online institutional memories that allow for edits and revisions. I love the minimalism of PMLR and bactra.org, but I also want to host a living mission statement and runbook on the same site. Let me know what platforms I should look into. Tech tips would be most appreciated!

Subscribe now

Part of the reason I brought up microconferences on Monday is that I was attending a great one this week on public feedback for AI, organized by Jessica Dai here at Berkeley. In the spirit of creating an archival footprint, Jess and I will have a lot to say about the workshop over the next few days. I’ll kick things off by describing what I talked about.

I opened with a provocation about the trap that policy-minded computer scientists and social scientists so easily fall into. Longtime argmin readers will recognize the pattern. If we want to raise the concerns of a public, we need to convincingly present “evidence” supporting those concerns to policymakers. “Evidence” means cold, hard quantifiable facts that are easy to explain to policymakers, not just anecdotes of harm. I’m sympathetic. If you want to advance an agenda you care about in a complex world, you have to make it simple for those in power to understand. Too many things are happening at once, there’s far too much nuance and ambiguity, policymakers can only keep so many in their heads, and only so many laws can be drafted and passed at any given time.

It makes sense then that people concerned with technocratic solutions spend so much of their time designing architectures of legibility. They focus on the right ways to summarize data, weigh competing interests, and write compelling reports. They build computational frameworks to compile complicated, singular events into useful statistical summaries. A nice chart is worth a thousand testimonies.

These architectures of legibility are what enable the inevitable quantification trap. To make things legible to decision makers, we quantify them. Since we agreed on transparent procedures, the quantified must be objective. Numbers are always objective, right? Objectivity buys analyses authority. And expert authority then becomes a tool of power.

The quantification trap has been a pervasive and mimetic signature of the information age. And it has become progressively invisible as computation has miniaturized and sublimated into ubiquity. Every moment of our lives is now surveilled and quantified, ready to be summarized into new systems of control.

I am not against quantification of social systems. I just want to consistently raise awareness of its hegemony. There are clear benefits to quantification. It buys us a level of intersubjectivity, as anyone can trace the path from evidence to summary statistic. This shared, standardized language lets us collaboratively govern complex societies.

On the other hand, the quantification trap removes discretion, forcing us to abide by rigid rules. Quantification erases individuals in its bucketing and summarization. And quantification enables structural violence, forcing citizens to constantly make themselves legible to those with power to avoid being punished.

The question I always get from technocrats after presenting these critiques of quantification and architectures of legibility is “What else could you do?” My answer is to think about an alternative type of social architecture, architectures of participation. Tim O’Reilly coined this term in the early aughts to describe what makes participatory culture work on the internet. Why do some software systems take off as collaborative efforts? Tim noted that many software and communication systems are designed for contribution. Open source software has countless success stories. We also have the legacy of internet communication, message boards, and the World Wide Web. We have the astounding body of knowledge that is Wikipedia. And we have a powerful public challenge to copyright that was Napster. This last example highlights that not every architecture of participation brings unambiguous good to every stakeholder. The lens of participation helps us think about what elevates voices and values directly through individual actions.

Architecture of participation subsumes many different perspectives on the design of social systems. It includes standard economic mechanism design, the rules of games we play, and the decision-making agreements established by anarchist groups. What distinguishes these systems from architectures of legibility is they aim to cultivate broad expertise from a broad group of people. They are ugly and organic by nature. They are structured agreements for interaction and discussion. They do not suppose a benevolent set of experts at the top will make decisions based on what they see. Mods often write explicit rules, but community patterns are encouraged to be emergent and reflexive. They let the community work together, in small spaces and large ones.

Architectures of participation focus on designing flexible agreements for flexible ends. They accept that there is no clear metric to maximize. Sure, you can build legibility dashboards to make sure your system isn’t crashing. Again, I’m not against quantification. However, the focus of participatory design is not quantification, but broadening engagement and diversifying served ends.

So what does this have to do with AI and public feedback? It is very weird that our contemporary AI took all of human knowledge, made something very interesting and very powerful, and then gave all of the rewards to a small group of people who whine all the time about how they have no power. This is a perversion of participation. Built on the labor and love of individuals, generative AI technology concentrated power. And then those in power convinced themselves they are powerless. San Francisco wants to abdicate all human agency and reduce us to making ourselves legible to an artificial bureaucratic god.

On the other hand, the data center protests are inspirational. Here you have a lot of people who are really upset for a complicated set of reasons about AI. It’s hard to say why they are so mad, and why this issue is so salient, but it’s undeniable that they are driving policy conversations. Their protests and organization have made them not legible, but unignorable.

AI doesn’t have to be exclusionary. We could build an AI that’s a public good. One that invites participation. One that involves known training data, participatory training data. We now know that if you train next-token predictors on collective intelligence, you end up with a quirky piece of software that speaks in natural language, solves impossible math problems, and does all of your coding. We can’t unsee that. But we don’t have to let a small group of whiny, weird people own it. We can use this insight to build a public infrastructure for collective, participatory intelligence.

Subscribe now

I ended my last post with a call for action. I stated, without evidence, that if we’re unhappy with the status quo, we can choose to do something else. I’m hoping this semester I can instead lead by doing (sorry that’s corny) and be a weird freak out in public on this blog. Let me start today by sharing something I’ve been working on: a plan to create alternative systems for generating, evaluating, and sharing knowledge.

While academia has exploded in size in the post-war period, the ideal size of a conference has remained constant. Small workshops with one or two dozen participants have consistently launched big ideas, forged multidisciplinary connections, and set new standards across fields. Look at the history of artificial intelligence (a marketing term I continue to loathe with every ounce of my being but one that is unfortunately useful given its current cultural dominance). The field was conceived at a small conference held at Dartmouth. Societal implications and multidisciplinary connections were hashed out at the Macy Conferences. The ICML conference began as a small workshop of AI researchers at CMU, a splinter group that reclaimed and rebranded the term machine learning. NeurIPS began as small workshops organized at Caltech, called “Hopfest,” where John Hopfield brought together neuroscientists, physicists, and computer engineers to rethink information processing. These workshops each had 10 to 30 people, were cross-disciplinary, aimed at bridging disciplinary boundaries, and focused on brewing creative, new ideas.

Machine learning conferences like ICML and NeurIPS now draw tens of thousands of participants. Somewhere along the way, computer scientists convinced themselves that conference proceedings should be valued as highly as journal publications in other disciplines. A combination of industrial interest and metric-maximizing left AI conferences as an undignified mix of job fairs and “vetting” of “publications” for CV padding. The evaluation is indistinguishable from coin tossing, and no particular result ever stands out. It’s more likely that a mishap at an industry party will get attention than a paper will change the field. This situation is unfortunate and untenable.

Of course, the small workshops—mocking scale, let’s call them microconferences—live on. A professor with a little initiative can put together an ad hoc meeting. There are non-profit institutions dedicated to hosting these sorts of events, including mathematics research institutes like the Institute for Pure and Applied Mathematics at UCLA, computing research institutes like the Simons Institute at UC Berkeley, and institutes that grew to define an entire field like the Santa Fe Institute. However, microconferences are considered luxury goods, something everyone wants to do all the time, but is only allowed to do after their paper writing, teaching, reviewing, and committee work. For some reason, in the modern calculus of the curriculum vitae, a refereed conference paper is more valuable than inspiring a new research direction at a small meeting. This is absurd.

The funniest part about the quantification trap is how we forget that the quantities were ad hoc and artificial in the first place. What makes a Science paper count as prestigious is nothing more than the salivation of a bunch of annoying academics who want Science papers for themselves. If enough of us just want “well-curated microconference proceedings,” then we can value these just as highly.

How could we go about increasing the value and worth of these meetings? I don’t propose constructing an elaborate counting mechanism to have microconferences register as yet another notch on a publication list. And I know we can’t break down our current system of bureaucratic accounting overnight. Deans will still rely on stodgy metrics, scholars will need to produce legible artifacts to appease said deans, and historical institutional prestige will still signal more than it should. Nonetheless, I’d like to offer an initial path for fields to return to the days when small-scale but intense evaluations were valued more than large-scale but cursory selection processes.

Valuing the outputs of small workshops wouldn’t be a departure from our academic traditions. Most of our publication system traces back to reading papers before the Royal Society of London. A paper was only considered published after it had been read aloud and debated in person. Oftentimes, people read other people’s papers and debated these. Lots of our modern systems try to carry on this legacy of debate. In statistics, a main contributed paper is accompanied by “discussions” by other authors and a rejoinder by the contributed paper’s authors. In economics, the working paper serves as a way to gather feedback before official publication, which can take years. Many fields use “special issues” to compile talks from their conferences into proceedings. These are all great ideas! We should draw from the best of these traditions.

One of the hardest things to avoid is making hard-and-fast rules. Mission statements are always forgotten in giant bureaucratic systems, leaving behind a fossilized set of rules and conventions that people assume have been there since time immemorial. These systems then invite the same old gaming and goal displacement. Fully aware that this is the case, let me first state some values I’ve compiled in discussions with friends and colleagues. I’ll offer a few guidelines after.

1. What: Microconferences are where the best ideas are debated. Some things do not and cannot scale.

2. Who: Microconferences should bring together people who didn’t know each other before. You can invite friends, but you can’t only invite your friends. Sometimes you need to get different people in the room and see what happens.

3. When: The turnaround time from conception to microconference should be as short as possible.

4. Organization: While a parent institution for these microconferences can streamline logistics for microconference organizers, it should not require a permanent home or dedicated staff. Its leadership should be minimal and rotate over time. It should be easy to organize a microconference.

5. Output: Not all outputs of a microconference need to be a publication, but sharing findings and insights with others is part of what academics do. While not everything needs to be written down, and ephemeral conversation is important, participants should err on the side of more documentation.

For this final bullet, I envision a documentation structure similar to what organizations like the aforementioned IPAM and Simons Institute currently maintain: a webpage. The Proceedings of Machine Learning Research (PMLR) provides an ideal model. Inspired by the minimal elegance of the Journal for Machine Learning Research, Neil Lawrence cooked up a GitHub repo to compile and curate conference proceedings. It does exactly what is needed and nothing more. The main page will cleanly list past events, and you’ll be able to search by title or participant. Individual microconference webpages could include a list of participants, a photo of the gathering, and a documentary record that links to slides, videos, or papers. What is archived and how it is archived would be up to the organizers.

In the style of arXiv overlay journals, these pages could signify the “blessing” of a particular version of arXiv papers by the participants, demarcating that a paper has been legitimately peer-reviewed. People might dismiss such peer review as cronyism, but if you see the names of participants, you can judge for yourself what credence you want to give to the review.

I am not claiming that microconferences solve all problems. One glaring example is the widespread belief that computer science graduate students need papers. What is a PhD if not a bunch of papers stapled together? Yikes. The obsession with using conferences as a proxy for grad student education is a mistake we are paying for dearly. This proposal does not fix the problem. That said, having microconferences all over the place would be nothing but a boon for graduate students. There will always be one near their university that they can apply to attend. And nothing in this model prevents them from organizing their own microconferences.

Whatever the case, the current situation is untenable. Conferences are impossibly overrun, and their publications fully depreciated. A framework for microconferences offers something else. It is not a replacement. It is a new path to credentialing and cultivating expertise. Will it also end up a bureaucratic mess in a few years? Probably so! But I hope you’ll join me in trying it out before declaring defeat.

Subscribe now

I’ve started and stopped four posts this week and couldn’t figure out why I was stuck. Finally this morning, in a delightful retelling of an Alan Moore comic, Henry Farrell struck the nail in my head.

Henry describes Moore’s society where a supergenius benevolent dictator named Abelard Snazz designs giant police robots to fight crime. The robots get too zealous and start bothering ordinary citizens. So Snazz deploys criminal robots to keep the police busy. The cycle continues until people are forced to flee the planet.

Henry, always keen to show us what sci-fi tells us about our lived experience, describes how this automation is playing out in our lives, with people heading to chatbots to deal with bureaucracies, and governments planning new AI systems to deal with the onslaught of ChatGPT protestations. We know where this ends. And this week has seen way too many stories of this abandonment of agency and succumbing to AI.

I drafted a blog about our annual complain-fest about the academic wreckage that is the NeurIPS conference. Everyone is taking to social media to complain that people are shirking their reviewing responsibilities. They scoff at new limits of 25 papers per author per conference. They lament that AI papers and referee reports are now better than those submitted by people. They say no one reads any of these papers. And yet, everyone continues to submit papers, review papers, and be area chairs.

It is completely unclear what everyone wants from this process anymore. With every purported fix, the process gets more onerous, and the problems only get worse. More papers are submitted, more hair is torn out in frustration. Every day I’m reminded of what Kevin Baker wrote about this acceleration of publishing and perishing: “Systems can persist in dysfunction indefinitely, and absurdity is not self-correcting.” But Kevin’s next sentence is more damning. “Whether the acceleration produces collapse or adaptation or simply more of the same is not a question about the technology, and it won’t be answered by debates about capabilities.“

I drafted a blog post about open letters from employees of tech companies. They warn of all of the potential harms of the technology they are building. They beg for the government to regulate them. They decry their helplessness as they collect their 7-figure remuneration. Apparently they can’t see a path to do anything themselves.

These people want us to trust them, but then they keep admitting they are committing crimes. They write about their software as a person, trying to deflect from their own incompetence in security standards. They demand strict testing of open models while bragging about their laughably sloppy practices. They demand that the government come in and regulate them, because they can’t get out of their own way.

Even in esoteric places, we see a deferral to the AI. The lore laundering machines are now good enough to prove actual theorems, finding the key clever steps needed to take the existing literature and close “major” open problems.1 There’s a lot to say about what’s happening here. It’s still lore laundering, and mathematicians are rightfully angry that the models remain horrible at proper attribution. But the new models are generating much more clever stitches to glue the proofs together.

OpenAI has been running a distributed denial-of-service attack on unsolved math problems, and their internal tooling has finally been able to solve some very hard ones. Only three people in the world know what a non-sofic group is and why it’s important,2 but the new OpenAI Astra model has managed to construct one. The mathematical progress has created an arms race in mathematics, where people are telling mathematicians that they need to do all their work with ChatGPT now or be left behind. Or even worse, that there’s no need for mathematicians anymore. What this means for mathematics and applied mathematics more generally remains to be seen. But I guarantee you that people are going to quickly tire of theory papers on arXiv that are exclusively transcripts of professors having conversations with ChatGPT.

As Henry writes, all of these cases are arms races. As with all arms races, the momentum can feel insurmountable. I’m surrounded by people throwing up their hands as if they have no agency. But sometimes it’s important to step back for a second and see that where we’re heading is not inevitable. We do have choices here.

Researchers don’t have to send papers by the tens of thousands to conferences where no one reads anything. They can collectively design better systems to generate, evaluate, and share knowledge. Employees can use the power of their labor to steer their companies, rather than pleading for someone else to step in. Mathematicians can continue to think, understand, and teach. We all have the power to do something else.

Subscribe now

I hope the intent of my call on Monday was clear: we should strive for something that any team can build from scratch as long as they have access to the training corpus, the software, and computing resources. We want something open to the public, as the models are a reflection of public intelligence and culture. As Colin Fraser remarked on Bluesky, “there’s no going back to the world before we knew that if you make a language model large enough it appears to become a little guy who sometimes solves open math problems and sometimes makes you insane.” These language models are built upon humanity’s collective, cultural intelligence, and I believe they should thus be open and accessible to everyone.

But what “open” exactly means is tricky. A laudable example of an open corpus model is Olmo from Ai2. Their model report details all of the data used. For pretraining, they use a mix of data from Wikipedia and Wikibooks, web pages extracted by Common Crawl, academic papers sourced from arXiv papers uploaded with LaTeX, code curated from GitHub repos with flexible licenses, and math webpages from FineMath 3. Earlier versions of the Dolma corpus also include Reddit threads, papers from Semantic Scholar, and public-domain books from Project Gutenberg. All of this data is available to everyone.

Now, here’s a question for the purists out there. FineMath 3 is generated using annotations from the Llama LLM. On the one hand, technically speaking, Llama is not an open corpus model. On the other hand, the FineMath dataset is free to download. I’d argue this still counts.

Even if you grant me that one, you’ll find a lot more use of LLMs in the data pipeline in the Olmo tech report. For instance, to generate some of the math training data in the later stages of training, the team uses Qwen. Qwen is not an open-corpus model. For some of the more complex reasoning, they use thinking traces from GPT4. That’s not an open model in any capacity. If you are using a closed-corpus model to generate training data for your open-corpus model, have we ended our game for full openness?

I’m not a purist, but the vague world of distillation is genuinely complicated. Distillation is not a cleanly defined action, but roughly describes the practice of collecting the text outputs from one language model to serve as the training set for another. What we’re allowed to use and not use is incredibly confusing. Even if we just go back to pure text, it’s hard to say what training data is actually legally acceptable under “the law.”

The law is confusing and unsettled. If you buy a physical book, scan it, run OCR, and add it to your training corpus, that counts as “fair use.” If you buy the exact same text as an ebook and add it to your training corpus, that’s violating the e-book licensing agreement. The fact that there is a distinction between these two actions is absurd and stupid. Stupidity is unavoidable in our complex legal code. It doesn’t get less stupid when you try to understand how distillation meshes with the terms of service for use of LLMs.

I’m thinking out loud and consequence free here on the newsletter. I’m happy to admit that it’s complicated. But we’re going to have to change or fight laws that prevent us from just outcomes. If we want to prevent a concentration of power at OpenAI or Anthropic, we need to think about what laws are just and right to prevent that.

Anthropic CEO Dario Amodei, in his typically blindered, insufferable way, chimed into the open models debate on Monday, arguing that the US should “ban industrial-scale distillation” but that he’s not calling for banning open-weight models. His letter is riddled with contradictions like this. I don’t care about his tortured reasoning because it’s still the case that the far more defensible position is banning closed-weight models.

Closed models are indefensible. Sure, we should give Alex Radford, Ilya Sutskever, Sam Altman, and Dario Amodei some credit because I would have never guessed that the models would be as useful as they are today. As Fraser said, there’s no going back to the time before we knew that. However, you don’t have to give them too much credit because you could also argue they should be in jail. If you think that is hyperbole, I’d like you to read the wikipedia page of Aaron Swartz. Or read this in-depth article in the Financial Times about the online libraries used to train our current language machines. The librarians are hunted across the globe by the FBI. The LLM entrepreneurs are gazillionaires and can pay billion-dollar settlements to cover their asses. We can have long, pedantic quibbles about what’s legal and what’s not. We can also ask, “What is right?” and “What is just?” The current situation where the best American models remain closed is deeply wrong.

Subscribe now

I applaud Jensen Huang and industry leaders for coming out in support of open large language models. The entire tech sector minus Anthropic has now signed on. Now that the movement has momentum, I urge the signatories to endorse something even more radical that would truly separate us in our economic competition with China: The USA should heavily invest in models not just with open weights, but with open source and open corpus.

Let me back up in case you missed it. Last Friday, Huang pasted a short letter on Twitter calling for American investment in “a strong, open ecosystem” around open source artificial intelligence. “Open source…created a shared foundation of knowledge on which generations of American engineers and entrepreneurs built their institutional sovereignty,” Huang predicts that a similar thing will happen with the embrace of open source AI.

Part of this letter was a plea to the Trump administration, which had been sending signals about banning open source software in a protectionist move to squash insurgent Chinese language machines. Seeing that this would benefit Anthropic and OpenAI and no one else, the CEOs of all of the big and small tech firms quickly signed on to Huang’s letter on Friday.

This is an amazing, positive development, and I want to air some points that Huang left out. I’ve been passionately calling for a deeper investment in open source language models on this blog for years. In fact, this post from last July makes a bunch of points about what I think is necessary for this to happen, and I don’t think any of it has changed.

You should go read that post, but let me summarize what I said for the present moment. What is needed to build a good language model is people, compute, and data. It’s not clear how much of each of these you need, but the answer seems to be a lot. We’ve always had the people, though the promise of impossible riches has lured many smart young minds away from more ethical paths. But now that we have the backing of the entire tech industry, I think this base is covered.

What about compute? Hyperscalers and their economic bedfellows always want you to believe that their massive datacenter buildout is a moat from us open source plebes. But the Chinese models have complicated this story and shown you can get by on the cheap. No one has precise estimates, but models from Chinese companies DeepSeek and Moonshot were arguably trained using under ten million dollars. Dozens of AI startup “NeoLabs” with few concrete ideas are currently being given 10x that by venture firms. Good VCs could make a small bet on the open ecosystem to juice their other investments. Compute for open models will require a massive industrial consortium, but it would cost each member of the consortium a minuscule fraction of their war chests. Let’s work together to build computing agreements for open source development.

Once these models are built out in the open, there is no doubt that they will only get dramatically more efficient. We’ve seen time and time again that machine learning is a field that innovates through “frictionless reproducibility.” Research, code, and data out in the open, evaluated by competitive benchmarking, rapidly improve machine learning systems. It also makes them more efficient. In last year’s post, I wrote about how high-quality ImageNet models went from something only trainable at Google to something you could build on a desktop in less than a year. These sorts of efficiency gains happened throughout the 2010s. The secrecy of labs in the 2020s has harmed the broader engineering field, even though the artifacts produced have been beyond impressive. Put everything out in the open, and we’ll figure out how to make it faster and more efficient. It’s guaranteed.

That brings us to the actual hard part. The data. As my friend and colleague Alyosha Efros loves to remind us, “It’s all about the data,” and language models are trained on unfathomable amounts of it. Discovery in lawsuits has revealed that companies trained these models on pirated libraries of books, academic papers, and copyrighted imagery. The models are trained on collaborative knowledge bases like Wikipedia, countless volunteer forums like Reddit, and all of the public code on GitHub. They are trained on the transcript of every video you post to YouTube. All of this collective work by human society gets slurried into proprietary software so a few zealots can get rich. This is a bad outcome!

The companies are all up front about this use. They have been found liable in court. They have admitted it in papers they have written. They claim this is all fair use because the training was transformative of the texts. Fine, if that’s the case, then it’s fair use to take the outputs of their models and build new ones. This is called “distillation.” Chatbot terms of service agreements do not negate my argument. That the Trump administration is flirting with banning distillation is a travesty.

But I want something bigger. The biggest step to making competitive open source models is allowing the broader community fair use access to the same material the companies used. This is the open corpus. If closed models can exist, then open models should be allowed a level playing field. This will require a long overdue rethinking of intellectual property and what we owe the people who create it.

I think all of these challenges are surmountable. We’re going to have to (a) get young people to care about open source instead of becoming impossibly rich. (b) get billionaires to collaborate in a non-winner-take-all fashion, and (c) have a long hard conversation about copyright law, intellectual property, and fair use standards. But we’ve thrived in an open ecosystem before ChatGPT. It hasn’t even been half a decade of closedness. With the blessing of the entire tech industry, it’s time to open things up again.

Subscribe now

The title of this post is a coinage of Kevin Kelly. His dedication also inspired me to write this post.

I’ve been obsessed with playing guitar and making music since I was thirteen. Obviously, as a giant nerd, I approach this in the nerdiest way possible. So much of my playing is thinking about the mathematical relationships in patterns of intervals and shapes. I love challenging abrasive music that involves odd counting or microtones. I geek out on music technology, be it software or hardware. But in the decades I’ve been doing this, I’ve never come across someone who thinks I’d be able to get better at music through science.

Why are there no science-based guitar lessons? There are countless books with varied methods on the fastest way to learn or the best way to get better. There’s beautiful math you can nerd out on. Just like in weightlifting, people sell their lessons online on Instagram. But none of the musicians I follow ever put up PubMed screenshots. Instead, you find a psychopathic Spaniard who learns to play impossibly hard drum parts with one arm tied behind his back.

Don’t the laws of adaptation apply to music? It’s certainly the case that the more you practice, the better you get. Much of music practice is just sensorimotor learning. You are adapting both neural pathways and strengthening muscles. There’s psychophysical adaptation in syncing your limbs up with a click track. There are fine motor skills involved in hyperefficient picking. It’s also clear that if you practice too much, you’ll make your hands bleed or develop tendonitis. Surely Selye’s General Adaptive Syndrome applies to woodshedding.

I mean, the concept of progressive overload clearly applies to practice. The way you learn a song is to set your metronome to half time and work your way through the notes. Once you have it down, you increase the beats per minute by one. Each time you come back, you play faster than last time. Next thing you know, you are shredding Chopin.

And it’s not like there’s one perfect way to learn how to be the best at your instrument. There are wide disagreements about the best auxiliary exercises to improve your chops. There is an infinite collection of tutorials and courses designed to improve your skills, telling you which rhythms to practice, which strings to skip, which positions to memorize. And yet, no one thinks you need to run RCTs on these books to crown the best one.

You might say music exists more for art than for competition, and I’d applaud your idealism. But the music industry is a rough one, and it gets pretty cutthroat when you try to make a career out of it.

I write all this to ask what it is about the gym that makes us think that we can science it. I don’t have a good answer to this yet. I have a few partial answers, but none are particularly satisfying to me.

First, there is a clear connection to medicine, and for a wide variety of reasons we’ve decided that medicine needs to be based in science. Physical therapy can only argue for legitimacy in the healthcare sphere if it can be proven a cost-efficient therapy. That means it must be based in our post-modern science of efficiency, and hence we are forced to run RCTs.

The medical reasoning breaks down when it comes to weight lifting. A lot of the biggest names in the science-based lifting community are obsessed with hypertrophy, the fancy scientific word for increased muscle size. While it’s true that to be a strength athlete you need to have big muscles, most of these guys are selling programs for aesthetics. Dudes want to grow their muscles because looking jacked builds their self-esteem. The sport of hypertrophy, bodybuilding, is competitive body dysmorphia. Bodybuilding is an astonishingly unhealthy sport. It involves taking ungodly doses of performance-enhancing drugs, alternating binge eating with starvation, and walking around at dangerously low body fat levels. The pro athletes in this sport die at rates far higher than football players. It’s a mess. There is no good health-related reason for you to look like a modern bodybuilder.

But perhaps the universal desire of perfect aesthetics is a core part of why science is so alluring. Not everyone wants to be able to play 10 over 11 polyrhythms, but everyone wants to look hot. And since it’s so universal, the nerds want a way to stick it to the jocks and claim a masculine domain that hadn’t originally been theirs. The authority of science gives the nerds an institutional leg up in a world where all men want to stake their claim. Science-based training is to the gym as the analytics department is to professional sports teams.

However, when it comes to the strength sports, the science takes you about as far as the analytics takes a football team. Strength sports are not as quantitative as they look. There’s no optimal answer for any person to follow to become a championship Olympic weightlifter, even though the goal of that sport is to get the sum of two numbers as high as possible. Working hard, resting, and eating get you 80% of the way there. Science can’t fill in the rest. If you want to do heavy clean and jerks and snatches, you have to show up and do them.

Subscribe now

Last week in Zócalo Public Square, I wrote a piece on optimizing diets, adapted from the second chapter of The Irrational Decision.1 It’s one of my favorite stories in the book. Though it predates computers by several years, it’s a microcosm of the computer age and fits in seamlessly with today’s oddly dominant online culture of wellness optimizers. In a spat with USDA nutritionist Hazel Stiebeling about what sorts of recommendations are acceptable for the government to publish, the prickly economist George Stigler solved a complicated tableau by hand to find a rather unpalatable “minimum cost subsistence diet” of wheat flour and navy beans. Go read the essay, and then come back here and read a few fun addenda.

I tell the story of the diet problem in most of my book talks, and I always receive fun feedback. Jeff Linderoth sent me a hilarious reflection by George Dantzig, the inventor of linear programming, on his apparently futile attempt to find his own optimal diet to lose weight. Though Dantzig knew Stigler’s optimization problem had nothing but absurd and disgusting solutions, he figured he was adept enough at building linear programming models to patch Stigler’s simplistic assumptions with appropriate shaping of objectives and constraints. He used spare cycles of the IBM 701 at the RAND Corporation to churn out ever more innovative meal plans. But he kept getting bizarre recommendations, like drinking gallons of vinegar or consuming mass quantities of bouillon. He diligently refined the diet over the course of a week before his wife, Anne, got fed up:

Speaking firmly so that I would know who was boss, she said, “I have been studying the various menus the computer has been generating. There are some good ideas there that I can use. I’ll put you on MY diet. She did and I lost 22 pounds.

Steve Stigler—not only an amazing statistician and historian but also George’s son—attended my talk at the University of Chicago. Steve told me how this paper made its way out of academia and into national newspapers, ruffling feathers from coast to coast. George would receive angry letters scolding him about how “this is no way to feed growing boys.” I thought it was pretty clear from reading the original paper that George didn’t think anyone should try to eat his diet. He was trying to prove a point about the impossibility of optimal diets and the paternalistic nature of government recommendations. But people ended up taking him literally. Papers that start as sardonic jokes can surprisingly take on a life of their own.

Stigler’s paper is part of a broader conversation about the scope of government policy. The idea of a computable government was central to economic discussions during the Great Depression and throughout the Second World War. Experts and government officials argued about what is optimal, what can be planned centrally, what individuals should be allowed to navigate for themselves, and what sort of information is beneficial and which is coercive. These debates strongly influenced von Neumann, as you can see in his and Morgenstern’s engagement with contemporary economic debate in the introduction to their revolutionary book on game theory. For another fun example of the people building computers closely interacting with the people designing policy, here’s a 1958 photograph of a sharply dressed Claude Shannon at the Center for Advanced Study in the Behavioral Sciences at Stanford, taken by George Stigler on Steve Stigler’s camera.2

The diet debate also raises the uncomfortable central theme in Elizabeth Popp Berman’s book, Thinking Like an Economist. Everyone across the political spectrum is arguing about efficiency, as if that’s the only thing the government should think about. Left-wing technocrats (aka the Democrats) apply this sort of economic thinking to the utility of the population. Right-wing policymakers (aka Republicans) apply economic thinking to the utility of the individuals in that population. No matter their politics, everyone is thinking like an economist. The valence of the arguments and the parties making those arguments remain uncannily similar today.

Given the grand scale and ambition of the federal government, USDA dietary guidelines should be a fourth-order concern. But there’s something about worrying about what we should eat that galvanizes the popular imagination. It’s fun to walk through the original arguments about what should be in the food pyramid, especially given the weird steak-centric geometry being pushed by RFK’s cuckoo version of HHS. A steak every day sure sounds more appealing than a bean pie.

Subscribe now

In the noisy chaos of science-based wellness on social media, there are a few reasonable voices. One of my favorites1 is Greg Nuckols, a champion powerlifter who has been blogging about the science of strength training for well over a decade. Over the weekend, I read his 2015 books The Art of Lifting and The Science of Lifting, and they hammer home how a simple topic can be made overly complicated when we decide to write scientific papers about it.

Nuckols’ two-volume series is less than two hundred pages long and written in an engaging, bloggy voice. Though the books link out to systematic reviews and randomized trial reports, he doesn’t get bogged down in that material. Instead, he focuses on one simple modeling principle and its consequences: adaptation.

Though the human body is impossibly complex, its many interconnected systems deal with stressors in a surprisingly uniform way. Stressor here means all sorts of things: wounds, infections, temperature changes, environmental changes, activity changes. Something out of the ordinary. The model of homeostasis posits that bodies actively work to stay the same as much as possible. Most of our daily activities don’t stress the body, and we remain in a pleasant, steady state. However, the body changes and adapts if it gets pushed far enough out of its comfort zone. If you apply enough stress, the body initiates a response to prevent harm or death. There are limits to this response, and too much stress will cause injury or death. But in the right Goldilocks range, the body will reconfigure itself to resist the stress next time it sees it.

Exercise can be modeled as a stressor. It’s certainly stressful. It increases muscle tension and body temperature. It increases demand for oxygen and nutrients. It lights up your sympathetic nervous system and triggers the release of adrenaline. Your body responds to exercise similarly to how it fights off other stressors like injuries and infections. It panics and then marshals resources to make sure it sucks less the next time you go to the gym.

There are many models for the temporal behavior of the stress reaction. The most common mathematical model is the fitness fatigue model developed by Tom Calvert, Eric Banister, and collaborators in the 1970s. Notably, Calvert was an electrical engineer who introduced his physiology colleagues to cybernetics and impulse responses. The fitness-fatigue model is a parametric model of adaptation: stressors introduce bad effects (fatigue) from which the body works hard to recover quickly. It also introduces positive training effects, the adaptations you were chasing. The fitness-fatigue model posits that the positive effects decay more slowly than the negative ones. For the dynamical systems nerds out there: they model this with a simple second-order linear system. Hence, if you repeat this process of enough but not too much stress over time, you nudge your body to handle more and more stress.

This feels science-y, no? The general adaptation syndrome is a well-tested, clean model that certainly helps guide how to train athletes and implement physical therapy. This model hasn’t changed since 1980. It prescribes working hard and continuing to do so over time to increase adaptation. It predicts that training harder than last time is beneficial. These two together form the principle of progressive overload, a very simple concept that influencers try to make absurdly complicated. Moreover, this model of adaptation tells you that you need to recover to hasten the elimination of fatigue. It even suggests implementing tapering periods before competition, a common practice.

OK great, now let’s try to apply this model to practice. What is the right shape of the actual stress to give you the results you want? If you want to get stronger, which exercises should you do? How frequently should you do them? What weights should you use? How many repetitions should you do for each exercise?

Unlike most of the charlatans in the science-based fitness space, Nuckols is brutally honest about this: the answer to all of these questions is that we don’t really know. Or at least, we at best only have partial answers. You probably need to go above some nominal effort to induce adaptation. Because there is so much intersubject variability, the exact perfect amount is hard to pin down. But if you work hard and make sure to rest and eat, you’ll get stronger over time.

Nuckols thinks the broader scientific research on sports training tells us more than I think it does. From my reading of too many studies in this space, the ns are always really small, the data are always poor, and the assessments are always ill defined. Do we need thousands of studies to tell us that we should “keep it simple, stupid?” People obviously can go to the gym and get stronger.

But what’s interesting to me about how Nuckols writes is that he doesn’t consider “science” to be some combination of biomolecular pathways and randomized trials. Instead, his scientific view is one of simple predictive models and their limitations. For an athlete like Nuckols, the scientific view helps systematize what matters in training. Working hard matters. Working harder over time matters. But resting and recovery are also critical to good performance. The right balance might not be easy to quantify, but the qualitative arc and predictions based on principles of adaptation are helpful. The general adaptation model for training tells us that training isn’t that complicated if you think carefully about what you are going to measure, and how you are going to track progress. This cybernetic view of biomedical science—one closer to what I would call engineering—feels like a promising path for many other aspects of medicine.

Subscribe now