Part of the reason I brought up microconferences on Monday is that I was attending a great one this week on public feedback for AI, organized by Jessica Dai here at Berkeley. In the spirit of creating an archival footprint, Jess and I will have a lot to say about the workshop over the next few days. I’ll kick things off by describing what I talked about.
I opened with a provocation about the trap that policy-minded computer scientists and social scientists so easily fall into. Longtime argmin readers will recognize the pattern. If we want to raise the concerns of a public, we need to convincingly present “evidence” supporting those concerns to policymakers. “Evidence” means cold, hard quantifiable facts that are easy to explain to policymakers, not just anecdotes of harm. I’m sympathetic. If you want to advance an agenda you care about in a complex world, you have to make it simple for those in power to understand. Too many things are happening at once, there’s far too much nuance and ambiguity, policymakers can only keep so many in their heads, and only so many laws can be drafted and passed at any given time.
It makes sense then that people concerned with technocratic solutions spend so much of their time designing architectures of legibility. They focus on the right ways to summarize data, weigh competing interests, and write compelling reports. They build computational frameworks to compile complicated, singular events into useful statistical summaries. A nice chart is worth a thousand testimonies.
These architectures of legibility are what enable the inevitable quantification trap. To make things legible to decision makers, we quantify them. Since we agreed on transparent procedures, the quantified must be objective. Numbers are always objective, right? Objectivity buys analyses authority. And expert authority then becomes a tool of power.
The quantification trap has been a pervasive and mimetic signature of the information age. And it has become progressively invisible as computation has miniaturized and sublimated into ubiquity. Every moment of our lives is now surveilled and quantified, ready to be summarized into new systems of control.
I am not against quantification of social systems. I just want to consistently raise awareness of its hegemony. There are clear benefits to quantification. It buys us a level of intersubjectivity, as anyone can trace the path from evidence to summary statistic. This shared, standardized language lets us collaboratively govern complex societies.
On the other hand, the quantification trap removes discretion, forcing us to abide by rigid rules. Quantification erases individuals in its bucketing and summarization. And quantification enables structural violence, forcing citizens to constantly make themselves legible to those with power to avoid being punished.
The question I always get from technocrats after presenting these critiques of quantification and architectures of legibility is “What else could you do?” My answer is to think about an alternative type of social architecture, architectures of participation. Tim O’Reilly coined this term in the early aughts to describe what makes participatory culture work on the internet. Why do some software systems take off as collaborative efforts? Tim noted that many software and communication systems are designed for contribution. Open source software has countless success stories. We also have the legacy of internet communication, message boards, and the World Wide Web. We have the astounding body of knowledge that is Wikipedia. And we have a powerful public challenge to copyright that was Napster. This last example highlights that not every architecture of participation brings unambiguous good to every stakeholder. The lens of participation helps us think about what elevates voices and values directly through individual actions.
Architecture of participation subsumes many different perspectives on the design of social systems. It includes standard economic mechanism design, the rules of games we play, and the decision-making agreements established by anarchist groups. What distinguishes these systems from architectures of legibility is they aim to cultivate broad expertise from a broad group of people. They are ugly and organic by nature. They are structured agreements for interaction and discussion. They do not suppose a benevolent set of experts at the top will make decisions based on what they see. Mods often write explicit rules, but community patterns are encouraged to be emergent and reflexive. They let the community work together, in small spaces and large ones.
Architectures of participation focus on designing flexible agreements for flexible ends. They accept that there is no clear metric to maximize. Sure, you can build legibility dashboards to make sure your system isn’t crashing. Again, I’m not against quantification. However, the focus of participatory design is not quantification, but broadening engagement and diversifying served ends.
So what does this have to do with AI and public feedback? It is very weird that our contemporary AI took all of human knowledge, made something very interesting and very powerful, and then gave all of the rewards to a small group of people who whine all the time about how they have no power. This is a perversion of participation. Built on the labor and love of individuals, generative AI technology concentrated power. And then those in power convinced themselves they are powerless. San Francisco wants to abdicate all human agency and reduce us to making ourselves legible to an artificial bureaucratic god.
On the other hand, the data center protests are inspirational. Here you have a lot of people who are really upset for a complicated set of reasons about AI. It’s hard to say why they are so mad, and why this issue is so salient, but it’s undeniable that they are driving policy conversations. Their protests and organization have made them not legible, but unignorable.
AI doesn’t have to be exclusionary. We could build an AI that’s a public good. One that invites participation. One that involves known training data, participatory training data. We now know that if you train next-token predictors on collective intelligence, you end up with a quirky piece of software that speaks in natural language, solves impossible math problems, and does all of your coding. We can’t unsee that. But we don’t have to let a small group of whiny, weird people own it. We can use this insight to build a public infrastructure for collective, participatory intelligence.
Your post rhymes with James Scott's "Seeing Like a State". The technocratic powers that be share Scott's description of "High-Modernist Ideology" which is a form of your quantification trap. A quasi-religious faith that scientific planning and centralized design can optimize every facet of human life. Given that ideology, the powers that be feel justified in pursuing authoritarianism to realize their utopia by steamrolling local/contextual wisdom and practice.
Integrating local/distributed participation is a obviously a good thing, but maybe even more important is a recognition of the failures of architectures of legibility and the slippery slope to authoritarianism (Scott's example of 'scientific forestry' is sobering). A strong dose of humility among technocrats will get them to think about smaller, reversible interventions that can learn and benefit from (hopefully locally sourced) feedback.
You have already made this point implicitly (or explicitly and I might have missed it). Quantification seems like a way to reduce complexity when making decisions, constraining both the space of information you consume and choices you make downstream. This is bad on a human level as you often point out.
However, it also allows you to benefit from decision making at scale. This seems both (via vague analogies) true for bureaucracies and for e.g. deep learning. For example, it seems like you could indeed design much more efficient solutions if you look at specific problems (and expand your information choices) but to enable general purpose learners, perhaps its useful to reduce the complexity of the feature space (all text!).