From legibility to participation

An emphasis shift for human-facing computing research

Part of the reason I brought up microconferences on Monday is that I was attending a great one this week on public feedback for AI, organized by Jessica Dai here at Berkeley. In the spirit of creating an archival footprint, Jess and I will have a lot to say about the workshop over the next few days. I’ll kick things off by describing what I talked about.

I opened with a provocation about the trap that policy-minded computer scientists and social scientists so easily fall into. Longtime argmin readers will recognize the pattern. If we want to raise the concerns of a public, we need to convincingly present “evidence” supporting those concerns to policymakers. “Evidence” means cold, hard quantifiable facts that are easy to explain to policymakers, not just anecdotes of harm. I’m sympathetic. If you want to advance an agenda you care about in a complex world, you have to make it simple for those in power to understand. Too many things are happening at once, there’s far too much nuance and ambiguity, policymakers can only keep so many in their heads, and only so many laws can be drafted and passed at any given time.

It makes sense then that people concerned with technocratic solutions spend so much of their time designing architectures of legibility. They focus on the right ways to summarize data, weigh competing interests, and write compelling reports. They build computational frameworks to compile complicated, singular events into useful statistical summaries. A nice chart is worth a thousand testimonies.

These architectures of legibility are what enable the inevitable quantification trap. To make things legible to decision makers, we quantify them. Since we agreed on transparent procedures, the quantified must be objective. Numbers are always objective, right? Objectivity buys analyses authority. And expert authority then becomes a tool of power.

The quantification trap has been a pervasive and mimetic signature of the information age. And it has become progressively invisible as computation has miniaturized and sublimated into ubiquity. Every moment of our lives is now surveilled and quantified, ready to be summarized into new systems of control.

I am not against quantification of social systems. I just want to consistently raise awareness of its hegemony. There are clear benefits to quantification. It buys us a level of intersubjectivity, as anyone can trace the path from evidence to summary statistic. This shared, standardized language lets us collaboratively govern complex societies.

On the other hand, the quantification trap removes discretion, forcing us to abide by rigid rules. Quantification erases individuals in its bucketing and summarization. And quantification enables structural violence, forcing citizens to constantly make themselves legible to those with power to avoid being punished.

The question I always get from technocrats after presenting these critiques of quantification and architectures of legibility is “What else could you do?” My answer is to think about an alternative type of social architecture, architectures of participation. Tim O’Reilly coined this term in the early aughts to describe what makes participatory culture work on the internet. Why do some software systems take off as collaborative efforts? Tim noted that many software and communication systems are designed for contribution. Open source software has countless success stories. We also have the legacy of internet communication, message boards, and the World Wide Web. We have the astounding body of knowledge that is Wikipedia. And we have a powerful public challenge to copyright that was Napster. This last example highlights that not every architecture of participation brings unambiguous good to every stakeholder. The lens of participation helps us think about what elevates voices and values directly through individual actions.

Architecture of participation subsumes many different perspectives on the design of social systems. It includes standard economic mechanism design, the rules of games we play, and the decision-making agreements established by anarchist groups. What distinguishes these systems from architectures of legibility is they aim to cultivate broad expertise from a broad group of people. They are ugly and organic by nature. They are structured agreements for interaction and discussion. They do not suppose a benevolent set of experts at the top will make decisions based on what they see. Mods often write explicit rules, but community patterns are encouraged to be emergent and reflexive. They let the community work together, in small spaces and large ones.

Architectures of participation focus on designing flexible agreements for flexible ends. They accept that there is no clear metric to maximize. Sure, you can build legibility dashboards to make sure your system isn’t crashing. Again, I’m not against quantification. However, the focus of participatory design is not quantification, but broadening engagement and diversifying served ends.

So what does this have to do with AI and public feedback? It is very weird that our contemporary AI took all of human knowledge, made something very interesting and very powerful, and then gave all of the rewards to a small group of people who whine all the time about how they have no power. This is a perversion of participation. Built on the labor and love of individuals, generative AI technology concentrated power. And then those in power convinced themselves they are powerless. San Francisco wants to abdicate all human agency and reduce us to making ourselves legible to an artificial bureaucratic god.

On the other hand, the data center protests are inspirational. Here you have a lot of people who are really upset for a complicated set of reasons about AI. It’s hard to say why they are so mad, and why this issue is so salient, but it’s undeniable that they are driving policy conversations. Their protests and organization have made them not legible, but unignorable.

AI doesn’t have to be exclusionary. We could build an AI that’s a public good. One that invites participation. One that involves known training data, participatory training data. We now know that if you train next-token predictors on collective intelligence, you end up with a quirky piece of software that speaks in natural language, solves impossible math problems, and does all of your coding. We can’t unsee that. But we don’t have to let a small group of whiny, weird people own it. We can use this insight to build a public infrastructure for collective, participatory intelligence.