Hi there, argmin readers! As the fall semester picks up, posting volume will, too. So I’m going to commit to writing short descriptive headers to help you sort through the different threads.
Today’s post is by Jessica Dai, writing her second post on the microconference on “public feedback for AI and beyond” that she ran at UC Berkeley in August. -Ben
Today, I’ll continue blogging readouts for the AI & public feedback workshop. As a reminder, here’s some of my motivating logic:
Proposition 1. “The public” has interesting and important things to say about their experiences with AI, but are not typically listened to by decision makers.
Proposition 2. “Evaluations” — and aggregated information, more broadly — are useful, in the sense that they can influence consequential decisions.
Corollary. AI evaluations from public feedback can be a meaningful way to “do something” about the emergent misalignment between those who control AI development and literally everyone else.
In the first post about the workshop, I wrote about who, or what, “the public” refers to. Today, I’ll continue with “Proposition 1,” and try to reason through some of the challenges in the process of actually providing feedback.
Easing the burdens of “participation.”
What does a participant experience in the process of sharing information? As an economist might say, engagement is ‘costly’ — this is why, for instance, human subjects studies typically compensate participants, and why the “representativeness” of the people who self-select to participate in light of these costs remains a central challenge (I discussed some of these issues in the prior post).
But costs can manifest concretely in ways that are difficult to quantify by economic measures. Moreover, they are not only about the initial decision to participate; the process of participation itself entails challenges that can affect the outcomes of data collection. For example, Samantha Dalal, the researcher with WAO, emphasized the importance of understanding the barriers to participation that might be specific to the relevant “slice of the public.” If data will be collected via a mobile app, it should be available on a variety of platforms (including older versions of Android and iOS), and small enough to be feasibly downloaded to a phone with limited storage or on limited cellular data; online forms should be readable on mobile browsers. While these factors may seem like basic design fundamentals, they are also a reminder of the friction inherent to collecting “real” data.
Some of this friction also involves active support from facilitators that shapes the feedback itself. Some of the preliminary findings in Humphrey Obuobi’s presentation about BLOOM’s work in central Oregon were results from a Polis-like platform, where participants could provide statements of their own positions on various topics as well as engage with previously-written statements (for example, by indicating support or disagreement). Some workshop attendees noticed that these statements varied widely not just in content, but style, with some statements being noticeably longer, more detailed, and having more complex sentence structure. Humphrey explained that BLOOM uses a mix of participant-generated and facilitator-written statements in the deliberation process; the latter can synthesize existing participant-generated positions, while also help support participants to develop finer-grained perspectives.
Both of these are examples of ways that facilitators can be actively involved in the process of collecting feedback, rather than passively waiting for data to arrive; they also suggest that this involvement can ultimately result in higher-quality feedback. Another lens for thinking about these examples is legibility. If data collection platforms are poorly designed, then there will be members of the public who are “illegible” to facilitators; meanwhile, the facilitator-written statements are directly increasing the “legibility” of participants’ original statements.
Pursuing and enabling legibility in this way feels important; why? Ben’s talk provides one conceptual answer. He spoke about the quantification trap, wherein the demand for legibility is the first in a series of dominoes that ultimately requires power to be enacted only through “objective” numbers, and conversely, endows numbers (“objective,” or otherwise) with power. I’ll discuss the latter part of this statement in a later post, but I want to highlight one of Ben’s arguments (really, Graeber’s) that his post glosses over: One vector through which people experience “structural violence” is that they must work to make themselves legible to the decision makers who hold power over their lives. Any illegibility, or irregularity, excludes them from bureaucratic accounting; in Graeber’s account, this can be dangerous and, in the worst case, subject them to material harm.1
When viewed from the perspective of legibility, therefore, the various ways in which facilitators can make participants’ lives easier is also a way to prevent their exclusion from the evaluation, and whatever future conversation this evaluation enables. Some people might find it otherwise difficult to make themselves or their feedback legible, and while their resulting exclusion might not result in anything as dramatic as material danger, it still feels worthwhile for facilitators to ease whatever burdens participants face to make themselves heard.
The costs of legibility.
It is not lost on me that there are also costs to legibility. For instance, many of the projects that involve analyzing transcripts can be fairly invasive from a privacy perspective, even when transcripts are donated voluntarily, and analysis does not rise to the level of privacy violations. Similarly, any “monitoring” approach, whether it’s of product usage or of social media (as in my r/ChatGPT paper), also essentially amounts to surveillance that subjects users to a level of scrutiny they may not feel entirely comfortable with. One of the other recent California bills that Deb Raji discussed was SB243, which requires chatbot providers to track and disclose the number of conversations that mention suicide or self-harm; while requirements for such disclosures seem important for accountability and transparency, there are obvious privacy tradeoffs as well.
Perceived discomfort seems to matter. Evi Micha, from USC, has done a lot of theoretical and algorithmic work on ensuring representativeness in citizens’ assemblies (e.g., demographic, regional, etc.). Representativeness might reasonably be thought of as a necessary precondition for such assemblies to produce high-quality discussions that can be effective proxies for viewpoints of the population as a whole. In her talk, she shared findings from recent work with a different methodological perspective: how do participants actually perceive “representativeness” in assembly selection? Perhaps unsurprisingly, people generally prioritized representation in the sense of political or issue-based agreement; they would be happiest to be represented by someone who shared their views on the topics to be discussed.
What was more surprising was the degree to which people seemed to hate the idea of representativeness measured via demographic attributes. Evi shared some of the free-text commentary from study participants, and I can’t over-emphasize how strongly negative this feedback was. Participants seemed to take offense at the very idea that demographics might have any correlation, and therefore relevance, to their views on substantive issues. While we know that demographics and political views often do actually correlate on the population level,
it seems like it’s worth considering how it feels to an individual for their worthiness as an assembly participant to be reduced to immutable demographic characteristics.
There’s of course a bit of a chicken-and-egg problem, in the sense that it would be difficult for any facilitator to select assemblies that were fairly representative of issue-level perspectives before even knowing what those perspectives might be or how they might be distributed across the population. It’s therefore understandable that demographics, which are easily “legible” a priori, become the fallback mechanism for ensuring “representativeness,” but this cheap legibility might be exactly what participants are chafing against.
The necessity of legibility.
In some cases, it might be necessary to explicitly impose the burden of legibility on participants, especially when the goal is to shift power to them. Ira Globus-Harris spoke about their work designing a “bias bounties” mechanism for auditing a deployed machine learning model. Specifically, the mechanism allows any population subgroup that perceives the deployed model to be inaccurate on them to submit a “bounty”; this brings the model developers’ attention to performance on that particular subgroup, allowing developers to iteratively improve the model in the future.
This framework is compelling, because it effectively identifies the power in the public as due to knowledge about their own experiences (i.e., groups of people can determine when the model is inaccurate on them specifically), and leverages that knowledge while also recognizing that only model developers have the power to actually change the model itself. The catch is that, for any given subgroup, model developers can only improve the model for that subgroup if it is statistically possible to do so. (One major takeaway from the fair machine learning research of the late 2010s is that what often appears as “bias” is often actually due to “variance” — it can be inherently harder to predict Y from X in some subgroups — so for a fixed measure of performance there may be subgroups for which that measure can never improve, no matter how complex the underlying model.)
Ira’s mechanism therefore requires that a submitted “bounty” for a given subgroup includes not just a statement that the model performs poorly on that subgroup, but also evidence that it is even possible to do better on that subgroup. This makes sense, because no model developer, no matter how benign, can do better than what is statistically achievable; on the flip side, as long as improvement is possible, “bounties” of this form allow the model developer a straightforward algorithmic approach to incorporate the reported information to implement the improvement. On the other hand, this also asks a lot of potential “bounty hunters”, including, perhaps, collecting their own data and training their own model — a requirement that might well be practically infeasible.
In this case, the mechanism specifies exactly what it means to be legible, without explicitly providing a pathway for participants to meet those criteria. Even so, I want to emphasize that the specification itself is, already, an invitation to the public. Since the goal is to change the deployed model, participants must share feedback in a way that can be legible to the model developer. The specification of what counts as “legible” is a starting point for helping participants to be seen the way they want to be seen — and ultimately, for making it clear that it is their voices we want to hear in the first place.
More accurately, Graeber argues that illegibility/irregularity itself can be punished by violence. Exclusion from accounting, while being the most salient part of this argument for our purposes, isn’t the focus for him.
