Hi there, argmin readers! Today’s post is a live blog of Class 16 of my graduate seminar “Forecasting: A Critical Retrospective.” In this class we’re going to dig into the bizarre world of silicon sampling. I remain befuddled! The syllabus and list of past posts are here.
In 2024, at a secluded conference in the high desert, Kevin Munger told me about silicon sampling. The motivating idea is simple. LLMs have passed the Turing Test and are hence indistinguishable from people. So if we need to do some sort of measurement or experiment or forecast that involves talking to people, we can query LLMs instead. I scoffed at him: there can’t be people who take this seriously, can there? He laughed at my naivety. Here we are in 2026, and there are multiple hot startups and hundreds of research papers excited about the promise of silicon samples.
And I’m still confused why anyone thinks this is a good idea! But we’re going to spend a class period trying to figure it out. My impression is that quantitative social science is also confused, and this could be another example of AI doing what AI does best: forcing us to reckon with whether our disciplinary practices have been asking the right questions in the first place.
Our lectures on polling brought up a critical tension: Are polls measurements or forecasts? A measurement must be tied to some construct that exists and can be quantified. A forecast predicts some quantitative outcome that has not happened yet. “The percentage of likely voters who would vote for candidate X today” is much less real than “the count of ballots for candidate X.” Trying to predict a hard quantifiable object is very very different from trying to estimate a fuzzy psychological aggregate.
If you are trying to do prediction, there’s nothing wrong with modeling. No one says you need to talk to people to predict the outcome of an election. Indeed, many experts still insist that election outcomes are highly predictable from “fundamentals” like presidential job approval, economic growth, and incumbency.
In fact, as we described last Thursday, even the most committed pollsters use their own bespoke set of “fundamentals” to build predictive models that adjust their poll results. They have to correct for the fact that they can’t collect random samples and they have high response bias. They probably could just use their predictive models and not worry about the polling information at all. This will likely be very predictive of the election outcome, but it is not a measurement.
Now, what does it mean to “measure” public opinion? 100 years after Walter Lippmann, we still don’t have a satisfactory answer. We assume that there is some aggregate percentage of people who agree with some concept, and we measure this by nagging them with carefully worded surveys. While the central limit theorem says we only need to talk to a few hundred random people to estimate a binary response, a survey doesn’t tell us what that binary response means or represents. You need some nomological network to map the complex stuff happening in individuals to a meaningful population aggregate that could be used in policymaking, political campaigns, marketing, or the social scientific understanding of a public.
Here enters the allure of the Silicon sample. If a social scientist has a perfect simulator of people, they have perfect control over experimental conditions and can tweak variables to see how “people” respond. They can change demographics, personal backgrounds, and material conditions. They can create whichever publics they imagine. This lets them perhaps build and test theories of psychology, sociology, and politics without being slowed down by the messy burdens of reality.
Of course, this requires us to believe these “digital twins” are accurate simulations of reality. And I have no idea why we think they’re anything more than the mesmerizing chatbot illusion Weizenbaum called out half a century ago, fleshed out with dozens of generations of Moore’s Law scaling.
There are a bunch of papers that ask a variety of social scientific questions and come out with whatever they want to see. I’ve linked a working bibliography below. I liked the one from Pew, which found that silicon samples and real samples were different. Which one is right, though? Do we trust the answers from the online survey panel?
In these papers, you’ll see the strong prediction/measurement split. For prediction, LLMs are excellent. Just yesterday, Eva Vivalt published new work showing that LLMs can predict the outcomes of randomized controlled trials preregistered in the American Economic Association’s registry and published in 2026 with surprising accuracy. The academic disagreements are all about measurement, interpretation, and construct validity. The construct of a “person with demographics” is weird in and of itself, and these papers can’t figure out what to make of the fact that you can match survey conditional probabilities but find no diversity inside the made-up categories of people. LLMs think all computer science professors in their forties who write a blog respond the same way. But is this a problem with the LLM or the questions fed to the LLM?
I’ll probably come back to this topic in the next lecture blog. I’m excited to see how the students make sense of these papers. I’m also curious whether you, my readers, have read any other papers you found persuasive on this topic. I’m particularly interested in reading the bullish arguments for silicon samples. If you have some good ones, link them in the comments. I’m less interested in the topline results of these papers and more interested in the theory. The bibliography below is a live documentation of social scientists grappling with the purpose of their fields. They’ve yet to reach consensus.
Working bibliography
Lisa P. Argyle et al. (2023). Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3), 337-351. https://doi.org/10.1017/pan.2023.2
Joon Sung Park et al. (2024) Generative Agent Simulations of 1,000 People. https://arxiv.org/abs/2411.10109
Bisbee, J., Clinton, J. D., Dorff, C., Kenkel, B., & Larson, J. M. (2024). Synthetic replacements for human survey data? The perils of large language models. Political Analysis, 32(4), 401-416. https://doi.org/10.1017/pan.2024.5
Jamie Cummins (2025). The threat of analytic flexibility in using large language models to simulate human data: A call to attention. https://arxiv.org/abs/2509.13397
Binz, M., Akata, E., Bethge, M. et al. A foundation model to predict and capture human cognition. Nature 644, 1002–1009 (2025). https://doi.org/10.1038/s41586-025-09215-4
Athena Chapekis, Arnold Lau, Samuel Bestvater, Sono Shah, Andrew Mercer, and Aaron Smith (2026). Can AI Stand In for Human Survey-Takers? Not Really. Technical Report. Pew Research. https://www.pewresearch.org/data-labs/2026/09/30/can-ai-stand-in-for-human-survey-takers-not-really/
Leif Weatherby and Benjamin Recht (2026). This Is What Will Ruin Public Opinion Polling for Good. The New York Times. https://www.nytimes.com/2026/04/06/opinion/ai-polling.html?eafs_enabled=false
Jiawei Chen et al. (2026). Towards real-world human behavior simulation: Benchmarking large language models on long-horizon, cross-scenario, heterogeneous behavior traces. arXiv preprint. arXiv:2604.08362.
Ashokkumar A, Hewitt L, Ghezae I, Willer R. Large language models can predict the results of social science experiments. Nature. 2026 Aug;656(8126):115-122. doi: 10.1038/s41586-026-10742-x. Epub 2026 Jul 8. PMID: 42420458.
Clinton, Joshua et al. (2026). Public Opinion in the Age of AI. APSA Preprints. doi: 10.33774/apsa-2026-1rg21-v5.
Eva Vivalt (2026) Can Artificial Intelligence Forecast Treatment Effects in Economics? Technical Report. https://evavivalt.com/papers/Can-AI-Forecast-Treatment-Effects-in-Economics.pdf
Ian Hacking (2006). Making Up People. The London Review of Books. Vol. 28 No. 16. https://www.lrb.co.uk/the-paper/v28/n16/ian-hacking/making-up-people
Since LLMs are by definition based 100 percent on what is written down, they are gloriously ignorant of everything about the human condition that isn't written down.
Suppose an LLM were asked, “Will people like New Coke or Old Coke better?” If an LLM were conscious of its limitations, it would answer, "How on earth would I know?”
the fake respondents at least answer the survey, which is more than i do