Hi there, argmin readers! Today’s post is a live blog of Class 8 of my graduate seminar “Forecasting: A Critical Retrospective.” The syllabus and list of past posts are here.
A bizarre central tenet of “rational decision-making” is that all optimal decisions can be made by computing an appropriate cost-benefit analysis. I riff on this in the introduction to The Irrational Decision, and always lead with more absurd examples when I talk about the book. Should you have a surgery? Should you force your kid to take violin lessons? Should you go for it on 4th down? According to the tenets of rational choice theory, you can answer all of these questions by forecasting a dollar value and probability of every outcome.
With these facts in hand, optimal decision making is merely a mechanical chain of sums and multiplications. This is ludicrous if you think about it for two seconds. And yet it’s become a standard social convention that utilitarian calculation is not only possible but the optimal way to live your life, run a business, or govern a nation.
In today’s class, we try to get at the roots of where this came from and how it became institutionalized. My two favorite references on the history are Theodore Porter’s Trust in Numbers and Elizabeth Popp Berman’s Thinking Like an Economist, both of which trace the history in the United States.
Porter starts before the war, looking at how cost-benefit analyses were formalized to justify water projects by the Army Corps of Engineers. He has a nice short article summarizing the book’s in-depth study. Water projects were crucial for preventing flood damage, routing water to farms, and making waterways more navigable. However, they were also classic pork-barrel projects, where elected officials would funnel money back to their districts. The Corps looked for means to “remove the politics” and demonstrate that each project was worth doing. They settled on cost-benefit analysis, establishing a rigorous system to enumerate all of the potential upsides and itemize all of the potential costs.
These calculations were eventually mandated in the 1936 Flood Control Act:
“...the Federal Government should improve or participate in the improvement of navigable waters or their tributaries, including watersheds thereof, for flood-control purposes if the benefits to whomsoever they may accrue are in excess of the estimated costs, and if the lives and social security of people are otherwise adversely affected.” (italics mine)
Cost-benefit analyses would leave them with a simple, clean, unitary number — the ratio between these costs and benefits — that they could present for project approval. All of the complexity could be reduced to two digits. These digits sufficed to make governance decisions. Significant expertise was needed to ensure these calculations held up to adversarial scrutiny. As Porter writes:
“Objectivity, then, meant above all the standardization of quantitative methods and the training up of people capable of performing them. Every failure of clarity, every gap in the reasoning, every loophole that left space for the quantifier to alter the results in a preferred direction, was a potential weakness, which opponents of the agency were certain to exploit, often in hearings before judges and administrators who would probably be ignorant of the fine points of economic quantification.”
Interestingly, no economists were consulted in constructing the estimates. The engineers prided themselves on their ruthless objectivity and ability to decouple their preferences from the cold hard facts. Moreover, the public preferred cost-benefit analyses to opaque expert judgment. Standard, transparent processes feel like they rule out arbitrariness and capriciousness of bureaucrats. Porter casts cost-benefit analysis as “a quantitative decision technology, practiced mainly in public bureaucracies, often in a highly politically-charged context.”
Popp Berman details how this technology spread through the government, with the establishment of various executive-branch offices staffed by experts to oversee complex problems like healthcare and education. It became institutionalized in policy schools, founded in the 1970s to provide graduates to staff said agencies.
Fast forward to the present, and we just take these cost-benefit analyses for granted. They give an institutionalized illusion of objectivity, but of course all of the calculations are subject to institutionalized norms of expert judgment. These norms tell you where you can commit rounding errors, ignore missing data, or disregard the unenumerable. But these are just institutional norms, and they don’t really hold up to scrutiny. As Larry Lohman details, the “objective” methods of institutionalized cost-benefit analysis are riddled with value-laden assumptions, and objectivity rests on absurd ideas of commensurability and the ability to price all preferences.
Moreover, Charles Manski describes the incredible uncertainty inherent to cost-benefit calculations.1 Manski notes that experts all know these uncertainties are present but choose not to report them for political reasons. You’ll often find cost-benefit analyses reported to three or four digits of precision, creating a further illusion of precision. Manski has a long list of critiques:2
Conventional certitude: A prediction that is generally accepted as true but is not necessarily true.
Dueling certitudes: Contradictory predictions made with alternative assumptions.
Conflating science and advocacy: Specifying assumptions to generate a predetermined conclusion.
Wishful extrapolation: Using untenable assumptions to extrapolate.
Illogical certitude: Drawing an unfounded conclusion based on logical errors.
Media overreach: Premature or exaggerated public reporting of policy analysis.
Together, these conventions conspire to communicate certainty where there is none. They justify decisions as rational by sweeping all of the uncertainty under the rug.
We’ll cover uncertainty quantification in later classes.
My impression from economist friends is that Manski has a longer list of critiques than what appears in his published works, but he is too prideful to go full Nicholas Polson and have AI air all of his grievances.
Keynes famously wrote about the problems with expected utility analysis:
"By "uncertain" knowledge, let me explain, I do not mean merely to distinguish what is known for certain from what is only probable. The game of roulette is not subject, in this sense, to uncertainty… Even the weather is only moderately uncertain. The sense in which I am using the term is that in which the prospect of a European war is uncertain, or the price of copper and the rate of interest twenty years hence, or the obsolescence of a new invention, or the position of private wealthowners in the social system in 1970. About these matters there is no scientific basis on which to form any calculable probability whatever. We simply do not know."
but then rather glumly concluded:
"Nevertheless, the necessity for action and for decision compels us as practical men to do our best to overlook this awkward fact and to behave exactly as we should if we had behind us a good Benthamite calculation of a series of prospective advantages and disadvantages, each multiplied by its appropriate probability, waiting to be summed."
He does not seem, to me, to be completely at ease with this conclusion. But he does say it!
I'm wondering how you choose what surgery to have or whether to force your child to take violin lessons. Surely you weigh up the pros and cons for that decision, which is a type of cost benefit analysis.