Acting On the Unknowable

Meehl's Philosophical Psychology, Lecture 8, part 7.

This post digs into Lecture 8 of Paul Meehl’s course “Philosophical Psychology.” You can watch the video here. Here’s the full table of contents of my blogging through the class.

Lecture 8 closes with the obvious but profound assertion that some questions can’t be answered, at least not given the current stage of development. Meehl notes that a question can be perfectly well posed, theoretically sound, and still unanswerable. We might not have the theoretical structure of auxiliaries necessary to design a good experiment. We might not have developed the instruments needed for appropriate observation or control. Some theory testing requires decades of development of intricate conceptual scaffolding.

For example, Einstein posited the existence of gravitational waves in 1916, but it took over one hundred years to develop the appropriate instrumentation and analysis methods to see them at LIGO. Miescher discovered DNA in 1869, but it took, among many other things, the development of crystallography and X-ray diffraction, the theoretical understanding of amino acids, and the experimental measurement of amino acid balance to understand its structure.

Meehl argues that the social sciences too often ignore this unanswerability issue. He attributes the oversight in psychology to a weird combination of remnant logical positivism and reliance on null hypothesis significance testing. Roughly, he says that if you believe that concepts are defined only if we can give procedures to measure them, and that theories are meaningful only if they are testable, and that all tests are NHSTs, you end up dropping theories that can’t be validated by our current statistical tool kit. 

But Meehl leaves out something even more deleterious: research in social sciences is driven by action bias. Methodological critiques are always met by the canned response, “But we have to do something!” The world won’t end if we have to wait another century for a particle collider more powerful than the LHC, but many will suffer if we don’t understand the causes of the opioid crisis, the impact of abortion bans, or how to nonpharmaceutically slow the spread of infectious diseases. We have to do something!

That something is, unfortunately, often just doing more of the same. We fool ourselves by thinking that any meaningful question must be answerable now. We just have to define our terms operationally so they are verifiable. We can verify things through the right identification strategy. We know there’s a problem of individual differences and sampling errors, but our estimators and stat packages will handle these. So any meaningful question must be answerable at this time, right? Well, sadly, no.

But if our social-scientific tools, reducing everything to Fisher’s Exact Test, aren’t up for the job, what do we do? Even if we have perfectly valid questions that we can’t answer with cold, statistical empiricism, we still have to make decisions. We still have to do something. So what do we do?

Shreeharsh Kelkar sent me a clarifying paper by Daniel Sarewitz “How science makes environmental controversies worse,” that proposes a path past technocracy. Though Sarewitz’s title focuses on climate science, he gives examples in agriculture and political science that buttress a broader argument.

His point follows from a Meehlian foundation. Science is always uncertain, and given the persuasions of any particular scientist, scientific theories can be attacked from a variety of different arguments about corroboration. Attack the other scientists’ methods. Attack the ceteris paribus clauses. Attack the instruments. Et cetera. Research programs in science advance by ignoring the mountains of uncertainty and focusing on generating new facts regardless. But uncertainty will always remain as long as there’s will of a large enough group of other scientists to keep the arguments going.

Nothing provides such will more than politics. This means that making a scientific question political, asking if science says we should do something, only increases uncertainty. We turn a problem of is into a problem of ought. As long as there are camps on either side, we sink into a morass of bothsiderism. The scientific method almost ensures you can never get a scientific answer to a political question.

Sarewitz’s conclusion is counterintuitive, but it points to the disutility of uncertainty quantification in policymaking. Scientists can see whatever they want to see if they try hard enough. Presentations from two competing camps only result in more questions. More heat. Little clarification.

But we have to do something! So what do we do? Sarewitz concludes that decisions under uncertainty are thus necessarily more about ethics and values than about optimization and uncertainty quantification. We decide based on what ought to be true. On what we’d like to make real. On the sort of society we want to be. Statistical measurement can’t tell us any of these things. And it never will be able to.

As a way forward, Sarewitz oddly enough ends up agreeing with Karl Popper, calling for some kind of piecemeal social engineering with incremental agile interventions and scientific monitoring of policy implementation. Decision-making about bit issues might be best served by the small. Quoting Rayner and Malone, Sarewitz argues “Sustainability is about being nimble, not being right.” I may be too primed to receive these messages from Popper, Sarewitz, Malone, and Rayner, as I reached similar conclusions in this blog series on the tradeoffs between action and impact.

Try different things in different places. Be willing and able to nimbly change course. As best you can, devise measurements to make sense of policy impacts in a deeply interconnected mess. This micro-policymaking still requires help from a research community to guide how to intervene, what to measure, and what is measurable. It also requires society to accept some things are unmeasurable and some questions are unanswerable. And when we find ourselves with scientifically unanswerable questions, we need to be open about making decisions based on our values, not based on our science.