The ouroboros of discovery and justification
Meehl's Philosophical Psychology, Lecture 3, Part 1.
This post digs into Lecture 3 of Paul Meehl’s course “Philosophical Psychology.” You can watch the video here. Here’s the full table of contents of my blogging through the class.
The fact that scientists place so much value in prediction should have already clued us in that we can't separate the sociology of science from the logical structure of science. We’re not just trying to figure out scientific truths, but what scientists like to think are reasonable proxies of scientific truths, what philosophers call verisimilitude. But just because we can’t separate scientific practice from scientific evidence doesn’t mean we can’t think of them as two entities. Meehl’s third lecture uses Reichenbach’s contexts as an axis to understand their interplay.
Reichenbach argues for a dichotomy between the "context of discovery” and the “context of justification.” The context of discovery concerns the psychological and sociological origin of scientific ideas. It concerns what was in the mind of a scientist when they invented a theory. The “context of justification” concerns how scientific claims are evaluated. This inspects the state of a theory with respect to the evidence base through logic, statistics, and data analysis.
Meehl describes Reichenbach’s core example of a clear context of discovery. Apparently, and apocryphally, August Kekulé had a dream of the ouroboros. This dream led him to posit the shape of a benzene ring. Surely, the structure of Benzene isn’t evaluated based on what Kekulé dreamed before writing it down. The dream is the context of discovery. The chemical evidence for the ringed structure of Benzene based on observed isomers is the context of justification. In this story, the two have nothing to do with each other.
But Meehl argues that science is never this clean. There is always a messy bleed between the contexts of discovery and justification in the scientific record. It’s critical to recognize and understand their interplay. You have to pay attention to the psychology of experimenters when evaluating the evidence, but it doesn’t mean there isn’t logical structure underneath the evidence. There is an inseparable discovery-justification axis along which there are many shades of gray.
This is why Meehl calls his class metatheory. It’s a mix of empirical history and philosophy, and you need both to evaluate and process scientific evidence. Meehl spends most of Lecture 3 explaining how separating the contexts is impossible and how this should impact evidence appraisal.
Meehl approaches this question as a clinician, not as a philosopher. He was a practicing psychologist, after all. Clinicians have to act. They have to act based on the evidence base before them. Not acting, as I belabor on this blog, is an action in and of itself. How can the clinician decide what to do in their clinic or lab? How can engineers decide which results to incorporate in their designs? How can a lab scientist decide which protocols they want to build upon or replicate?
Even the most curious practitioner can’t make decisions simply by looking at the scientific record. We all know that social and psychological factors influence what you see in published papers, reviews, and textbooks. These factors—and your limited bandwidth—mean you are only seeing a part of the assembled evidence. When assessing the literature, you are solving a missing data problem. Knowing that there is missing data or why there is missing data can and should affect your inferences. If you have knowledge about the data-generating process—in this case, the process that results in conference papers, journal articles, and tweets—you should use this in your assessment of any given intervention. In assessing the current state of the literature, you must ask, “Why am I seeing what I am seeing?”
But this gets us into an uncomfortable position. If you know something about the psychology and sociology behind an evidence base, Meehl argues it’s perfectly rational to use this information to weigh the evidence. You don’t have to be a Bayesian to agree. A rational understanding of science requires accounting for the varied biases of scientists’ training, beliefs, interpersonal relationships, fads, etc. These strongly influence the scientific record. Meehl gives three examples of such psycho-social dispositions: dispositions of individual experiments or labs, dispositions of the literature in any given field, and dispositions of the funding of science. They are important enough to devote a blogpost to each.
But before I turn to these, I wanted to make one important point. Meehl notes that these psycho-social events are unavoidable, though we’d probably be better off without some of them (like petty rivalries or secular quasi-religious beliefs in superintelligence). However, one particular behavior must be firmly excluded and harshly punished when discovered: Faking data.
“Making an error in reasoning is frowned upon, but one can be forgiven. Making an error in your mathematics can be corrected. Somebody comes along and sees that there's a slip here in the formalism. But one of the unpardonable scientific sins is faking the data. What makes that such a gross sin is not only that it has a greater moral flavor to it than making an honest mistake, but the fact that it's not easily corrigible.”
Faked data means scientists and practitioners must grapple with a deliberate error in their corpus of facts. Fraudulent facts can pollute inferences and decisions for years before the community uncovers misbehavior or decides these should be removed from the corpus. Either way, as Meehl says, lying is not only morally wrong, but it’s actively harmful to the reconstruction of a scientific verisimilitude. It’s easier to fill in the gaps where information is missing than to remove dishonesty where it has been certified.
on relationship between science and predictions, i think you might enjoy freeman dyson's musings on the topic. loosely quoting (~4:40) "what scientists do is to arrange things in an experiment to be as unpredictable as possible, and then do the experiment... you might say that if something is predictable, then it's not science". in some ways i feel this hints at weighing the "context of discovery" (aka coming up with the hypothesis) just as much as that of the justification.
https://www.youtube.com/watch?v=8xFLjUt2leM
on fake data, how does one think about 'noise' in the way of stochastic processes or such, where the real data is hard to measure with existing equipment? side note, i wonder what Meehl (and you) would think about the increasing use of 'synthetic datasets' ubiquitous in the current paradigm of internet scale datasets?