I think the single thing that most caught my attention and gave me an "aha" moment was the comparison of p-value testing to the chart where they switched what they were feeding the mice. I do think people get stuck in that mentality of framing every study as a hypothesis test, and seeing a concrete alternative was really helpful.
Meehl's equation is certainly something beautiful to beckon. As you write so aptly, it is a very elegant framework in which one can reconcile social studies of science with logical positivism.
I found also very interesting the points of view offered on the meaning of "prediction" and "inference". I think that it is useful to recall the notions of "intuitive mathematical theories" in contrast with "axiomatic theories" which include abstract algebras. The latter are useful for computation but, importantly, truth is not a thing in them - that is something that can only be assessed in a particular interpretation grounded on the facts of an intuitive theory. Or in other words, I can compute the state x_n that follows from initial condition x_0 and f(x,u) - a mathematical model of a (controllable) physical process - and control sequence U=u_0, u_1, ..., u_{n-1}. But the fact that I can do the computation has actually little to no bearing to whether the statement "x_n is reachable from x_0 via U" is true. That is something that can only be assessed by the intuitive theory one uses to formalize the interpretation of x's, u's and f's.
Axiomatic theories are good for computation but they cannot provide (by design) insights into the ultimate truth of the statements which we can prove to be theorems in those theories. The actual insight comes when one cannot prove a statement to be a theorem, while knowing its truth under certain conditions in the intuitive theory.
I was also deeply struck by the analogy between current and 35-year old critiques of science and research. I still think there are ways forward, and I agree with you they have to do with "degrowing" scientific outputs. I am under the impression that many institutions and colleagues around the world would agree that the value of having a publication on a certain venue has steadily decreased over time in many fields. We should foster meaningful communication (as in face-to-face in person or digitally) between researchers, and communal, transparent mechanisms to give high-quality, detailed, actionable feedback to fellow researchers. For instance, the recent launch of alphaxiv.org filled me with joy. I am not sure it is the solution, but I reckon is a bold step in the right direction.
my current view is that it is really hard to look at a numerical expression of probability (even if with confidence intervals etc) and not think of it as being both precise and accurate — even if the probability number was created with the intention of being qualitative and impressionistic. as a communication tool for uncertainty, the main affordance of numbers seems to be to create the impression that the uncertainty is [precisely and accurately quantifiable] even if it is not. the mismatch between [an intended-to-be qualitative numeric probability expression of true uncertainty] and [how a numeric probability is easily/default interpreted as accurate and precise] is problematic when the interpretation guides how people act on the uncertainty in question.
i've been writing on this and other problems associated with unquantifiable forms of not-knowing: https://vaughntan.org/notknowing
Still trying to figure out whether we agree substantively on probability or not. From where I stand, Probability 1 also covers mathematical models relying on such idealizations as continuity, countable and uncountable infinity, etc. So there’s room for quantitative constructs in Probability 1, but they quantify propositions about the domain of Probability 2. The bridging laws are given by various limit theorems, fair betting odds, and what have you.
We agree that Probability 1 *can* be quantitative. But the realization that it can also *not* be quantitative was pretty revelatory for me. It's an obvious point, but helpful for me understanding how people use probability, and why it is such a dangerous tool in the hands of sophists.
You're after a harder question though. Idealizations *have* to be probability 1 because they are metalinguistic. How we interface the quantitative with qualitative parts of probability 1 space is also a fascinating question. The best treatment on it I've read has been your substack!
The most important distinction for me is Carnap's: Probability 1 is uncertainty in belief and Probability 2 is relative frequency. But a really important point is this distinction is not as absolute as it seems. It is certainly not as absolute as the religious probabilists make it out to be. We can disagree about what counts as Probability 1 and what counts as Probability 2! (Matt Hoffmann and I got into fights about where to bucket different concepts a few blogs back.) On the boundary between the quantifiable and unquantifiable---the verifiable and the unverifiable---I can make no strong claims.
Yep, this is right on target. The fact that probabilities are relative magnitudes subject to Kolmogorov's axioms is what allows one to make qualitative statements -- e.g., by exploiting things like independence (which is a structural assumption that you may have to defend), additivity, etc. It seems that what Carnap was after with his "state description of the world" idea was something like postulating the probabilities of some primitive events as axioms in a formal system or as synthetic propositions (in the Kantian sense), so that all other probabilities can be derived deductively and can be viewed as analytic propositions. I think Meehl has demolished that whole edifice pretty conclusively.
Here's my overview of the history which informs the way I read this series
The way we (theorists and people in general) think about probability has evolved. Before Pascal there was just Chance/Fortune. Gamblers must have had some idea that two dice were more likely to sum to 7 than to 2 or 12, but no way to express it. But once you apply symmetry you have a useful tool, at least for gambling. It's taken quite a while to nail this down, as shown by puzzles about the Principle of Insufficient reason.
Frequentism gives you another form of symmetry - lots and lots of indistinguishable cases, but ends up as a dead end. Still, it's the only notion available when classical hypothesis testing emerges. And most people still didn't understand probability except in the context of betting.
Classical hypothesis testing is basically useful for answering only one question. In a tightly controlled experiment, could an observed difference between treatment and control groups have arisen by chance. (Taking a break, so I'll post this bit and come back later).
I think the single thing that most caught my attention and gave me an "aha" moment was the comparison of p-value testing to the chart where they switched what they were feeding the mice. I do think people get stuck in that mentality of framing every study as a hypothesis test, and seeing a concrete alternative was really helpful.
Meehl's equation is certainly something beautiful to beckon. As you write so aptly, it is a very elegant framework in which one can reconcile social studies of science with logical positivism.
I found also very interesting the points of view offered on the meaning of "prediction" and "inference". I think that it is useful to recall the notions of "intuitive mathematical theories" in contrast with "axiomatic theories" which include abstract algebras. The latter are useful for computation but, importantly, truth is not a thing in them - that is something that can only be assessed in a particular interpretation grounded on the facts of an intuitive theory. Or in other words, I can compute the state x_n that follows from initial condition x_0 and f(x,u) - a mathematical model of a (controllable) physical process - and control sequence U=u_0, u_1, ..., u_{n-1}. But the fact that I can do the computation has actually little to no bearing to whether the statement "x_n is reachable from x_0 via U" is true. That is something that can only be assessed by the intuitive theory one uses to formalize the interpretation of x's, u's and f's.
Axiomatic theories are good for computation but they cannot provide (by design) insights into the ultimate truth of the statements which we can prove to be theorems in those theories. The actual insight comes when one cannot prove a statement to be a theorem, while knowing its truth under certain conditions in the intuitive theory.
I was also deeply struck by the analogy between current and 35-year old critiques of science and research. I still think there are ways forward, and I agree with you they have to do with "degrowing" scientific outputs. I am under the impression that many institutions and colleagues around the world would agree that the value of having a publication on a certain venue has steadily decreased over time in many fields. We should foster meaningful communication (as in face-to-face in person or digitally) between researchers, and communal, transparent mechanisms to give high-quality, detailed, actionable feedback to fellow researchers. For instance, the recent launch of alphaxiv.org filled me with joy. I am not sure it is the solution, but I reckon is a bold step in the right direction.
I think this series of reflections are interesting enough that I could imagine it as a book. Has anyone approached you about that?
Not yet, but glad to hear you’d be interested!
my current view is that it is really hard to look at a numerical expression of probability (even if with confidence intervals etc) and not think of it as being both precise and accurate — even if the probability number was created with the intention of being qualitative and impressionistic. as a communication tool for uncertainty, the main affordance of numbers seems to be to create the impression that the uncertainty is [precisely and accurately quantifiable] even if it is not. the mismatch between [an intended-to-be qualitative numeric probability expression of true uncertainty] and [how a numeric probability is easily/default interpreted as accurate and precise] is problematic when the interpretation guides how people act on the uncertainty in question.
i've been writing on this and other problems associated with unquantifiable forms of not-knowing: https://vaughntan.org/notknowing
Still trying to figure out whether we agree substantively on probability or not. From where I stand, Probability 1 also covers mathematical models relying on such idealizations as continuity, countable and uncountable infinity, etc. So there’s room for quantitative constructs in Probability 1, but they quantify propositions about the domain of Probability 2. The bridging laws are given by various limit theorems, fair betting odds, and what have you.
We agree that Probability 1 *can* be quantitative. But the realization that it can also *not* be quantitative was pretty revelatory for me. It's an obvious point, but helpful for me understanding how people use probability, and why it is such a dangerous tool in the hands of sophists.
You're after a harder question though. Idealizations *have* to be probability 1 because they are metalinguistic. How we interface the quantitative with qualitative parts of probability 1 space is also a fascinating question. The best treatment on it I've read has been your substack!
The most important distinction for me is Carnap's: Probability 1 is uncertainty in belief and Probability 2 is relative frequency. But a really important point is this distinction is not as absolute as it seems. It is certainly not as absolute as the religious probabilists make it out to be. We can disagree about what counts as Probability 1 and what counts as Probability 2! (Matt Hoffmann and I got into fights about where to bucket different concepts a few blogs back.) On the boundary between the quantifiable and unquantifiable---the verifiable and the unverifiable---I can make no strong claims.
Yep, this is right on target. The fact that probabilities are relative magnitudes subject to Kolmogorov's axioms is what allows one to make qualitative statements -- e.g., by exploiting things like independence (which is a structural assumption that you may have to defend), additivity, etc. It seems that what Carnap was after with his "state description of the world" idea was something like postulating the probabilities of some primitive events as axioms in a formal system or as synthetic propositions (in the Kantian sense), so that all other probabilities can be derived deductively and can be viewed as analytic propositions. I think Meehl has demolished that whole edifice pretty conclusively.
will there be many modifications from your optimization book? will audit be possibly considered?
Here's my overview of the history which informs the way I read this series
The way we (theorists and people in general) think about probability has evolved. Before Pascal there was just Chance/Fortune. Gamblers must have had some idea that two dice were more likely to sum to 7 than to 2 or 12, but no way to express it. But once you apply symmetry you have a useful tool, at least for gambling. It's taken quite a while to nail this down, as shown by puzzles about the Principle of Insufficient reason.
Frequentism gives you another form of symmetry - lots and lots of indistinguishable cases, but ends up as a dead end. Still, it's the only notion available when classical hypothesis testing emerges. And most people still didn't understand probability except in the context of betting.
Classical hypothesis testing is basically useful for answering only one question. In a tightly controlled experiment, could an observed difference between treatment and control groups have arisen by chance. (Taking a break, so I'll post this bit and come back later).