I'm curious to hear you thoughts on Hume's "An Enquiry Concerning Human Understanding" particularly section VII on "necessary connection". Hume ultimately relegates "causality" to a metaphysical concept and suggests all we can reason on is prediction. The field of "causal" inference, particularly in the social sciences where data is observational and collected from a complex system, seems to be very confused about what statistics can and cannot do. Terms like randomization, ignorability, and stable unit treatment values are invoked with almost liturgical regularity, as if their mere presence absolves one from deeper epistemological scrutiny. My field of economics often feels like a causality-themed masquerade ball. Conversations with applied microeconomists these days often resemble a Carrollian tea party: elaborate, incoherent, and indifferent to the actual question at hand. I'm not sure how to best dislodge some of the more ritualistic invocations of “identification” and provoke a more coherent discussion about what we’re actually doing when we claim to estimate causal effects. I'm inclined to offer Bruno de Finetti as the ultimate corrective, a reminder that coherence trumps counterfactual fantasies.
Lastly, I'm not sure how the average treatment effect became the center of focus in "causal" studies since we are ultimately interested in who (which subgroups) specifically might benefit from a treatment. Summary results on clinical trials like the "average treatment effect" may be non-representative of the treatment effect for a typical patient in the trial. Wouldn't a cluster based analysis on benefits and risk be more appropriate?
I like how we've spent 300 years trying to convince ourselves that Hume was wrong. We're still at it! :)
I 100% agree with you about observational causal inference. I have written about it before and considered writing another piece this morning, but got sidetracked in some questionable public health statistics and ran out of time. Perhaps on Monday!
Perhaps the worst thing about observational causal inference is the idea that you can reduce all causation to an RCT, be it real or imaginary. This gold standardization is the original sin that can't be corrected with robustness checks.
And yes, I also agree that the average treatment effect is indeed an odd metaphysical object. If we ignore process and instead hyperfocus on association, we convince ourselves that we need the law of large numbers to distinguish reality from hallucination. But not all that is learned from past experience is statistical. You don't need statistics to know that penicillin is effective...
There is certainly some math that claims to study process (stochastic processes, dynamical systems, etc.). But I also think these fields are only marginally helpful for describing complex feedback systems.
I think this is most easily illustrated with the function concept, which on the set theoretical definition is already essentially static. Sure, there are functions which depend on “time”, but in the end they too are just sets of tuples. Now in your “distribution shift” case what you want is a function whose extension changes in time, but the function should remain the same - this is not something to be had in mathematics. You can come up with approximations to what your are looking for, but they always involve a decision in terms of a cut-off (you ignore certain scales in a dynamical system, or impose limited memory on your stochastic process).
I asked myself "What are the main results of cybernetics / complex systems theory" and all I could think of was establishment of common language, especially subsuming Shannon Information, and notable examples in the wild having interesting non-trivial properties. I.e., the claims are mostly of the "There exists an X" rather than "For all X". To me, that's a sign that it's still in the bug-collecting phase.
Tell me I'm wrong? What are the major results in Cybernetics that have the form "For all X..."
Cybernetic ideas applied to automation underpin all of our technology (I'd argue that most of engineering systems theory, including controls, signal processing, and communications, has a cybernetic heritage).
Cybernetic ideas applied to social systems, on the other hand, haven't really gone anywhere.
This distinction appears to be an important lesson staring us in the face...
Very happy to see the direct segue from “cybernetics nut who no one listens to” to “Kevin Munger”, I feel seen.
But yes, as you know, I think what’s interesting about RCTs isn’t the causal inference, it’s the action, it’s changing the world and being changed by the world.
I love Sanjoy, but I also fondly remember him explaining to me how quantum measurement is merely a Kalman Filter running in imaginary time. My label applies.
A distribution of distributions is itself a distribution, but causality breaks that by positing some deterministic, non-cyclic process for choosing the distribution. In this lens, statistics is just the study of the local tangent space of process, and "the floor sort of falls out" once you introduce any curvature, and it drops out completely if there's non-differentiability.
As you say,
> Once you become obsessed with the problem of process, you either become a complex systems nut or a cybernetics nut, and no one listens to either.
I think the difficulty is that it's not enough to simply *posit* some deterministic process that we can study - that's easy nerd bait. For there to be real science at stake, there has to be some repeatable factorizability into studiable latent entities, which is the reason why distributions of distributions were interesting in the first place.
The red wine example is not a good one. A cross-sectional epidemiological study showed a correlation of red wine drinking and lower heart disease. Of course, the French drink a lot of red wine, but they also have a different diet and lifestyle. The hunt for the possible effect of red wine narrowed down to resveratrol. This then generated studies on teh effect og resveratrol on animals. It had minor effects. Most likely, the red wine example is a classic correlation but not causation. Resveratrol is no longer the "hot" compound to reduce heart disease. It is likely that the same effect is occurring with statins to reduce cholesterol. The epidemiological study showed a correlation of heart disease and early death with high cholesterol levels. Statins that block the enzyme in teh cholesterol synthesis pathway were seen as the "silver bullet" targeting cholesterol levels directly. (It also affects downstream pathways that utilize cholesterol, such as sex hormones.)
Bug Pharma persuaded doctors that all patients with "high cholesterol" should take statins. The blockbuster drug that was needed for a chronic condition made fortunes. But it now seems that cholesterol may not have been the cause of heart disease. Attempts have been made to talk about "good" HDL vs "bad" LDL cholesterol to keep the plates spinning on the money tree, but the long-term effects do not seem to be borne out, with meta studies showing only a small positive effect of statins. (And now we have a push to offer men testosterone to offset those lowered levels, which might be caused by statins blocking the cholesterol required for sex hormone production.)
This is why so much nutrition "science" advertised to the population is mostly incorrectly alighting on some variable that is of interest to the food industry, as it is full of cross-sectional epidemiological studies that are just correlations rather than the few longitudinal studies that are time-consuming in humans, and may not pan out.
Parabiosis in mice does seem to work, reversing aging in old mice receiving young mouse blood. This has led wealthy people to vampirishly pay youngsters for their blood to be transfused into their aging bodies in the hope of staying younger for longer. One hopes that the important factors in teh blood can be isolated so that this trade can end and all the population can eventually benefit. Long-term studies will need to be done to see if it truly works in humans.
We are an impatient society. Waiting for drug trials to be done over time for effect rather than using a proxy variable, is almost encouraging "snake oil" drug development for treatments that only make sense if the end result is achieved. Hope for the "fountain of youth" springs eternal.
Is there a difference between a complex systems nut and cybernetics nut? I personally identify as both. Process is truly fascinating
fair!
I'm curious to hear you thoughts on Hume's "An Enquiry Concerning Human Understanding" particularly section VII on "necessary connection". Hume ultimately relegates "causality" to a metaphysical concept and suggests all we can reason on is prediction. The field of "causal" inference, particularly in the social sciences where data is observational and collected from a complex system, seems to be very confused about what statistics can and cannot do. Terms like randomization, ignorability, and stable unit treatment values are invoked with almost liturgical regularity, as if their mere presence absolves one from deeper epistemological scrutiny. My field of economics often feels like a causality-themed masquerade ball. Conversations with applied microeconomists these days often resemble a Carrollian tea party: elaborate, incoherent, and indifferent to the actual question at hand. I'm not sure how to best dislodge some of the more ritualistic invocations of “identification” and provoke a more coherent discussion about what we’re actually doing when we claim to estimate causal effects. I'm inclined to offer Bruno de Finetti as the ultimate corrective, a reminder that coherence trumps counterfactual fantasies.
Lastly, I'm not sure how the average treatment effect became the center of focus in "causal" studies since we are ultimately interested in who (which subgroups) specifically might benefit from a treatment. Summary results on clinical trials like the "average treatment effect" may be non-representative of the treatment effect for a typical patient in the trial. Wouldn't a cluster based analysis on benefits and risk be more appropriate?
I like how we've spent 300 years trying to convince ourselves that Hume was wrong. We're still at it! :)
I 100% agree with you about observational causal inference. I have written about it before and considered writing another piece this morning, but got sidetracked in some questionable public health statistics and ran out of time. Perhaps on Monday!
Perhaps the worst thing about observational causal inference is the idea that you can reduce all causation to an RCT, be it real or imaginary. This gold standardization is the original sin that can't be corrected with robustness checks.
And yes, I also agree that the average treatment effect is indeed an odd metaphysical object. If we ignore process and instead hyperfocus on association, we convince ourselves that we need the law of large numbers to distinguish reality from hallucination. But not all that is learned from past experience is statistical. You don't need statistics to know that penicillin is effective...
"Modern statistics is ill-equipped to deal with process, but we don’t have a clear alternative class to offer."
Not only statistics, it is really modern mathematics that has formalized away any notion of process in the name of rigor.
Can you expand on what you mean?
There is certainly some math that claims to study process (stochastic processes, dynamical systems, etc.). But I also think these fields are only marginally helpful for describing complex feedback systems.
I think this is most easily illustrated with the function concept, which on the set theoretical definition is already essentially static. Sure, there are functions which depend on “time”, but in the end they too are just sets of tuples. Now in your “distribution shift” case what you want is a function whose extension changes in time, but the function should remain the same - this is not something to be had in mathematics. You can come up with approximations to what your are looking for, but they always involve a decision in terms of a cut-off (you ignore certain scales in a dynamical system, or impose limited memory on your stochastic process).
I asked myself "What are the main results of cybernetics / complex systems theory" and all I could think of was establishment of common language, especially subsuming Shannon Information, and notable examples in the wild having interesting non-trivial properties. I.e., the claims are mostly of the "There exists an X" rather than "For all X". To me, that's a sign that it's still in the bug-collecting phase.
Tell me I'm wrong? What are the major results in Cybernetics that have the form "For all X..."
Cybernetic ideas applied to automation underpin all of our technology (I'd argue that most of engineering systems theory, including controls, signal processing, and communications, has a cybernetic heritage).
Cybernetic ideas applied to social systems, on the other hand, haven't really gone anywhere.
This distinction appears to be an important lesson staring us in the face...
Very happy to see the direct segue from “cybernetics nut who no one listens to” to “Kevin Munger”, I feel seen.
But yes, as you know, I think what’s interesting about RCTs isn’t the causal inference, it’s the action, it’s changing the world and being changed by the world.
right, and in the context of action, the randomization takes on an entirely different meaning.
anyway, I raise my glass in solidarity with the process nuts no one listens to.
I, for one, first learned about Whitehead’s process philosophy from Sanjoy Mitter. Real control theorists know.
I love Sanjoy, but I also fondly remember him explaining to me how quantum measurement is merely a Kalman Filter running in imaginary time. My label applies.
He’s been on that since 1979, and also he’s not wrong. https://mitter.lids.mit.edu/publications/24_ontheanalogybetween.pdf
Yeah the knock on cybernetics/control theory nuts is never that they’re “wrong” haha
This!
Relatedly, the idea that quantum measurement is a form of Bayesian filtering has been thrown out there by Stratonovich as well.
are the lectures for this class available online? Thnx
A distribution of distributions is itself a distribution, but causality breaks that by positing some deterministic, non-cyclic process for choosing the distribution. In this lens, statistics is just the study of the local tangent space of process, and "the floor sort of falls out" once you introduce any curvature, and it drops out completely if there's non-differentiability.
As you say,
> Once you become obsessed with the problem of process, you either become a complex systems nut or a cybernetics nut, and no one listens to either.
I think the difficulty is that it's not enough to simply *posit* some deterministic process that we can study - that's easy nerd bait. For there to be real science at stake, there has to be some repeatable factorizability into studiable latent entities, which is the reason why distributions of distributions were interesting in the first place.
The red wine example is not a good one. A cross-sectional epidemiological study showed a correlation of red wine drinking and lower heart disease. Of course, the French drink a lot of red wine, but they also have a different diet and lifestyle. The hunt for the possible effect of red wine narrowed down to resveratrol. This then generated studies on teh effect og resveratrol on animals. It had minor effects. Most likely, the red wine example is a classic correlation but not causation. Resveratrol is no longer the "hot" compound to reduce heart disease. It is likely that the same effect is occurring with statins to reduce cholesterol. The epidemiological study showed a correlation of heart disease and early death with high cholesterol levels. Statins that block the enzyme in teh cholesterol synthesis pathway were seen as the "silver bullet" targeting cholesterol levels directly. (It also affects downstream pathways that utilize cholesterol, such as sex hormones.)
Bug Pharma persuaded doctors that all patients with "high cholesterol" should take statins. The blockbuster drug that was needed for a chronic condition made fortunes. But it now seems that cholesterol may not have been the cause of heart disease. Attempts have been made to talk about "good" HDL vs "bad" LDL cholesterol to keep the plates spinning on the money tree, but the long-term effects do not seem to be borne out, with meta studies showing only a small positive effect of statins. (And now we have a push to offer men testosterone to offset those lowered levels, which might be caused by statins blocking the cholesterol required for sex hormone production.)
This is why so much nutrition "science" advertised to the population is mostly incorrectly alighting on some variable that is of interest to the food industry, as it is full of cross-sectional epidemiological studies that are just correlations rather than the few longitudinal studies that are time-consuming in humans, and may not pan out.
Parabiosis in mice does seem to work, reversing aging in old mice receiving young mouse blood. This has led wealthy people to vampirishly pay youngsters for their blood to be transfused into their aging bodies in the hope of staying younger for longer. One hopes that the important factors in teh blood can be isolated so that this trade can end and all the population can eventually benefit. Long-term studies will need to be done to see if it truly works in humans.
We are an impatient society. Waiting for drug trials to be done over time for effect rather than using a proxy variable, is almost encouraging "snake oil" drug development for treatments that only make sense if the end result is achieved. Hope for the "fountain of youth" springs eternal.