I think Gelman has been very important in social science. He can be tactless, he can be Difficult, I think. But he's been vocal for years, criticizing bad science, criticizing noisy estimates with large standard errors and little theoretical reason to believe them. I think that has been helpful to social science.
But Gelman also does lots of work with observational data. He'll model, for example, vote as a function of state, income, education, gender etc. using the kind of logistic regression model you do not like. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9203002/
So that's something I was wondering: how do we think of this kind of work from the perspective you are developing in this blog?
"On one hand, I do not see how one can get scientifically strong causal inferences from observational data alone without strong theory; it seems to me a hopeless task to throw a data matrix at a computer program and hope to learn about causal structure (more on this below). On the other hand, I recognize that much of our everyday causal intuition, while not having the full quality of scientific reasoning, is still useful, and it seems a bit of a gap to simply use the label “descriptive” for all inference that is not supported by experiment or very strong theory. [...] For example, my own work demonstrates that income, religion, and religious attendance predict voter choice in different ways in different parts of the country; my colleagues and I have also found regional variation in attitudes on economic and social issues (Gelman et al. 2009). But I don’t know exactly what would be learned by throwing all these variables into a multivariate model."
I think it's a reasonable and measured perspective. But I think it opens the door to saying something like "religion causes vote choice”. Or maybe not “cause” but “explain” or “predict”? But it sounds all very metaphysical, but it’s still something Gelman finds worth doing!
Excellent questions. I'm going to go in a very different direction next week. But at some point I want to unpack what we learn from these demographic studies. I can never tell how to be sure we don't just see what we want to see in the patterns. Anyway, too long to get into in a comment, but I will think about how to best engage with your points.
I think one historically common approach to increase power, and mitigate corruption, and lies is with meta-analysis of several RCTs measuring a presumed similar effect.
I think Sander Greenland (emeritus UCLA) has spent many years thinking of the use and abuse of RCTs etc. Not sure you've read his stuff so far.
Maybe meta-analysis can work in theory but this on why it often doesn't work in practice is interesting. But maybe you've seen it it was circulated a lot on Twitter.
requesting door 4 !
I'm also in favour of 4.
I think Gelman has been very important in social science. He can be tactless, he can be Difficult, I think. But he's been vocal for years, criticizing bad science, criticizing noisy estimates with large standard errors and little theoretical reason to believe them. I think that has been helpful to social science.
But Gelman also does lots of work with observational data. He'll model, for example, vote as a function of state, income, education, gender etc. using the kind of logistic regression model you do not like. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9203002/
So that's something I was wondering: how do we think of this kind of work from the perspective you are developing in this blog?
From his essay on Causality and Statistical Learning Essay (https://www.journals.uchicago.edu/doi/epdf/10.1086/662659):
"On one hand, I do not see how one can get scientifically strong causal inferences from observational data alone without strong theory; it seems to me a hopeless task to throw a data matrix at a computer program and hope to learn about causal structure (more on this below). On the other hand, I recognize that much of our everyday causal intuition, while not having the full quality of scientific reasoning, is still useful, and it seems a bit of a gap to simply use the label “descriptive” for all inference that is not supported by experiment or very strong theory. [...] For example, my own work demonstrates that income, religion, and religious attendance predict voter choice in different ways in different parts of the country; my colleagues and I have also found regional variation in attitudes on economic and social issues (Gelman et al. 2009). But I don’t know exactly what would be learned by throwing all these variables into a multivariate model."
I think it's a reasonable and measured perspective. But I think it opens the door to saying something like "religion causes vote choice”. Or maybe not “cause” but “explain” or “predict”? But it sounds all very metaphysical, but it’s still something Gelman finds worth doing!
Excellent questions. I'm going to go in a very different direction next week. But at some point I want to unpack what we learn from these demographic studies. I can never tell how to be sure we don't just see what we want to see in the patterns. Anyway, too long to get into in a comment, but I will think about how to best engage with your points.
I think one historically common approach to increase power, and mitigate corruption, and lies is with meta-analysis of several RCTs measuring a presumed similar effect.
I think Sander Greenland (emeritus UCLA) has spent many years thinking of the use and abuse of RCTs etc. Not sure you've read his stuff so far.
The problem with meta-analysis is that it's impossible to avoid all of the pitfalls of observational data analysis. They are very tricky to do well.
Maybe meta-analysis can work in theory but this on why it often doesn't work in practice is interesting. But maybe you've seen it it was circulated a lot on Twitter.
https://datacolada.org/104
The fourth answer please!
#4
Let’s hear it for the fourth way!