One idea for “what else can we do?” is here: https://doi.org/10.1016/j.tics.2022.05.008 In cognitive science we can treat each participant as a replication of an N=1 study, then formally quantify how this generalises to the population by estimating the population-level experimental replication probability (the prevalence): https://elifesciences.org/articles/62461 I think this approach could also be useful in other areas.
I don't have a very sophisticated understanding of statistics, so please correct me if I'm wrong, but I think your core point is that large data might be useful for finding small effect sizes, but we've unnecessarily lost focus on small studies with large effect sizes. Is that right? But don't small studies often show small effect sizes? Isn't the deeper issue that we've picked all the low hanging fruit in science?
Large effect sizes have always been rare, and it's not evident that having more scientists or studies is related to the number of breakthroughs that occur. Science is a weird and complicated dance!
My issue is in our push to read tea leaves from statistics, we don't stop and think about what these statistics mean (and often they mean nothing at all). The larger the data set, the less an scientist can understand what happened to cause any particular data point. This leads to inferences that need to be questioned more thoroughly.
I like your premise questioning and I think statistics and science need more of it. It's just so hard to do any sort of premise questioning with anyone, especially people with entrenched jobs/methodologies/etc., so I think there's an interesting meta question here of how to go about premise questioning. I'm a new subscriber coming through the recent controversy with Gelman et al. and while I think there are some interesting details in that mini-debate, your higher level, premise questioning points seemed to go past them.
I fight many quixotic battles, but I know that I can only do so much. That's fine! I've always enjoyed rustling academic feathers, and my collaborators and I have often made some decently useful contributions in the process.
The most important part is that I enjoy the process. Yes, I get some folks grumpy with me on the internet. But I learn a lot even when people get mad at me.
Thanks again for commenting, and please do set me straight when I spout BS.
You'll enjoy this pithy little chestnut from Charles Geyer:
"The story about n going to infinity is even less plausible in spatial statistics and statistical genetics where every component of the data may be correlated with every other component. Suppose we have data on school districts of Minnesota. How does Minnesota go to infinity? By invasion of surrounding states and provinces of Canada, not to mention Lake Superior, and eventually by rocket ships to outer space? How silly does the n goes to infinity story have to be before it provokes laughter instead of reverence?"
A lot of good quips in there for sure. But my beef is more with why we think these probability distributions (or likelihoods) are worth estimating in the first place. That jump from "here's a sequence of things" to "this sequence was exchangeable samples from distribution p" is just as unjustified as the thought experiments sending n to infinity, no?
recent subscriber but i like the kick you’re on here. The concepts in this post remind me of the database torture / observational study mayhem during the covid-publishing era.
I’m not sure a return to small data is possible or necessary, but if it is, how do you imagine it happening? It’s much easier to curate large observational datasets than ones where you must design a study and execute it. what incentives could exist which would drive the latter even though it is much more difficult?
I always wonder: if everyone knows that observational studies can say whatever you want them to say, then why to we all accept them as scientific work in the first place? Science is full of so many funny bad practices that everyone knows are wrong. For now, I'll just keep pointing them out here...
But also, in the case of medical evidence, case studies and small n trials still do exist and still play an important role. Discussions of cases in grand rounds are still a real thing. I think it's easy to see the more popularized data mining studies, but, for example the NEJM still publishes case studies in every issue. Here's a case study from the latest issue:
One idea for “what else can we do?” is here: https://doi.org/10.1016/j.tics.2022.05.008 In cognitive science we can treat each participant as a replication of an N=1 study, then formally quantify how this generalises to the population by estimating the population-level experimental replication probability (the prevalence): https://elifesciences.org/articles/62461 I think this approach could also be useful in other areas.
Looking forward to reading these! I'm a huge fan of N-of-1 studies.
I don't have a very sophisticated understanding of statistics, so please correct me if I'm wrong, but I think your core point is that large data might be useful for finding small effect sizes, but we've unnecessarily lost focus on small studies with large effect sizes. Is that right? But don't small studies often show small effect sizes? Isn't the deeper issue that we've picked all the low hanging fruit in science?
Large effect sizes have always been rare, and it's not evident that having more scientists or studies is related to the number of breakthroughs that occur. Science is a weird and complicated dance!
My issue is in our push to read tea leaves from statistics, we don't stop and think about what these statistics mean (and often they mean nothing at all). The larger the data set, the less an scientist can understand what happened to cause any particular data point. This leads to inferences that need to be questioned more thoroughly.
I like your premise questioning and I think statistics and science need more of it. It's just so hard to do any sort of premise questioning with anyone, especially people with entrenched jobs/methodologies/etc., so I think there's an interesting meta question here of how to go about premise questioning. I'm a new subscriber coming through the recent controversy with Gelman et al. and while I think there are some interesting details in that mini-debate, your higher level, premise questioning points seemed to go past them.
First, thanks for subscribing and thanks for the thoughtful comments. I appreciate them!
Second, I have written a bit about why critiques fail to get capture (for example: https://www.argmin.net/p/difficult-statistics). Herd trends are indeed powerful.
I fight many quixotic battles, but I know that I can only do so much. That's fine! I've always enjoyed rustling academic feathers, and my collaborators and I have often made some decently useful contributions in the process.
The most important part is that I enjoy the process. Yes, I get some folks grumpy with me on the internet. But I learn a lot even when people get mad at me.
Thanks again for commenting, and please do set me straight when I spout BS.
You'll enjoy this pithy little chestnut from Charles Geyer:
"The story about n going to infinity is even less plausible in spatial statistics and statistical genetics where every component of the data may be correlated with every other component. Suppose we have data on school districts of Minnesota. How does Minnesota go to infinity? By invasion of surrounding states and provinces of Canada, not to mention Lake Superior, and eventually by rocket ships to outer space? How silly does the n goes to infinity story have to be before it provokes laughter instead of reverence?"
Read the whole thing, it's worth it: https://www.stat.umn.edu/geyer/lecam/simple.pdf
A lot of good quips in there for sure. But my beef is more with why we think these probability distributions (or likelihoods) are worth estimating in the first place. That jump from "here's a sequence of things" to "this sequence was exchangeable samples from distribution p" is just as unjustified as the thought experiments sending n to infinity, no?
recent subscriber but i like the kick you’re on here. The concepts in this post remind me of the database torture / observational study mayhem during the covid-publishing era.
I’m not sure a return to small data is possible or necessary, but if it is, how do you imagine it happening? It’s much easier to curate large observational datasets than ones where you must design a study and execute it. what incentives could exist which would drive the latter even though it is much more difficult?
I always wonder: if everyone knows that observational studies can say whatever you want them to say, then why to we all accept them as scientific work in the first place? Science is full of so many funny bad practices that everyone knows are wrong. For now, I'll just keep pointing them out here...
But also, in the case of medical evidence, case studies and small n trials still do exist and still play an important role. Discussions of cases in grand rounds are still a real thing. I think it's easy to see the more popularized data mining studies, but, for example the NEJM still publishes case studies in every issue. Here's a case study from the latest issue:
https://www.nejm.org/doi/full/10.1056/NEJMcps2307935