>Meehl makes it very clear that he is only speaking about observational studies, not randomized experiments.
But Meehl does extend his view to some randomized experiments in the 1990 paper: he says that any experiment where the target effect hinges on an interaction between the (randomly assigned) treatment and some other covariate that wasn't randomly assigned is essentially observational and the same critiques therefore apply.
Yes, agree! In the lecture, he multiply emphasizes that he only is talking about observational studies, but it seems like he's got some more nuance in this paper. He mentions in the lecture that his colleague David Lykken is convinced that the obfuscators more generally apply to experiments, so maybe Lykken nudged Meehl a bit more down this path.
Since I am involved in research that is much more qualitative, I often read both observational and controlled studies. I agree that observational studies are highly highly flawed, but I also think it receives unwarranted hate way too often (while controlled studies receive too much love by comparison). Here’s my argument to hopefully give some love for observational studies. In a socio-political context where there are many actors in different situations, they are put in an environment with artifacts and objects that highly influence their behavior. While all the actors may share similar goals for this study (as economists may do to simplify their analysis), the means in which they get to that goal varies greatly and is highly context dependent. Often times, the only way to uncover issues and problems in certain social/political/etc.. context is to do an observational study. Actors with different backgrounds and contexts are placed in an environment with similar artifacts. What do they have to do to get to a goal? How do they do it? And why do they do it the way they do it? The only way to do this is to observe these actors. Then if you can, you must talk to them. This is, in my opinion, how we identify problems. Now, say you found the problem, and you want to modify the environment to help address some of the problems. You hypothesize that by making this intervention, you may be able to see some changes. Again, in a socio-political world, a controlled study is just not possible. So you do this intervention, and you observe the changes again. You repeat the observational study again and perhaps you found some issues with this intervention. Then you rinse and repeat.
>Meehl makes it very clear that he is only speaking about observational studies, not randomized experiments.
But Meehl does extend his view to some randomized experiments in the 1990 paper: he says that any experiment where the target effect hinges on an interaction between the (randomly assigned) treatment and some other covariate that wasn't randomly assigned is essentially observational and the same critiques therefore apply.
Yes, agree! In the lecture, he multiply emphasizes that he only is talking about observational studies, but it seems like he's got some more nuance in this paper. He mentions in the lecture that his colleague David Lykken is convinced that the obfuscators more generally apply to experiments, so maybe Lykken nudged Meehl a bit more down this path.
Since I am involved in research that is much more qualitative, I often read both observational and controlled studies. I agree that observational studies are highly highly flawed, but I also think it receives unwarranted hate way too often (while controlled studies receive too much love by comparison). Here’s my argument to hopefully give some love for observational studies. In a socio-political context where there are many actors in different situations, they are put in an environment with artifacts and objects that highly influence their behavior. While all the actors may share similar goals for this study (as economists may do to simplify their analysis), the means in which they get to that goal varies greatly and is highly context dependent. Often times, the only way to uncover issues and problems in certain social/political/etc.. context is to do an observational study. Actors with different backgrounds and contexts are placed in an environment with similar artifacts. What do they have to do to get to a goal? How do they do it? And why do they do it the way they do it? The only way to do this is to observe these actors. Then if you can, you must talk to them. This is, in my opinion, how we identify problems. Now, say you found the problem, and you want to modify the environment to help address some of the problems. You hypothesize that by making this intervention, you may be able to see some changes. Again, in a socio-political world, a controlled study is just not possible. So you do this intervention, and you observe the changes again. You repeat the observational study again and perhaps you found some issues with this intervention. Then you rinse and repeat.