yikes. I just re-read that comment and it is incomprehensible. Here is a conversation that may help.
Medical commission of Country X: There are too many C-sections in this hospital. We are going to recommend that doctors decrease C-sections.
Doctor in Hospital: We are more than happy to decrease C-sections. From now on, every time a patient comes in, we will call you and you should tell us whether to do a C-section.
Medical Commission: But that is not my job, that is yours.
Doctor: Yup. That is what we are doing. We keep telling you that our patients are different which is why we have so many C-sections.
(That's a real conversation, by the way)
I think this leads to a situation that cannot be resolved through argumentation. But,
Medical Commission: We are going to reduce how much you guys are paid for C-sections.
Researcher some months later shows that C-sections went down without any harm to mother or infants. Policy is kept in place.
It seems to me that shifting things back to a population-based policy is one way to get around the tricky issue of the reference class.
Am I completely off track on this? Apologies if unrelated to your course--I am intrigued by the connection to Meehl!
I have a full paper on this that I am trying my damnedest to finish this week. I hope it answers your questions.
But a short answer for now: If we insist on evaluating the outcomes of individuals on average, then the actuarial policies will always be superior. But why must we insist on such statistical metrics? That's what the doctors are (implicitly) pushing back against.
Yes! Exactly! I was thinking of this as leading to a `non-argumentable' situation. And that is what always confused me about Meehl's argument. He starts with actuarial versus clinical decisions, argues that the reference class is a key concept, but then moves back to evaluating the relative effectiveness of both on metrics based on averages. But the whole point of the reference class is that the average does not matter, and, as you say, if the epistemic authority is going to load on the average, the discussion is over, since the actuarial policy is by definition the best way forward. In the broken leg discussions, Meehl is forced to concede that the broken leg raises problems for actuarial policies, but then goes on the argue that the broken leg is just extremely rare. I wonder if that is true--or was ever true. Really look forward to reading the paper!
Yes, I am a huge fan of Meehl, but his obsession with quantification was his Achilles heel. The weakest points in all of his arguments are always when he insists everything is quantifiable.
What made you decide to remove the problem sets? Yes, the problem sets were hugely painful and quite hard when I did them, but they were a rewarding experience when you got it right. I can’t say I can do the questions again though since I’m majorly out of practice now. But I walked out of the class like yea I can actually do these problems given I have a textbook and a couple of hours on my hand.
I really wish I could take this course. One question: I am intrigued by your statement on how a sample-based view resolves the issues raised by Meehl (I have read your previous posts on this). I work in healthcare, and the usual response I receive from doctors on why they do not follow evidence-based guidelines is that their patients are `different.' I have always thought of this problem of the broken leg in Meehl as a question of the reference class, and have gradually arrived at the conclusion that the response `my patient is different' leads to the dead-end of a non-argumentable situation. In that, one could (not sure yet, how) formally show that the propositions `following the guidelines is good' and `I don't because my patients are different' cannot be resolved through argumentation. Consequently, to recover the value of a prediction metric, you are forced to move your recommendation back to population-based policy, instead of individual-level predictions, which is what Meehl was worried about?
yikes. I just re-read that comment and it is incomprehensible. Here is a conversation that may help.
Medical commission of Country X: There are too many C-sections in this hospital. We are going to recommend that doctors decrease C-sections.
Doctor in Hospital: We are more than happy to decrease C-sections. From now on, every time a patient comes in, we will call you and you should tell us whether to do a C-section.
Medical Commission: But that is not my job, that is yours.
Doctor: Yup. That is what we are doing. We keep telling you that our patients are different which is why we have so many C-sections.
(That's a real conversation, by the way)
I think this leads to a situation that cannot be resolved through argumentation. But,
Medical Commission: We are going to reduce how much you guys are paid for C-sections.
Researcher some months later shows that C-sections went down without any harm to mother or infants. Policy is kept in place.
It seems to me that shifting things back to a population-based policy is one way to get around the tricky issue of the reference class.
Am I completely off track on this? Apologies if unrelated to your course--I am intrigued by the connection to Meehl!
I have a full paper on this that I am trying my damnedest to finish this week. I hope it answers your questions.
But a short answer for now: If we insist on evaluating the outcomes of individuals on average, then the actuarial policies will always be superior. But why must we insist on such statistical metrics? That's what the doctors are (implicitly) pushing back against.
Yes! Exactly! I was thinking of this as leading to a `non-argumentable' situation. And that is what always confused me about Meehl's argument. He starts with actuarial versus clinical decisions, argues that the reference class is a key concept, but then moves back to evaluating the relative effectiveness of both on metrics based on averages. But the whole point of the reference class is that the average does not matter, and, as you say, if the epistemic authority is going to load on the average, the discussion is over, since the actuarial policy is by definition the best way forward. In the broken leg discussions, Meehl is forced to concede that the broken leg raises problems for actuarial policies, but then goes on the argue that the broken leg is just extremely rare. I wonder if that is true--or was ever true. Really look forward to reading the paper!
Yes, I am a huge fan of Meehl, but his obsession with quantification was his Achilles heel. The weakest points in all of his arguments are always when he insists everything is quantifiable.
So psyched. Feeling lucky to learn from you through the internet.
What made you decide to remove the problem sets? Yes, the problem sets were hugely painful and quite hard when I did them, but they were a rewarding experience when you got it right. I can’t say I can do the questions again though since I’m majorly out of practice now. But I walked out of the class like yea I can actually do these problems given I have a textbook and a couple of hours on my hand.
Oh, do not worry, my friend, these project questions are *in addition* to the p-sets, not a substitution.
:: lawsuits from statistics department on the way
What is a sample-based actuarial view? Could you post some links?
Patience, my friend. I'm going to post as much course content as I can here on the blog as I go.
Ok, thanks.
I really wish I could take this course. One question: I am intrigued by your statement on how a sample-based view resolves the issues raised by Meehl (I have read your previous posts on this). I work in healthcare, and the usual response I receive from doctors on why they do not follow evidence-based guidelines is that their patients are `different.' I have always thought of this problem of the broken leg in Meehl as a question of the reference class, and have gradually arrived at the conclusion that the response `my patient is different' leads to the dead-end of a non-argumentable situation. In that, one could (not sure yet, how) formally show that the propositions `following the guidelines is good' and `I don't because my patients are different' cannot be resolved through argumentation. Consequently, to recover the value of a prediction metric, you are forced to move your recommendation back to population-based policy, instead of individual-level predictions, which is what Meehl was worried about?