In practice, the anticipated treatment effect of a trial isn’t (shouldn’t be) entirely a guess. In trials of medical interventions, for example, researchers may be guided by the idea of a clinically significant effect size. This is based on existing understanding of the condition being treated and is the difference you would want to bring about for there to be a meaningful change in the condition’s effects or prognosis, such as improvement of x points on the main outcome measure. Still an elaborate theater, but with slightly less guesswork than you implied.
Another question: I don't understand the dimensions in the final formula for K. K and N are dimensionless, but ATE isn't? Something must be assumed about how we measure treatment effect. Is it binary?
Yeah, it's annoying. Any time you see "ATE" it's really "The ATE divided by the possible dynamic range of the ATE." That second quantity is dimensionless. But by assumption, the dynamic range of the ATE is 1.
I don't have a good explanation for the units of log(N), however.
What do we do if N = infinity? I'm thinking of a case where you can continue to generate new "test subjects" without any restriction. E.g you continue to run an experiment "as long as it takes" and (for simplicity) assume each day's results are independent from the past and future days.
This is a job for Rev Bayes! Given a prior distribution on the benefit of the treatment, Blackwell's theorem gives you the value of info, and you can use that to determine the marginal value of an additional data point. In the sequential case, you update as you go.
In practice, the anticipated treatment effect of a trial isn’t (shouldn’t be) entirely a guess. In trials of medical interventions, for example, researchers may be guided by the idea of a clinically significant effect size. This is based on existing understanding of the condition being treated and is the difference you would want to bring about for there to be a meaningful change in the condition’s effects or prognosis, such as improvement of x points on the main outcome measure. Still an elaborate theater, but with slightly less guesswork than you implied.
Another question: I don't understand the dimensions in the final formula for K. K and N are dimensionless, but ATE isn't? Something must be assumed about how we measure treatment effect. Is it binary?
Yeah, it's annoying. Any time you see "ATE" it's really "The ATE divided by the possible dynamic range of the ATE." That second quantity is dimensionless. But by assumption, the dynamic range of the ATE is 1.
I don't have a good explanation for the units of log(N), however.
Thanks. Another follow-up question :)
What do we do if N = infinity? I'm thinking of a case where you can continue to generate new "test subjects" without any restriction. E.g you continue to run an experiment "as long as it takes" and (for simplicity) assume each day's results are independent from the past and future days.
Where does K come from in the formula for G? It doesn't appear in the formulas for E, E_c or ATE.
K only appears when you take an expected value with respect to the sampling of the experimental group.
I'm writing this whole argument up formally and will post it very soon here: https://www.argmin.net/p/patterns-predictions-and-actions-585
Stay tuned.
Thanks! I have your book with the same title, and looked for that argument in it, but couldn't find it :)
This is a job for Rev Bayes! Given a prior distribution on the benefit of the treatment, Blackwell's theorem gives you the value of info, and you can use that to determine the marginal value of an additional data point. In the sequential case, you update as you go.