That “probabilities are different if you know the future” line cracked me up. I’m a Steelers fan, so I’ve watched plenty of Bengals games hoping they’d lose, and I’m convinced they have some sort of constant lately (except against us, apparently). Next Gen Stats should just bake in a Bengals coefficient of minus 10% win probability on arrival.
Wouldn't this be an excellent case for confidence intervals? You have written a lot about how they don't really have any use case, but the probabilities reported seem to be overly precise for the amount of data available. Sure the point estimates are 1% point apart, but with so few actual occurrences we are talking confidence intervals from [0.02, 0.22] or something to that effect. Plenty wide to be able to discount any minor differences in outcome probabilities between the two strategies.
edit: I understand this is besides the point of the limited applicability of analytics to non-stationary, reactive/feedback, <other jargon>, real world systems that you are going on about which I agree with. This is more of a "ok we'll play your analytics game" and show there is still nothing to stand on.
I hear you, but I'm not even sure if confidence intervals make sense here. The win probabilities are based on rather complex models, where I don't know how you'd do proper error analysis.
Interestingly, the win probabilities at the beginning of the game are based on the assumption that the game's outcome is normally distributed around the point spread, with a standard deviation of two touchdowns... It's all downhill from there.
That “probabilities are different if you know the future” line cracked me up. I’m a Steelers fan, so I’ve watched plenty of Bengals games hoping they’d lose, and I’m convinced they have some sort of constant lately (except against us, apparently). Next Gen Stats should just bake in a Bengals coefficient of minus 10% win probability on arrival.
Wouldn't this be an excellent case for confidence intervals? You have written a lot about how they don't really have any use case, but the probabilities reported seem to be overly precise for the amount of data available. Sure the point estimates are 1% point apart, but with so few actual occurrences we are talking confidence intervals from [0.02, 0.22] or something to that effect. Plenty wide to be able to discount any minor differences in outcome probabilities between the two strategies.
edit: I understand this is besides the point of the limited applicability of analytics to non-stationary, reactive/feedback, <other jargon>, real world systems that you are going on about which I agree with. This is more of a "ok we'll play your analytics game" and show there is still nothing to stand on.
I hear you, but I'm not even sure if confidence intervals make sense here. The win probabilities are based on rather complex models, where I don't know how you'd do proper error analysis.
https://www.argmin.net/p/what-is-the-chance-of-a-beast-quake
Interestingly, the win probabilities at the beginning of the game are based on the assumption that the game's outcome is normally distributed around the point spread, with a standard deviation of two touchdowns... It's all downhill from there.