Love this series. A medium amount of statistics is a dangerous thing. I used to believe this football analytics stuff but I'm with you --- there's just not enough isolation and repeatability to believe any of this.
Thanks! I had to cut it because of flow, but I think a reasonable defense of analytics is to suggest some aggressive moves are not crazy. If you can show that, under reasonable modeling, the go-for-2-down-8 move has a comparable probability of success to kicking the extra point, that's helpful as it adds extra possibilities to the strategy list.
Sure, but to show it's "comparable", you should only need a small amount of statistics, not a medium amount, right? It's like computational biology + ML results in the late 1990's / early 2000's (and probably today, who knows) --- if the effect is big enough that it's obvious when you plot it, you don't need the fancy statistics, and if it's not, the fancy statistics aren't going to stop you from hurting yourself.
In some sense, the fact that a winning pro coach would consider the move is already the best evidence it's not crazy? Not irrefutable --- sometimes winning pro coaches make obvious clock errors, and I'm sure if you plotted percentages you could make charts showing obvious clock errors are bad, but usually the next day the coach says "that was a dumb clock error..."
Right. I do think that the game has changed: more people are trying moves that would have been considered risky twenty years ago, and teams definitely have internal analytics departments that crunch numbers. How causally linked those two are is not clear.
I guess the point I'm trying to make is that (with moderate confidence), I don't think you can expect nearly as much from football analytics as you can from baseball or basketball --- the situations seem much more causally complex and the number of repeats lower. Compared to baseball or basketball, football feels more like social science (if you can't see it on the graph, you're probably fooling yourself) and less like hedge funds (you can win by accumulating a bunch of barely detectable advantages). What do you think?
Love this series. A medium amount of statistics is a dangerous thing. I used to believe this football analytics stuff but I'm with you --- there's just not enough isolation and repeatability to believe any of this.
Thanks! I had to cut it because of flow, but I think a reasonable defense of analytics is to suggest some aggressive moves are not crazy. If you can show that, under reasonable modeling, the go-for-2-down-8 move has a comparable probability of success to kicking the extra point, that's helpful as it adds extra possibilities to the strategy list.
Sure, but to show it's "comparable", you should only need a small amount of statistics, not a medium amount, right? It's like computational biology + ML results in the late 1990's / early 2000's (and probably today, who knows) --- if the effect is big enough that it's obvious when you plot it, you don't need the fancy statistics, and if it's not, the fancy statistics aren't going to stop you from hurting yourself.
In some sense, the fact that a winning pro coach would consider the move is already the best evidence it's not crazy? Not irrefutable --- sometimes winning pro coaches make obvious clock errors, and I'm sure if you plotted percentages you could make charts showing obvious clock errors are bad, but usually the next day the coach says "that was a dumb clock error..."
Right. I do think that the game has changed: more people are trying moves that would have been considered risky twenty years ago, and teams definitely have internal analytics departments that crunch numbers. How causally linked those two are is not clear.
I guess the point I'm trying to make is that (with moderate confidence), I don't think you can expect nearly as much from football analytics as you can from baseball or basketball --- the situations seem much more causally complex and the number of repeats lower. Compared to baseball or basketball, football feels more like social science (if you can't see it on the graph, you're probably fooling yourself) and less like hedge funds (you can win by accumulating a bunch of barely detectable advantages). What do you think?
yes, I totally agree with that.