So you're definitely agreeing with Watt that these stats are meaningless? I have to say ... I don't know!
I'd love to get your thoughts on a couple specific examples.
Imagine a defensive player P who is extremely fast and strong, and gets a lot of sacks. The offense adjusts by consistently double-teaming P. As a result, P gets very few sacks. It seems likely to me that PFF can and does observe this, and they ought to grade P highly because of it. It might not show up in P's team winning more, because maybe the rest of P's team isn't very good, or maybe the number of wins is just too noisy (it's important to note that football has an order of magnitude fewer games per season than baseball or baskbetball), but there is some real sense in which P is doing a better job than a different defensive player P' who's only getting single-teamed but getting the same number of stats.
I'm also interested in your thoughts on the "interceptable passes" stat. This passes my smell test: NFL receivers are good but not arbitrarily good, and how often they catch a pass "to them" sure looks like it's related to how accurate or tight the pass is. It seems quite plausible to me that someone who watches a lot of football and says "That pass wasn't intercepted because it was nowhere near the defensive player, but *that* pass wasn't intercepted only because the offense got lucky " is saying something that isn't total nonsense. (I guess they're saying "Basically no passes that look like the first one get intercepted, but a good fraction of passes that look like the second get intercepted.") Combine this with, again, the relatively small numbers involved in NFL football, which imply that even if we were observing some perfect random process we'd expect a lot of variance around infinite populatin average over the course of a season, and I can easily imagine this is doing something that points in the direction of meaningful.
Being a loyal Badger fan and also an admirer of his game and analysis, I do side with JJ.
There's three things here: First, the pash rush/interception stats are the easiest stats to understand and they have questionable utility. Once you get into the weird ways they grade offensive linemen, it does seem like they're just making stuff up.
But second, JJ argues that these statistics are relying on all sorts of counterfactuals that the PFF people not only can't know, but also have no insight into because
And finally, JJ also points out that PFF needs to be controversial as otherwise no one will listen to them. They are necessarily cooking their ratings to spice up engagement.
I get your points about how you can argue that these different stats maybe are capturing something, but in general, statistics flattens experience into some clean numbers. And it's just funny that in football where you have such small sample sizes, stating those clean numbers without error bars is inevitably just infotainment, no?
Ive watched lots of Texans game (big houston sports fan here), and I can only imagine CJ stroud has a low PFF score is because he throws really risky balls that somehow always end up landing perfectly in the WR hands. It’s what makes him so special because he makes low-medium risk high reward plays that 99% of QBs would be high risk high reward plays. Some are crazy behind the shoulder plays that are like inches away from the CBs or Safetys hand, but no cigar. And he does this on a really consistent basis. This is where I suppose the eye test beats out statistics and statistics just cant capture the nuances of how good CJ Stroud is.
CJ Stroud is such a great example because no one predicted he would be as good as he is, especially with the rest of that roster. And yet, analytics...
CJ Stroud is a good example of how analytics probably cannot capture how good he is because he makes very ballsy throws that analytics would frown upon. But he passes all the eyes test because his throws are pretty incredible and impossible to make for most QBs. I do think there is merit in analytics though, and I would be pretty upset if my Houston teams (in NBA/NFL/MLB) disregarded analytics as it would probably make them a worse team. In this regard, given that sports is a game of margins and most teams do not have the massive allowance and allure that teams in LA/NYC has (see MLB), analytics has helped smaller market teams get an edge over huge market teams with their multitude of superstars. Ultimately though, in sports, nothing beats natural talent and good old-fashioned proper coaching, practice, and execution (which will always beat analytics on its own).
The term "analytics" is so ill defined that I don't know what to think. Here's a dumb example: one of Bill Belichick's famous moves in New England was switching to a 3-4 defense because he thought it would be easier to manage under the salary cap. This clearly required analysis and cost-benefit projections. Is that analytics?
Yes I would consider that analytics. I think every good coach regardless of the time period did analytics to make decisions on what plays to run against certain teams. Now you just have bigger technology to help you facilitate that faster than before, and you would be a fool not to take advantage of that as a sport team. Things like machine learning / data science is just one of many analytics a team should be operating under. But having that doesn’t mean coaches should (and good coaches generally don’t) disregard old school ways of doing “analytics” like reviewing video tapes of their opposing team to prepare plays and formations to maximize their own team’s chance of winning. In the world of experimental design, we would call this a good old fashioned observational study of a participant’s reaction against certain treatments (i.e. I noticed a trend that opposing team does poorly against this formation and play so we should practice this play and execute it against them to give ourselves more points and increase our chances of beating them.)
"It’s a beautiful example of how statistics, even if they are meaningless, can have tangible negative consequences." Maybe I'm dense, but I don't see any examples of tangible negative consequences given. What were awards and contracts based beforehand? Is the suggestion that we should stop collecting stats, or simply interpret them better?
I agree that it's especially hard to interpret stats in an adversarial setting, but I don't have a better idea for running things in general than "collect as much data as you can and carefully apply all your knowledge when interpreting them.".
If the stats are made up, but a major network that carries major games starts arguing they are important, that's letting weird media pressure make statistics meaningful for the sake of infotainment. Why is that good?
I didn't say any of this is good. I asked for examples of tangible harm, which you claimed exist in your original post.
I agree it's reasonable to suspect that people are reading too much into stats that seem hard to interpret correctly, but again, I'm also asking: what's your proposed alternative?
So you're definitely agreeing with Watt that these stats are meaningless? I have to say ... I don't know!
I'd love to get your thoughts on a couple specific examples.
Imagine a defensive player P who is extremely fast and strong, and gets a lot of sacks. The offense adjusts by consistently double-teaming P. As a result, P gets very few sacks. It seems likely to me that PFF can and does observe this, and they ought to grade P highly because of it. It might not show up in P's team winning more, because maybe the rest of P's team isn't very good, or maybe the number of wins is just too noisy (it's important to note that football has an order of magnitude fewer games per season than baseball or baskbetball), but there is some real sense in which P is doing a better job than a different defensive player P' who's only getting single-teamed but getting the same number of stats.
I'm also interested in your thoughts on the "interceptable passes" stat. This passes my smell test: NFL receivers are good but not arbitrarily good, and how often they catch a pass "to them" sure looks like it's related to how accurate or tight the pass is. It seems quite plausible to me that someone who watches a lot of football and says "That pass wasn't intercepted because it was nowhere near the defensive player, but *that* pass wasn't intercepted only because the offense got lucky " is saying something that isn't total nonsense. (I guess they're saying "Basically no passes that look like the first one get intercepted, but a good fraction of passes that look like the second get intercepted.") Combine this with, again, the relatively small numbers involved in NFL football, which imply that even if we were observing some perfect random process we'd expect a lot of variance around infinite populatin average over the course of a season, and I can easily imagine this is doing something that points in the direction of meaningful.
Or maybe it's all garbage!
Being a loyal Badger fan and also an admirer of his game and analysis, I do side with JJ.
There's three things here: First, the pash rush/interception stats are the easiest stats to understand and they have questionable utility. Once you get into the weird ways they grade offensive linemen, it does seem like they're just making stuff up.
But second, JJ argues that these statistics are relying on all sorts of counterfactuals that the PFF people not only can't know, but also have no insight into because
And finally, JJ also points out that PFF needs to be controversial as otherwise no one will listen to them. They are necessarily cooking their ratings to spice up engagement.
I get your points about how you can argue that these different stats maybe are capturing something, but in general, statistics flattens experience into some clean numbers. And it's just funny that in football where you have such small sample sizes, stating those clean numbers without error bars is inevitably just infotainment, no?
Ive watched lots of Texans game (big houston sports fan here), and I can only imagine CJ stroud has a low PFF score is because he throws really risky balls that somehow always end up landing perfectly in the WR hands. It’s what makes him so special because he makes low-medium risk high reward plays that 99% of QBs would be high risk high reward plays. Some are crazy behind the shoulder plays that are like inches away from the CBs or Safetys hand, but no cigar. And he does this on a really consistent basis. This is where I suppose the eye test beats out statistics and statistics just cant capture the nuances of how good CJ Stroud is.
CJ Stroud is such a great example because no one predicted he would be as good as he is, especially with the rest of that roster. And yet, analytics...
CJ Stroud is a good example of how analytics probably cannot capture how good he is because he makes very ballsy throws that analytics would frown upon. But he passes all the eyes test because his throws are pretty incredible and impossible to make for most QBs. I do think there is merit in analytics though, and I would be pretty upset if my Houston teams (in NBA/NFL/MLB) disregarded analytics as it would probably make them a worse team. In this regard, given that sports is a game of margins and most teams do not have the massive allowance and allure that teams in LA/NYC has (see MLB), analytics has helped smaller market teams get an edge over huge market teams with their multitude of superstars. Ultimately though, in sports, nothing beats natural talent and good old-fashioned proper coaching, practice, and execution (which will always beat analytics on its own).
The term "analytics" is so ill defined that I don't know what to think. Here's a dumb example: one of Bill Belichick's famous moves in New England was switching to a 3-4 defense because he thought it would be easier to manage under the salary cap. This clearly required analysis and cost-benefit projections. Is that analytics?
Yes I would consider that analytics. I think every good coach regardless of the time period did analytics to make decisions on what plays to run against certain teams. Now you just have bigger technology to help you facilitate that faster than before, and you would be a fool not to take advantage of that as a sport team. Things like machine learning / data science is just one of many analytics a team should be operating under. But having that doesn’t mean coaches should (and good coaches generally don’t) disregard old school ways of doing “analytics” like reviewing video tapes of their opposing team to prepare plays and formations to maximize their own team’s chance of winning. In the world of experimental design, we would call this a good old fashioned observational study of a participant’s reaction against certain treatments (i.e. I noticed a trend that opposing team does poorly against this formation and play so we should practice this play and execute it against them to give ourselves more points and increase our chances of beating them.)
"It’s a beautiful example of how statistics, even if they are meaningless, can have tangible negative consequences." Maybe I'm dense, but I don't see any examples of tangible negative consequences given. What were awards and contracts based beforehand? Is the suggestion that we should stop collecting stats, or simply interpret them better?
I agree that it's especially hard to interpret stats in an adversarial setting, but I don't have a better idea for running things in general than "collect as much data as you can and carefully apply all your knowledge when interpreting them.".
If the stats are made up, but a major network that carries major games starts arguing they are important, that's letting weird media pressure make statistics meaningful for the sake of infotainment. Why is that good?
I didn't say any of this is good. I asked for examples of tangible harm, which you claimed exist in your original post.
I agree it's reasonable to suspect that people are reading too much into stats that seem hard to interpret correctly, but again, I'm also asking: what's your proposed alternative?