Charles Manski is really interesting in this area, because he gives us a way to think of a continuum of approaches, rather than Two Cultures. You can start with a purely empirical data-driven model, then start adding parametric and structural assumptions to it until your forecasting perrformance degrades to the point where you don't believe the model any more.
Actually, thinking about it, I'm actually talking about David Hendry and the "General-to-specific" approach to macro modelling, it's just tied up with Manski in my mind because they both have a similar way of thinking about adding parametric structure a little at a time, like flour to a cake
In Breiman's list, some - most? - of the problems don't really support the "throw a neural net at it" for simple prediction. First of all, it'd probably be gradient boost models, whose superiority for some of these would prove your point, however their clear limitation - you need supervision and you'd prefer already-tabular data - is close to the limit of your argument too.
For the "Sources of court delay" what exactly should I throw the net at? One hears this "just use NN" stuff all the time, and yet one looks at things and sees that often there's simply not enough unambiguous data to train something without some strong assumptions. Or one needs to come up with some architectural decisions that aren't "basic" without some statistical considerations. (These might operate in some different space though, like for VAE, which is an awesome idea - great that we can ponder statistics about a well-behaved subspace we construct, not the real world, but that's more of a shift, not foreclosure of statistics). It's also a selection problem. If you have no data and don't want to pay someone for assumptions you're either not solving it at all, or use some heuristics that you neither fit nor prove, and it's like it's not "data science" at all, thus not in the purview of your argument, while it is a major part of ad hoc statistics people use in their work. Sometimes you really need to forget about cool kids' NN architectures and make an HMM or something. Or as it happens to me sometimes, use an HMM or another strong model on top of whatever neural ones gave you
Thanks for your compelling case for experimentation! What about agentic AI, where interventions can change both the system and the environment we're trying to understand. How do we distinguish experiments that genuinely improve our understanding from those that merely improve our predictions?
Charles Manski is really interesting in this area, because he gives us a way to think of a continuum of approaches, rather than Two Cultures. You can start with a purely empirical data-driven model, then start adding parametric and structural assumptions to it until your forecasting perrformance degrades to the point where you don't believe the model any more.
Actually, thinking about it, I'm actually talking about David Hendry and the "General-to-specific" approach to macro modelling, it's just tied up with Manski in my mind because they both have a similar way of thinking about adding parametric structure a little at a time, like flour to a cake
[I hope in all my heart that the title here is an intended subsubstacking of the recent deranged mechinterp hype bomb from yesterday]
In Breiman's list, some - most? - of the problems don't really support the "throw a neural net at it" for simple prediction. First of all, it'd probably be gradient boost models, whose superiority for some of these would prove your point, however their clear limitation - you need supervision and you'd prefer already-tabular data - is close to the limit of your argument too.
For the "Sources of court delay" what exactly should I throw the net at? One hears this "just use NN" stuff all the time, and yet one looks at things and sees that often there's simply not enough unambiguous data to train something without some strong assumptions. Or one needs to come up with some architectural decisions that aren't "basic" without some statistical considerations. (These might operate in some different space though, like for VAE, which is an awesome idea - great that we can ponder statistics about a well-behaved subspace we construct, not the real world, but that's more of a shift, not foreclosure of statistics). It's also a selection problem. If you have no data and don't want to pay someone for assumptions you're either not solving it at all, or use some heuristics that you neither fit nor prove, and it's like it's not "data science" at all, thus not in the purview of your argument, while it is a major part of ad hoc statistics people use in their work. Sometimes you really need to forget about cool kids' NN architectures and make an HMM or something. Or as it happens to me sometimes, use an HMM or another strong model on top of whatever neural ones gave you
Thanks for your compelling case for experimentation! What about agentic AI, where interventions can change both the system and the environment we're trying to understand. How do we distinguish experiments that genuinely improve our understanding from those that merely improve our predictions?