> There’s an important lesson for the optimization community in understanding why PyTorch became super popular, but GAMS remained niche for 50 years.
I would say there's a lot of software-specific and community-specific factors behind the divergent outcomes, to me it's not just a lack of a "silver bullet" modelling format in optimization. As you noted a while ago in the "Github descent" posts, one salient factor behind the rise of modern ML and DL methods is widespread code sharing, reuse, and comparison on different benchmarks. You can pip install pytorch and get ready to learn deeply in a matter of seconds, whereas if you run GAMS or AMPL you have to start by checking whether you qualify for some free license or another or have your employer procure it. Nowadays the open-source optimization stack has improved a lot, but there are still limitations. Let's say you work in Python: *
- You can use Pyomo, which covers a lot of what you want in a mathematical programming language, but whose performance is (or was) still a bit constrained by the interpretation overhead and the reliance on the AMPL NL library. For nonlinear stuff, if you use anything beyond the standard mathematical operations set you needed to manually build e.g. interpolants.
- You can use CasADi, which is great at what it does, but that does not cover any mixed-integer stuff.
- You can use something like Python-MIP which has a simpler syntax than Pyomo but, again, you can't go nonlinear.
- You can use Cvxpy, which implements Boyd's DCP approach, but then nonconvex operations or domains are out of the scope.
- You can use the GAMS bindings, but that's again commercial.
There is a project that is kind of covering all this space in a single modelling language: JuMP, which allows you to model all the problem classes I mention and even use external functions through the registering mechanism. The downside? You have to opt-in to Julia, which is nice but still niche. While it does not have all the bells and whistles of commercial systems (e.g. as I understand GAMS can do things like automatic exact linearization of binary products so that you don't have to manually reformulate), it does deliver a fully-featured open-source optimization modelling language. So I'm a bit more optimistic on the "MP language dream", even though I fully agree with the tool chest comment when applied to solvers (where the commercial / open-source divide is more striking than in the mathematical modelling libraries). The only caveat here is that, in many instances, you'd rather use a different interface for an optimization component that does not go through any modelling layer, but feeds itself the functions and derivatives to a lighter solver interface.
Now, if you use PyTorch, you don't just have a passable open-source solution: you have an industry-standard tool along with an ecosystem of open-source implementations for basically any DNN architecture ever published.
Another factor, I'd say, is the scope of the "optimization problem": things like "image classification" problems are all alike, but every optimization problem is a mathematical problem in its own way. This disparity makes it harder to achieve a critical sharing mass, especially when many of the most experienced researchers are working in competitive enterprise settings.
* Some of these impressions might be outdated, please correct me in that case.
I understand this is a technical course, but I am curious about your thoughts on the ethical implications of some of your observations here---in this post and the post prior to this one. Do solution degeneracy and modeling choice options have implications when viewed from an optimization as a social problem solving perspective (See: http://bactra.org/weblog/918.html)
I look forward to your post addressing the questions (and the book). Since you are not opposed to looking back in time and dabble with history of ideas, I'll plug Ida Hoos 70s book Systems Analysis in Public Policy here (https://www.ucpress.edu/books/systems-analysis-in-public-policy/paper). The book is a criticism of application of systems analysis methods of the lore and imo relevant to the present as well.
old tool chest in the garage is so based. Truly, this is the way
Lauren complained that we don't have room for a tool chest in our house. We may have to move out to the burbs.
Truly based is having the CNC mill in the garage.
Did I ever tell you about Andrew Nobel having a Craftsman tool chest in his office at UNC? ‘Cause that’s what he has in his office.
because he does his own office repairs rather than calling facilities?
I dunno man, but he likes industrial music like Throbbing Gristle.
> There’s an important lesson for the optimization community in understanding why PyTorch became super popular, but GAMS remained niche for 50 years.
I would say there's a lot of software-specific and community-specific factors behind the divergent outcomes, to me it's not just a lack of a "silver bullet" modelling format in optimization. As you noted a while ago in the "Github descent" posts, one salient factor behind the rise of modern ML and DL methods is widespread code sharing, reuse, and comparison on different benchmarks. You can pip install pytorch and get ready to learn deeply in a matter of seconds, whereas if you run GAMS or AMPL you have to start by checking whether you qualify for some free license or another or have your employer procure it. Nowadays the open-source optimization stack has improved a lot, but there are still limitations. Let's say you work in Python: *
- You can use Pyomo, which covers a lot of what you want in a mathematical programming language, but whose performance is (or was) still a bit constrained by the interpretation overhead and the reliance on the AMPL NL library. For nonlinear stuff, if you use anything beyond the standard mathematical operations set you needed to manually build e.g. interpolants.
- You can use CasADi, which is great at what it does, but that does not cover any mixed-integer stuff.
- You can use something like Python-MIP which has a simpler syntax than Pyomo but, again, you can't go nonlinear.
- You can use Cvxpy, which implements Boyd's DCP approach, but then nonconvex operations or domains are out of the scope.
- You can use the GAMS bindings, but that's again commercial.
There is a project that is kind of covering all this space in a single modelling language: JuMP, which allows you to model all the problem classes I mention and even use external functions through the registering mechanism. The downside? You have to opt-in to Julia, which is nice but still niche. While it does not have all the bells and whistles of commercial systems (e.g. as I understand GAMS can do things like automatic exact linearization of binary products so that you don't have to manually reformulate), it does deliver a fully-featured open-source optimization modelling language. So I'm a bit more optimistic on the "MP language dream", even though I fully agree with the tool chest comment when applied to solvers (where the commercial / open-source divide is more striking than in the mathematical modelling libraries). The only caveat here is that, in many instances, you'd rather use a different interface for an optimization component that does not go through any modelling layer, but feeds itself the functions and derivatives to a lighter solver interface.
Now, if you use PyTorch, you don't just have a passable open-source solution: you have an industry-standard tool along with an ecosystem of open-source implementations for basically any DNN architecture ever published.
Another factor, I'd say, is the scope of the "optimization problem": things like "image classification" problems are all alike, but every optimization problem is a mathematical problem in its own way. This disparity makes it harder to achieve a critical sharing mass, especially when many of the most experienced researchers are working in competitive enterprise settings.
* Some of these impressions might be outdated, please correct me in that case.
I understand this is a technical course, but I am curious about your thoughts on the ethical implications of some of your observations here---in this post and the post prior to this one. Do solution degeneracy and modeling choice options have implications when viewed from an optimization as a social problem solving perspective (See: http://bactra.org/weblog/918.html)
Thanks!
This is a topic that's very important to me, and I talk a lot about it in a book that I just wrote (hopefully out next year).
But I'll definitely address this question during the semester. Coming soon. I'll also read cosma's post.
I look forward to your post addressing the questions (and the book). Since you are not opposed to looking back in time and dabble with history of ideas, I'll plug Ida Hoos 70s book Systems Analysis in Public Policy here (https://www.ucpress.edu/books/systems-analysis-in-public-policy/paper). The book is a criticism of application of systems analysis methods of the lore and imo relevant to the present as well.