Hah, I often need to brush up, too. But also, the hard part for me is figuring out what the fundamentals _are_! Articulating them clearly has been a constant pedagogical challenge.
"Despite the rhetoric on Twitter, I think machine learning does have design principles beyond Scale-HODL-YOLO. I think these design principles are worth explaining and yield better machine learning systems."
I would add one more thing -- even if machine learning *didn't* have design principles, I think it eventually *would* have them, and that they would be worth explaining and yield better machine learning systems. For instance, people were doing things we retrospectively recognize as statistics for a long time without any real principles of experimental design or analysis, and with time such principles were put together, and they actually did help us do much better statistics (even though I know you think those principles are *by now* too rigidly and unthinkingly adhered to!)
Yes, though prediction by pattern recognition is older than probability itself. The challenge in ML is disentangling the unsolvable parts of pattern recognition from the parts that embody good current practice.
Have really been enjoying your trajectory here as I also struggle with these ideas while bracing for the new semester.
I am curious if you will engage with, or think your students ought to engage with, the like social & economic reasons (or consequences) for the prominence of 'bitter lesson' thinking? Is it really just pure chance or poor pedagogy have led to this belief that access to capital, not knowledge, is most important?
A friend of mine wrote me to say that there is a good analogy between Moore's Law and Bitter Lesson Scaling. Moore's Law wasn't a law of nature, it was a neutral observation that became an industrial desideratum. A similar case can definitely be made about the Bitter Lesson.
I used to think Feyerabend derived "anything goes" from a critical analysis epistemological foundations. But I've since concluded that it was motivated reasoning, designed to justify his faith in "alternative medicine"
"Build an agent to go out into the world and learn from scratch in any environment without any prior knowledge" - Why is this sci fi nonsense? We don't have any algorithms for this, yet. Do you believe this isn't possible in principle? Why?
Being at a “fairly top” ml lab, I would say way more people than you’d expect need to learn the fundamentals. I even often want a brush up.
Hah, I often need to brush up, too. But also, the hard part for me is figuring out what the fundamentals _are_! Articulating them clearly has been a constant pedagogical challenge.
Would argue that the real fundamentals of AI are buried deep in 18th/19th century epistemology and 20th century philosophy of science.
This was still one of the best classes I have ever taken! I’m glad you’re continuing it.
Wow, Kevin. Thank you so much.
"Despite the rhetoric on Twitter, I think machine learning does have design principles beyond Scale-HODL-YOLO. I think these design principles are worth explaining and yield better machine learning systems."
I would add one more thing -- even if machine learning *didn't* have design principles, I think it eventually *would* have them, and that they would be worth explaining and yield better machine learning systems. For instance, people were doing things we retrospectively recognize as statistics for a long time without any real principles of experimental design or analysis, and with time such principles were put together, and they actually did help us do much better statistics (even though I know you think those principles are *by now* too rigidly and unthinkingly adhered to!)
Yes, though prediction by pattern recognition is older than probability itself. The challenge in ML is disentangling the unsolvable parts of pattern recognition from the parts that embody good current practice.
Have really been enjoying your trajectory here as I also struggle with these ideas while bracing for the new semester.
I am curious if you will engage with, or think your students ought to engage with, the like social & economic reasons (or consequences) for the prominence of 'bitter lesson' thinking? Is it really just pure chance or poor pedagogy have led to this belief that access to capital, not knowledge, is most important?
A friend of mine wrote me to say that there is a good analogy between Moore's Law and Bitter Lesson Scaling. Moore's Law wasn't a law of nature, it was a neutral observation that became an industrial desideratum. A similar case can definitely be made about the Bitter Lesson.
Great stuff . Thank you for your insights
I used to think Feyerabend derived "anything goes" from a critical analysis epistemological foundations. But I've since concluded that it was motivated reasoning, designed to justify his faith in "alternative medicine"
"Build an agent to go out into the world and learn from scratch in any environment without any prior knowledge" - Why is this sci fi nonsense? We don't have any algorithms for this, yet. Do you believe this isn't possible in principle? Why?