Hah, I often need to brush up, too. But also, the hard part for me is figuring out what the fundamentals _are_! Articulating them clearly has been a constant pedagogical challenge.
"Despite the rhetoric on Twitter, I think machine learning does have design principles beyond Scale-HODL-YOLO. I think these design principles are worth explaining and yield better machine learning systems."
I would add one more thing -- even if machine learning *didn't* have design principles, I think it eventually *would* have them, and that they would be worth explaining and yield better machine learning systems. For instance, people were doing things we retrospectively recognize as statistics for a long time without any real principles of experimental design or analysis, and with time such principles were put together, and they actually did help us do much better statistics (even though I know you think those principles are *by now* too rigidly and unthinkingly adhered to!)
Yes, though prediction by pattern recognition is older than probability itself. The challenge in ML is disentangling the unsolvable parts of pattern recognition from the parts that embody good current practice.
Have really been enjoying your trajectory here as I also struggle with these ideas while bracing for the new semester.
I am curious if you will engage with, or think your students ought to engage with, the like social & economic reasons (or consequences) for the prominence of 'bitter lesson' thinking? Is it really just pure chance or poor pedagogy have led to this belief that access to capital, not knowledge, is most important?
A friend of mine wrote me to say that there is a good analogy between Moore's Law and Bitter Lesson Scaling. Moore's Law wasn't a law of nature, it was a neutral observation that became an industrial desideratum. A similar case can definitely be made about the Bitter Lesson.
"Build an agent to go out into the world and learn from scratch in any environment without any prior knowledge" - Why is this sci fi nonsense? We don't have any algorithms for this, yet. Do you believe this isn't possible in principle? Why?
This was still one of the best classes I have ever taken! I’m glad you’re continuing it.
Wow, Kevin. Thank you so much.
Being at a “fairly top” ml lab, I would say way more people than you’d expect need to learn the fundamentals. I even often want a brush up.
Hah, I often need to brush up, too. But also, the hard part for me is figuring out what the fundamentals _are_! Articulating them clearly has been a constant pedagogical challenge.
"Despite the rhetoric on Twitter, I think machine learning does have design principles beyond Scale-HODL-YOLO. I think these design principles are worth explaining and yield better machine learning systems."
I would add one more thing -- even if machine learning *didn't* have design principles, I think it eventually *would* have them, and that they would be worth explaining and yield better machine learning systems. For instance, people were doing things we retrospectively recognize as statistics for a long time without any real principles of experimental design or analysis, and with time such principles were put together, and they actually did help us do much better statistics (even though I know you think those principles are *by now* too rigidly and unthinkingly adhered to!)
Yes, though prediction by pattern recognition is older than probability itself. The challenge in ML is disentangling the unsolvable parts of pattern recognition from the parts that embody good current practice.
Have really been enjoying your trajectory here as I also struggle with these ideas while bracing for the new semester.
I am curious if you will engage with, or think your students ought to engage with, the like social & economic reasons (or consequences) for the prominence of 'bitter lesson' thinking? Is it really just pure chance or poor pedagogy have led to this belief that access to capital, not knowledge, is most important?
A friend of mine wrote me to say that there is a good analogy between Moore's Law and Bitter Lesson Scaling. Moore's Law wasn't a law of nature, it was a neutral observation that became an industrial desideratum. A similar case can definitely be made about the Bitter Lesson.
Great stuff . Thank you for your insights
"Build an agent to go out into the world and learn from scratch in any environment without any prior knowledge" - Why is this sci fi nonsense? We don't have any algorithms for this, yet. Do you believe this isn't possible in principle? Why?