[A long time reader of your argmin blog here, and also a twitter follower from behind the 🔒. So I'm taking the opportunity to interact here, because I thought about sharing the same thing replying to your Twitter thread on that subject].
A very interesting read.
Here's an attempt to "save" some of Shannon's prediction abilities by using a different source [while keeping in mind the caveat that if you make enough predictions, maybe something eventually hits...].
Earlier this year I've had a look at the following book, which similar to that CBS show was also part of the 100 anniversary celebration of MIT:
It is basically the transcription of the lectures + discussion of a conference they had where they invited all the "who and who" to talk about their wild predictions for the future of computing. Obviously some predictions are completely absurd but some are quite interesting (and particularly how they imagined future tech would look like given their current tech; there's an interesting chapter about online libraries for example).
Other than the lectures themselves and the Q&A, there were also longer invited "comments" for each lecture. One of them is a comment by Claude Shannon on the lecture of John Pierce (said Pierce was some Bell Labs bigshot who later on would be responsible for the gov invited report on Machine Translation that is said to be putting that field into its own "Winter"). In that comment, Shannon raises some pretty good predictions, *and* a much more nuanced view on the relation between biological brains and computers. I'm copying from the Twitter thread in which I shared some of this (which is the reason for all the [..] skipped parts):
[QUOTE]
[...] I believe that, in fact, there is very little similarity between the methods of operation of the computers and the brain [..]. [First] the wiring circuitry of the computers are extremely precise and methodical. A single incorrect connection will generally cause errors and malfunction. The connections in the brain appear, at least locally, to be rather random, and even large number of malfunctioning parts do not cause complete breakdown of the system. [Second] computers work on a generally serial basis, doing one small operation at a time. The nervous system, otoh, appears to be more of a parallel-type computer with a large fraction of the neurons active at any given time. [Third] it may be pointed out that most computers are either digital or analog. The nervous system seems to have a complex mixture of both representations of data.
[..]
These and other arguments suggest that efficient machines for such problems as pattern recognition, language translation, and so on, may require different type of computer than any we have today. It is my feeling that this computer will be so organized that single components do not carry out simple, easily described functions. One cannot say that this transistor is used for this purpose, but rather that this group of components together performs such and such function. If this is true, the design of such a computer may lead us into something very difficult for humans to invent [..].
Most machines are invented by breaking the overall problem down into a series of simple and perhaps previously solved problems [..] In a machine of the type I'm suggesting, it would be impractical to describe the purpose of any single component. I know of very few devices in existence which exhibit this property of diffusion of function over many components. [...] Can we design [..] a computer whose natural operation is in terms of patterns, concepts, and vague similarities rather than the sequential operations on 10 digit numbers?
What a great find! I'm headed to the library now to grab a copy of this book.
I wonder what would cause Shannon to have such nuance here whereas he expressed irrational exuberance in the CBS show. Maybe Jerry Wiesener goaded him into putting on a show for the cameras...
Right! I think many these heroes had both the "absolutely wild predictions/ideas" and the "super insightful and nuanced ideas" at the same time.
At least regarding the first few points of that quote, this issue of computation "reliability" (resilience to all kinds of noise, perturbation, etc) was something that early cybernetic people (e.g. Warren McCulloch and von Neumann; and most likely Shannon knew about that stuff) were really interested in. And the comments about a machine working in a "different way" (more statistical/distributed etc) echoes ideas put forward in "the computer and the brain" by von Neumann.
I've posted quite a few excepts about such issues on Twitter in the past, maybe I should migrate some of these to a new substack before the bird platform goes completely down the drain....
Interestingly, these are all the same points that Rosenblatt was making in "Principles of Neurodynamics." Or von Neumann in "The Computer and the Brain."
It's a hard question, I wish I knew ... But I think the key is to make precise and restricted predictions. I suppose Moore was very knowledgeable about large electronics. He saw they could get smaller. He saw that people liked to have products (in 65 they were big products, like cars, TVs, washing machines). They could get smaller, with more circuits. Whatever computers they had at the lab Moore was leading, he saw they could eventually fit in an object the size of a TV. There was a clear(ish) path from how things were like in 65 to how things could be if the prediction turned out right. In Shannon’s prediction, I guess the path was less clear. In some sense, eyes existed at the time: video and photo cameras. These pictures or videos could be featurized. But how exactly would they use these very large matrices of inputs with “flexible computer programmed to learn from experience”. Shannon was very knowledgable too, of course. But it just sounds less precise, all-encompassing ... nebulous.
All this is easy to say in retrospect. Perhaps I’m all wrong. I have no idea! From another angle, I think the first problem is mostly about engineering, while the second is about “magic”. Magic in the sense that we don’t exactly know what we want, but we know it will be awesome. It’s going to be awe-dropping. I’ve seen this pattern a few times in the social sciences with the use of machine learning. Not exactly clear what we want, but it will be a “game changer”, will make a “big splash”, will lead to a “breakthrough in the field”.
p.s. I’ve started to follow you recently on Twitter (and read a few of your blogs). Great online presence. Really appreciate the writings. You and a few others seem just more clear eyed than most on what people (even big names) do with statistics these days.
[A long time reader of your argmin blog here, and also a twitter follower from behind the 🔒. So I'm taking the opportunity to interact here, because I thought about sharing the same thing replying to your Twitter thread on that subject].
A very interesting read.
Here's an attempt to "save" some of Shannon's prediction abilities by using a different source [while keeping in mind the caveat that if you make enough predictions, maybe something eventually hits...].
Earlier this year I've had a look at the following book, which similar to that CBS show was also part of the 100 anniversary celebration of MIT:
https://mitpress.mit.edu/9780262570022/computers-and-the-world-of-the-future/
It is basically the transcription of the lectures + discussion of a conference they had where they invited all the "who and who" to talk about their wild predictions for the future of computing. Obviously some predictions are completely absurd but some are quite interesting (and particularly how they imagined future tech would look like given their current tech; there's an interesting chapter about online libraries for example).
Other than the lectures themselves and the Q&A, there were also longer invited "comments" for each lecture. One of them is a comment by Claude Shannon on the lecture of John Pierce (said Pierce was some Bell Labs bigshot who later on would be responsible for the gov invited report on Machine Translation that is said to be putting that field into its own "Winter"). In that comment, Shannon raises some pretty good predictions, *and* a much more nuanced view on the relation between biological brains and computers. I'm copying from the Twitter thread in which I shared some of this (which is the reason for all the [..] skipped parts):
[QUOTE]
[...] I believe that, in fact, there is very little similarity between the methods of operation of the computers and the brain [..]. [First] the wiring circuitry of the computers are extremely precise and methodical. A single incorrect connection will generally cause errors and malfunction. The connections in the brain appear, at least locally, to be rather random, and even large number of malfunctioning parts do not cause complete breakdown of the system. [Second] computers work on a generally serial basis, doing one small operation at a time. The nervous system, otoh, appears to be more of a parallel-type computer with a large fraction of the neurons active at any given time. [Third] it may be pointed out that most computers are either digital or analog. The nervous system seems to have a complex mixture of both representations of data.
[..]
These and other arguments suggest that efficient machines for such problems as pattern recognition, language translation, and so on, may require different type of computer than any we have today. It is my feeling that this computer will be so organized that single components do not carry out simple, easily described functions. One cannot say that this transistor is used for this purpose, but rather that this group of components together performs such and such function. If this is true, the design of such a computer may lead us into something very difficult for humans to invent [..].
Most machines are invented by breaking the overall problem down into a series of simple and perhaps previously solved problems [..] In a machine of the type I'm suggesting, it would be impractical to describe the purpose of any single component. I know of very few devices in existence which exhibit this property of diffusion of function over many components. [...] Can we design [..] a computer whose natural operation is in terms of patterns, concepts, and vague similarities rather than the sequential operations on 10 digit numbers?
[END QUOTE]
What a great find! I'm headed to the library now to grab a copy of this book.
I wonder what would cause Shannon to have such nuance here whereas he expressed irrational exuberance in the CBS show. Maybe Jerry Wiesener goaded him into putting on a show for the cameras...
Right! I think many these heroes had both the "absolutely wild predictions/ideas" and the "super insightful and nuanced ideas" at the same time.
At least regarding the first few points of that quote, this issue of computation "reliability" (resilience to all kinds of noise, perturbation, etc) was something that early cybernetic people (e.g. Warren McCulloch and von Neumann; and most likely Shannon knew about that stuff) were really interested in. And the comments about a machine working in a "different way" (more statistical/distributed etc) echoes ideas put forward in "the computer and the brain" by von Neumann.
I've posted quite a few excepts about such issues on Twitter in the past, maybe I should migrate some of these to a new substack before the bird platform goes completely down the drain....
Well, I'm upset with myself for not have been following you on Twitter. You should definitely start a substack. I'd subscribe.
Ok, for what it's worth, i'm going to give it a try. Here's my first post:
https://liorfox.substack.com/p/visual-pattern-recognition-in-1955
subscribed!
Interestingly, these are all the same points that Rosenblatt was making in "Principles of Neurodynamics." Or von Neumann in "The Computer and the Brain."
Yes, this really echoes von Neumann's ideas. I've also thought about the very same book, as I mentioned in my other reply to Ben.
Welp, I should have read your other reply more carefully!
It's a hard question, I wish I knew ... But I think the key is to make precise and restricted predictions. I suppose Moore was very knowledgeable about large electronics. He saw they could get smaller. He saw that people liked to have products (in 65 they were big products, like cars, TVs, washing machines). They could get smaller, with more circuits. Whatever computers they had at the lab Moore was leading, he saw they could eventually fit in an object the size of a TV. There was a clear(ish) path from how things were like in 65 to how things could be if the prediction turned out right. In Shannon’s prediction, I guess the path was less clear. In some sense, eyes existed at the time: video and photo cameras. These pictures or videos could be featurized. But how exactly would they use these very large matrices of inputs with “flexible computer programmed to learn from experience”. Shannon was very knowledgable too, of course. But it just sounds less precise, all-encompassing ... nebulous.
All this is easy to say in retrospect. Perhaps I’m all wrong. I have no idea! From another angle, I think the first problem is mostly about engineering, while the second is about “magic”. Magic in the sense that we don’t exactly know what we want, but we know it will be awesome. It’s going to be awe-dropping. I’ve seen this pattern a few times in the social sciences with the use of machine learning. Not exactly clear what we want, but it will be a “game changer”, will make a “big splash”, will lead to a “breakthrough in the field”.
p.s. I’ve started to follow you recently on Twitter (and read a few of your blogs). Great online presence. Really appreciate the writings. You and a few others seem just more clear eyed than most on what people (even big names) do with statistics these days.
All great points.
And thank you very much for the kind words.