Ilya Sutskever, Co-Founder and Chief Scientist at OpenAI, that developed ChatGPT, says that GPT's architecture, Transformers, can obviously get us to AGI.
He also adds: We shouldn't don't think about it in terms of binary "is it enough", but "how much effort, what will be the cost of using this particular architecture"? Maybe some modification, can have enough computation efficiency benefits. Specialized brain regions are not fully hardcoded, but very adaptible and plastic. Human cortex is very uniform. You just need one big uniform architecture.
Video form: https://twitter.com/burny_tech/status/1725578088392573038
Interviewer: One question I've heard people debate a little bit is the degree to which the Transformer based models can be applied to sort of the full set of areas that you'd need for AGI. If you look at the human brain for example, you do have reasonably specialized systems, or all neural networks, be specialized systems for the visual cortex versus areas of higher thought, areas for empathy, or other sort of aspects of everything from personality to processing. Do you think that the Transformer architectures are the main thing that will just keep going and get us there or do you think we'll need other architectures over time?
Ilya Sutskever: I understand precisely what you're saying and have two answers to this question. The first is that in my opinion the best way to think about the question of Architecture is not in terms of a binary "is it enough" but "how much effort, what will be the cost of using this particular architecture"? Like at this point I don't think anyone doubts that the Transformer architecture can do amazing things, but maybe something else, maybe some modification, could have have some computer efficiency benefits. So better to think about it in terms of compute efficiency rather than in terms of can it get there at all. I think at this point the answer is obviously yes. To the question about the human brain with its brain regions - I actually think that the situation there is subtle and deceptive for the following reasons: What I believe you alluded to is the fact that the human brain has known regions. It has a speech perception region, it has a speech production region, image region, face region, it has all these regions and it looks like it's specialized. But you know what's interesting? Sometimes there are cases where very young children have severe cases of epilepsy at a young age and the only way they figure out how to treat such children is by removing half of their brain. Because it happened at such a young age, these children grow up to be pretty functional adults, and they have all the same brain regions, but they are somehow compressed onto one hemisphere. So maybe some information processing efficiency is lost, it's a very traumatic thing to experience, but somehow all these brain regions rearrange themselves. There is another experiment, which was done maybe 30 or 40 years ago on ferrets. The ferret is a small animal, it's a pretty mean experiment. They took the optic nerve of the feret which comes from its eye and attached it to its auditory cortex. So now the inputs from the eye starts to map to the speech processing area of the brain and then they recorded different neurons after it had a few days of learning to see and they found neurons in the auditory cortex which were very similar to the visual cortex or vice versa, it was either they mapped the eye to the ear to the auditory cortex or the ear to the visual cortex, but something like this has happened. These are fairly well-known ideas in AI, that the cortex of humans and animals are extremely uniform, and that further supports the idea that you just need one big uniform architecture, that's all you need.
Ilya Sutskever in No Priors podcast in 26:50 on Youtube https://www.youtube.com/watch?v=Ft0gTO2K85A
[link] [comments]