We know that biological neural networks learn faster and orders of magnitude more efficiently than synthetic neural nets. The amount of data/stimulus required to teach a growing human language, motor, social and a variety of other skills, is tiny - when compared to the mass amounts of data required to train SOTA models today.
The question is are there techniques we can adopt from bio neural nets that can enhance the training speed and efficiency of synthetic neural nets?
> The question is are there techniques we can adopt from bio neural nets that can enhance the training speed and efficiency of synthetic neural nets?
I always wonder if groups of our brain cells are able to do something like finding an energy minimum through the electromagnetic field that surrounds them (i.e. fast and efficient).
When you take into account the human brain is the product of thousands, millions of years of evolution, the comparison is hardly as remarkable as you make it seem.
The comparison is still remarkable, doubly so for the reason you mention. Evolution is like Kaizen where modern LLMs have been like a transformative leap.
The claim about sample efficiency sounds a bit strange, since they did not include the state of the art sample efficient algorithms. Like dreamer or tdmpc. Also PPO is known to be not efficient, just compute efficient.
We know that biological neural networks learn faster and orders of magnitude more efficiently than synthetic neural nets. The amount of data/stimulus required to teach a growing human language, motor, social and a variety of other skills, is tiny - when compared to the mass amounts of data required to train SOTA models today.
The question is are there techniques we can adopt from bio neural nets that can enhance the training speed and efficiency of synthetic neural nets?
> The question is are there techniques we can adopt from bio neural nets that can enhance the training speed and efficiency of synthetic neural nets?
I always wonder if groups of our brain cells are able to do something like finding an energy minimum through the electromagnetic field that surrounds them (i.e. fast and efficient).
> The question is are there techniques we can adopt from bio neural nets that can enhance the training speed and efficiency of synthetic neural nets?
The results seem to indicate that the limiting factor is rather the hardware current-state AI is running on than the algorithms.
When you take into account the human brain is the product of thousands, millions of years of evolution, the comparison is hardly as remarkable as you make it seem.
The comparison is still remarkable, doubly so for the reason you mention. Evolution is like Kaizen where modern LLMs have been like a transformative leap.
The claim about sample efficiency sounds a bit strange, since they did not include the state of the art sample efficient algorithms. Like dreamer or tdmpc. Also PPO is known to be not efficient, just compute efficient.