Nobody will throw rocks, I think most people are curious/suspicious about the big players and wants more hands-on since we suspect that this all will come down in cost soon enough.
Seems interesting, I've been messing with a lot of continuous learning approaches lately and it's cool to see something that's built from the ground up for avoiding catastrophic forgetting. Worth a clone for sure
Is this architecture actually able to generalize or is it mostly based on memorization? Have you tried some basic tasks that require generalization? e.g. number addition etc?
The model is way too small and undertrained to make any generalization claims. I want to wait until it reads the whole corpus I gave and then test it on some simple established benchmarks to see how it will behave.
What's the advantage of doing this, versus becoming good at context management and RAG? I always found trained knowledge unreliable, given that it is lossy by construction.
Have you thought about making the whole thing "self-similar"? Every time I hear about MoE I think (and I know it's way easier thought than done) "why stay shallow"?
I mean by that: would it be possible to extend/adapt the architecture so that an expert can be a previously trained Mini-AGI model? And recurse like this? Inuitively I would think some form of generalization could happen, as higher level experts (in the recurrence stack) would become sort of the "intuition" layer.
I also like how it is very organic. It naturally grows and deletes unused elements, so in addition to traditional backprop there is also a natural selection happening in the background. Each new expert has 16 parents by the way, lol.
It interleaves random streams of 32K characters long each when reading the whole corpus, but each such stream reads continuously as you would expect. This is a necessary step to prevent just normal, not catastrophic, forgetting. I have not tested it in any other regimes yet with bigger or smaller windows. You can imagine a person that changes the activity from time to time, so I think it is justified. So there is not really "early in the stream".
What I did test though is reading 524K characters of chess data only and see how other domains have degraded. The results are in the readme under "How continual learning works" section. Spoiler: it just barely degraded the performance.
This is slop. 8M parameter dense model with context length 64 that you train on enwik9 in 2h will have 1.15 bpb. This model has 1.8 (bits per byte, lower is better).
I would like to see a benchmark and/or a video of it in action. Or at least a transcript of a chat session.
Has anyone else tested this?
This is the first thing I see in my life that really looks like proto-AGI, it deserves its name.
Nobody will throw rocks, I think most people are curious/suspicious about the big players and wants more hands-on since we suspect that this all will come down in cost soon enough.
Seems interesting, I've been messing with a lot of continuous learning approaches lately and it's cool to see something that's built from the ground up for avoiding catastrophic forgetting. Worth a clone for sure
It doesn't show any indications of solving catastrophic forgetting.
Is this architecture actually able to generalize or is it mostly based on memorization? Have you tried some basic tasks that require generalization? e.g. number addition etc?
The model is way too small and undertrained to make any generalization claims. I want to wait until it reads the whole corpus I gave and then test it on some simple established benchmarks to see how it will behave.
What kind of hardware are you using for training?
nm, I found it:
> RTX 3070 Laptop GPU with 8 GB
Super impressive.
What's the advantage of doing this, versus becoming good at context management and RAG? I always found trained knowledge unreliable, given that it is lossy by construction.
Have you thought about making the whole thing "self-similar"? Every time I hear about MoE I think (and I know it's way easier thought than done) "why stay shallow"? I mean by that: would it be possible to extend/adapt the architecture so that an expert can be a previously trained Mini-AGI model? And recurse like this? Inuitively I would think some form of generalization could happen, as higher level experts (in the recurrence stack) would become sort of the "intuition" layer.
Seeing 'Mini-AGI' and '8GB VRAM' in the same sentence is a breath of fresh air. Maybe local AGI isn't so far-fetched.
Getting conceptually closer to how the human brain works. Looking forward to more of this.
I also like how it is very organic. It naturally grows and deletes unused elements, so in addition to traditional backprop there is also a natural selection happening in the background. Each new expert has 16 parents by the way, lol.
what character prediction rates are you getting on some unseen datasets?
The held-out scores reported in the Readme IS the unseen dataset.
Have you tested what it remembers from early in the stream after a shift in the topics thrown at it?
It interleaves random streams of 32K characters long each when reading the whole corpus, but each such stream reads continuously as you would expect. This is a necessary step to prevent just normal, not catastrophic, forgetting. I have not tested it in any other regimes yet with bigger or smaller windows. You can imagine a person that changes the activity from time to time, so I think it is justified. So there is not really "early in the stream".
What I did test though is reading 524K characters of chess data only and see how other domains have degraded. The results are in the readme under "How continual learning works" section. Spoiler: it just barely degraded the performance.
This is slop. 8M parameter dense model with context length 64 that you train on enwik9 in 2h will have 1.15 bpb. This model has 1.8 (bits per byte, lower is better).
THANK YOU SO MUCH. This is the missing piece.
throwing crumpled paper ball