Seems like just an extension of Short-conv in the linear attention methods where we instead carry the EMA of the entire sequence instead of nearest 4 tokens.
It would make sense in my opinion to do the evals based on the reason for needing Short-conv in the architectures that use it in the first place, where it was added to support exact retrieval/needle-in-haystack when we compress the attention state.
It's odd to make this arch extension and not test beyond a 2k context window, or with the needle-in-haystack benchmark.
In my head this should fail because the EMA cannot carry this granular information
Seems like just an extension of Short-conv in the linear attention methods where we instead carry the EMA of the entire sequence instead of nearest 4 tokens.
It would make sense in my opinion to do the evals based on the reason for needing Short-conv in the architectures that use it in the first place, where it was added to support exact retrieval/needle-in-haystack when we compress the attention state.
It's odd to make this arch extension and not test beyond a 2k context window, or with the needle-in-haystack benchmark.
In my head this should fail because the EMA cannot carry this granular information
How significant is a .98 decrease in perplexity. I'm a researcher in this field and I'm having trouble interpreting the implication of this.
Interesting approach to scale compute, never thought of using the per layer state like that
Isn't that "Infini-attention"? https://arxiv.org/abs/2404.07143 (2024)
I am so excited for RNNs to be cool again