As an NLP person it's very hard for me to endorse this explanation, because the transformer as it is described here simply does not work until you add the most important ingredient of all: the positional encoding.
Most of the semantics of a sentence are indicated by the precise order the tokens appear. This is easiest to see in a language like English, since it is visible in two very prominent ways.
First, the morphology is fairly derivational, meaning you can keep adding suffixes to things and English speakers will generally get what you mean. For example: derivation-al-ish-ly. You've probably never heard this specific derivation, but you probably have some idea of what it means anyway. The transformer can only capture these semantics if it has some representation of the fact that the suffixes inflect the meaning of the word's root (presumably something like `deriv`). English encodes this entirely by the position of the suffixes relative to the root. Without the positional information, there is no way to recover it, and sentences would lose much of their meaning, since English depends heavily on this construction.
Second, English has relatively few cases, and thus relies very heavily on the order of words to indicate what is the subject and what is the object. English specifically is Subject-Verb-Object, and the instinct of native English speakers to use SVO ordering is so strong, you will find them inserting nouns and pronouns into sentences that do not actually have subjects at all. "It's raining outside" is the most famous example. This sentence has no subject at all, and yet native English speakers feel a strong need to place "it's" at the beginning.
If the transformer did not have positional encodings it would be completely hopeless to recover what is the subject and what is the object from English sentences. There simply is no other way to reliably tell, other than inspecting the word positions.
It's worth noting we also do have the counterfactual. Generative language models that explicitly exclude positional information, like Latent Dirichlet Allocation, are only able to (essentially) cluster word co-occurrences. That is not as useless as it sounds—if I say "fire house burning ambulence" you kind of get the point even without a subject or an object—but it certainly does not generate the lucid, provoking sentences that modern LLMs do.
Rule #1 of reading CS papers - don't trust the anthropomorphized terms. Rule #2 - definitely don't extrapolate those terms to the real mind.
Of course, attention is almost certainly a heuristic readers use, but to even bring it up as key to intelligence begs so many questions about LLMs and the mind...
It's misleading. When it's right, it's right by coincidence, since the vast majority of algorithmic terms that seem like they should provide insight into the human mind, like "attention" could just as easily be called "finger" or "tape head".
And suddenly we're looking at people who read ticker tape or by following with their finger for insights into neuroscience?
Humans will alway try to conflate mysteries because of our drive toward parsimony.
We survived comparisons to hydraulics, clockwork, steam, circuits, information processing, and networks. I'm positive we'll survive comparisons to attention and models too.
If it's an argument from pedantism, let us remind ourselves to be language descriptivist than language prescriptivists. Prescriptivists never win.
It's bad metaphysics. If you really want to do bad metaphysics, then you can, but it's also hardly an attack on you if others promote norms of not doing bad metaphysics.
The answer is clearly "no." The fact that neural networks can decipher patterns in our language without knowing what the sentence is about or even what the individual words mean makes it obvious that the entire A.I. "boom" is based on quicksand. These A.I. companies may sprinkle in some stuff that makes it look intelligent, but the current transformer/attention system will only get us so far.
"Sentences with pronouns such as “it” in them illustrate the challenge that attention overcomes. Consider: “The animal didn’t cross the street because it was too tired.” And: “The animal didn’t cross the street because it was too wide.” To translate these sentences correctly into French, an AI model must determine that, in the first one, “it” refers to the animal, which is a masculine word in French, so the translation should use that form of the pronoun, il. But in the second sentence, “it” refers to the street, a feminine word that requires a different pronoun, elle. How can the machine figure this out?"
"Bahdanau remembered doing translation exercises when he was learning English in school. To write each next word of his translated sentence, he’d have to glance all around the source text to figure out what should come next. “Literally you have your source sentence on the left and your writing on the right, and you’ll be, like, turning your head, left-right, left-right,” he said. “That process is what I expressed as a neural mechanism.”
That simplicity seems to me the real tell that this is the real insight. The author does a good job of walking the ready past a long list of distractors and conjuring a sense of a path to an insight, and then, there's the insight - a simple encoding of a simple task: "What is the next word? How do I figure that out? Left-right. Left-right"
An IMO under-appreciated aspect of attention is that it's an ingredient for building a Turing-complete evaluator, since (for example) it lets you implement mutable RAM by treating the attended data as a write log: https://yager.io/neural/turing/turing.html
As an NLP person it's very hard for me to endorse this explanation, because the transformer as it is described here simply does not work until you add the most important ingredient of all: the positional encoding.
Most of the semantics of a sentence are indicated by the precise order the tokens appear. This is easiest to see in a language like English, since it is visible in two very prominent ways.
First, the morphology is fairly derivational, meaning you can keep adding suffixes to things and English speakers will generally get what you mean. For example: derivation-al-ish-ly. You've probably never heard this specific derivation, but you probably have some idea of what it means anyway. The transformer can only capture these semantics if it has some representation of the fact that the suffixes inflect the meaning of the word's root (presumably something like `deriv`). English encodes this entirely by the position of the suffixes relative to the root. Without the positional information, there is no way to recover it, and sentences would lose much of their meaning, since English depends heavily on this construction.
Second, English has relatively few cases, and thus relies very heavily on the order of words to indicate what is the subject and what is the object. English specifically is Subject-Verb-Object, and the instinct of native English speakers to use SVO ordering is so strong, you will find them inserting nouns and pronouns into sentences that do not actually have subjects at all. "It's raining outside" is the most famous example. This sentence has no subject at all, and yet native English speakers feel a strong need to place "it's" at the beginning.
If the transformer did not have positional encodings it would be completely hopeless to recover what is the subject and what is the object from English sentences. There simply is no other way to reliably tell, other than inspecting the word positions.
It's worth noting we also do have the counterfactual. Generative language models that explicitly exclude positional information, like Latent Dirichlet Allocation, are only able to (essentially) cluster word co-occurrences. That is not as useless as it sounds—if I say "fire house burning ambulence" you kind of get the point even without a subject or an object—but it certainly does not generate the lucid, provoking sentences that modern LLMs do.
Well, the masking could be causal and I think that's been investigated by several groups.
That's a great theory but they actually work fine without positional encodings: https://arxiv.org/abs/2512.12167
The Chinese will eventually test this theory with their 80000 character set.
Rule #1 of reading CS papers - don't trust the anthropomorphized terms. Rule #2 - definitely don't extrapolate those terms to the real mind.
Of course, attention is almost certainly a heuristic readers use, but to even bring it up as key to intelligence begs so many questions about LLMs and the mind...
Sure. but how far can we apply this logic?
--
A computer can't read a file, it is just sending electric signal, sensing the voltage, probing the magnetic field, etc.
"read" a file is anthropomorphized terms for what the machine do when processing a file.
--
Maybe we should just stop worrying and embrace the new meaning of those words?
Why not? Who has earned a monopoly on anti-anthropomorphism?
It's misleading. When it's right, it's right by coincidence, since the vast majority of algorithmic terms that seem like they should provide insight into the human mind, like "attention" could just as easily be called "finger" or "tape head".
And suddenly we're looking at people who read ticker tape or by following with their finger for insights into neuroscience?
Humans will alway try to conflate mysteries because of our drive toward parsimony.
We survived comparisons to hydraulics, clockwork, steam, circuits, information processing, and networks. I'm positive we'll survive comparisons to attention and models too.
If it's an argument from pedantism, let us remind ourselves to be language descriptivist than language prescriptivists. Prescriptivists never win.
It's bad metaphysics. If you really want to do bad metaphysics, then you can, but it's also hardly an attack on you if others promote norms of not doing bad metaphysics.
The answer is clearly "no." The fact that neural networks can decipher patterns in our language without knowing what the sentence is about or even what the individual words mean makes it obvious that the entire A.I. "boom" is based on quicksand. These A.I. companies may sprinkle in some stuff that makes it look intelligent, but the current transformer/attention system will only get us so far.
And what does that "without knowing what the sentence is even" even mean? Care to elaborate/define it?
This is the core insight, some in the original grad student author's own words:
https://www.quantamagazine.org/is-attention-the-essence-of-i...
"Sentences with pronouns such as “it” in them illustrate the challenge that attention overcomes. Consider: “The animal didn’t cross the street because it was too tired.” And: “The animal didn’t cross the street because it was too wide.” To translate these sentences correctly into French, an AI model must determine that, in the first one, “it” refers to the animal, which is a masculine word in French, so the translation should use that form of the pronoun, il. But in the second sentence, “it” refers to the street, a feminine word that requires a different pronoun, elle. How can the machine figure this out?"
"Bahdanau remembered doing translation exercises when he was learning English in school. To write each next word of his translated sentence, he’d have to glance all around the source text to figure out what should come next. “Literally you have your source sentence on the left and your writing on the right, and you’ll be, like, turning your head, left-right, left-right,” he said. “That process is what I expressed as a neural mechanism.”
That simplicity seems to me the real tell that this is the real insight. The author does a good job of walking the ready past a long list of distractors and conjuring a sense of a path to an insight, and then, there's the insight - a simple encoding of a simple task: "What is the next word? How do I figure that out? Left-right. Left-right"
An IMO under-appreciated aspect of attention is that it's an ingredient for building a Turing-complete evaluator, since (for example) it lets you implement mutable RAM by treating the attended data as a write log: https://yager.io/neural/turing/turing.html
This IS interesting and nifty!
I thought it was scaling
Among my peers, some would want to interpret "it was tired" to mean that the road had too many vehicles on it.
That would be pattern recognition
barely have enough attention to read through the subtitle