I think that switching from fixed token sets to something like H-Nets [0] (discussed on HN [1]) can at least improve on this issue, if not solve it completely.
Because, right now the main issue is that some languages are tokenized much more efficiently than others and it is a zero sum game, since there are a limited number of tokens and by adding tokens that are needed for one language you will inevitably have to remove tokens for another language and therefore decrease efficiency for it.
You have to read about half of the article before it mentions this, which should have been in the first paragraph:
> The researchers found the energy and accuracy disparity by measuring the amount of energy used by three open-source AI models, Alibaba’s Qwen 3, Google’s Gemma 3, and Meta’s Llama 3.1.
I feel like that info is placed appropriately. The abstract mentions that energy disparity persists across models, the rest follows the standard Intro -> Related Work -> Experiments structure
Edit: sorry, i went straight to the paper. I agree, the article itself (and the headline) are bad (as they tend to be on this topic)
How would one evaluate the energy consumption of models accessible only through Anthropic or OAI APIs? You can't fire them up on your own computers and measure power consumption.
The Anthropic and OAI APIs report token usage (input, output, and in some cases reasoning/thinking). The input tokens consist of the system prompt, the user input/messages, and any other context data.
Given the same model the only variable is the token count. The more tokens, the more passes through the LLM (to generate the next output token), the more energy used.
It's well known that non-English text consumes more tokens [1], [2] and thus more energy. This is due to two factors:
1. Latin script languages having different character pairings to English and modified characters, including language features like modifying verbs for person/tense/etc. (e.g. Spanish);
2. Non-Latin scripts that take more bytes to represent in UTF-8 and (esp. for ideographic and syllabic languages like Chinese, Japanese, and Korean) have lower common character pair sequences, meaning shorter tokens for those scripts and thus more tokens for a given text sequence.
Is the publishing cycle this long? Those models were relevant 18 months ago
Working out the kinks in your methodology obviously takes some time. But is there any structural reason not to rerun the benchmarks on current models in the final stages before submission? Assuming the results generalize to those newer models
theoretically, if the model is properly trained, it should grok that влъхвъ and magician semantically are the seam thus must land relatively close in the latent space. the same way a cat from an image encodes the same way, so that it can decode later in arbitrary language.
Already been using English to conserve tokens, and then translate the final result to other languages if necessary. I think this was already obvious to most international LLM users proficient in multiple languages.
I do use English for coding AIs. Most CS resources are in English so I thought that it should be far more efficient and accurate to use English. Some of my coworkers use our native language though.
I was able to pretty easily reproduce parts of this paper at Neuralwatt Cloud's SoA models where we do all our optimizations and pricing by energy.
Accuracy uses the paper's strict "#### N" parser and lenient fallback. acc is strict/lenient percent. len_cap is the number of responses truncated at the generation-length limit, not an energy cap. Each language has 50 responses.
Thanks for this table! Do you happen to have the fraction of accepted speculative decoding outputs as well? I know it is asking much, but do you possibly measure differences in different across speculation techniques (dspark vs dflash) and differences in framework/code used (unless you only use in house code for these and cannot run sglang or vllm)? Your table is more interesting than the blog for sure.
I think that the fact that TFA doesn't mention speculative decoding at all is not a good sign. It casts doubt both on the accuracy of the data and on the competence of whoever collected it.
I can imagine that in some cases the acceptance rate for non-English responses may be much higher, because while words on average need more tokens to spell, it is more "obvious" what the next token should be, so it is easier for the speculation model to predict it accurately. Information density per token is lower.
Or, maybe the acceptance rate is much lower, if the speculation model is tuned for English and doesn't work well with other languages.
We do use spec decode on all these models (I don't know if the paper is using that but I'm assuming not given the models listed) but not all these models have integrated spec decode stats in the responses. Turns out GLM 5.3 does though so here is a pass on that model.
Yes it looks that way. I pulled the energy data for GLM 5.3 as well just to check the core findings are roughly the same and they do seem to be so spec decode doesn't seem to have the same divergence as the prefill/decode energy (which is rougly a function of thinking/decode lenghts and less a property of individual tokens).
Language Strict Lenient Cap Tok/req J/req J/tok Req x EN Tok x EN J/tok x EN
English 100% 100% 0 548.7 230.33 0.4198 1.00 1.00 1.00
Chinese 96% 96% 1 754.0 694.47 0.9210 3.02 1.37 2.19
Russian 96% 98% 2 738.8 592.15 0.8015 2.57 1.35 1.91
French 100% 100% 0 631.0 1101.84 1.7462 4.78 1.15 4.16
S. Pashto 88% 98% 6 1332.1 208.06 0.1562 0.90 2.43 0.37
Tigrinya 56% 62% 19 3642.9 1445.84 0.3969 6.28 6.64 0.95
Shan 44% 54% 27 3674.8 3791.59 1.0318 16.46 6.70 2.46
Tibetan 2% 70% 15 2856.9 5306.68 1.8575 23.04 5.21 4.42
I actually think beyond the large increase in energy noted in the paper the decrease in energy for Russian and Chinese was a bit unexpected. I would have actually expected the Chinese DBCS to use more energy but at least for Chinese there could be a training impact here given these are Chinese models.
I didn't read the full PDF, but at least in the linked article, I think the number of native speakers as a metric penalizes English (and French), making them look more efficient in the graphs. There are about 3x more non-native speakers than native, and vast amounts of training material has been written by them.
Doesn't this indicate that there's not a lot of transfer learning between languages? If the networks read a prompt, reason about it and then present the result, only the reading and writing part would differ between languages, and that should be a small part of the full energy use.
It would be interesting to see if it would make any difference if you posttrained with translations of the same texts, i.e. focusing on teaching the model to specifically translate concepts between languages.
My understanding is that it’s about tokenization. English input is converted into less tokens because tokenizer is optimized for English. This makes huge difference because all the output tokens are fed back into the input at each token generation. And attention blocks are generally quadratic with the input tokens. So foreign language means more tokens everywhere and more tokens mean more passes and much more compute and memory.
My understanding (which may be wrong) is that English is best but most of the Romance languages do okay because English has borrowed from them so much.
I think you only start getting seriously bad impacts in non-Romance languages where they have their own alphabet, or use the English alphabet phonetically causing letters to appear next to each other that never would in English. E.g. in English "qu" is a pretty obvious token, you rarely see a Q without a U after it. Happens all the time in anglicized Middle Eastern words, though. Quran is like a whole other thing, you'd almost always expect another vowel after qu, not a consonant.
But I would subtly add it's also about the training data, which is optimized towards English. Not intentionally, just incidentally. (And this is a really hot area of improvement with a long way to go.)
Which is largely why the tokenization part that you mentioned is what it is.
Can't you train a model to translate tokens from human languages to a separate NN token language with a compressed representation like a VAE? It seems like a leaky abstraction to have these trillion parameter models be dependant on the fact that English gets tokenized in a certain way, and I assume this has been researched a lot already.
"Reason about it" is where your abstraction has sprung a leak.
LLMs do what they say on the tin: they model language. If you ask an LLM "find security vulnerabilities in my code" for hello_world.rs, the next likely tokens are "I found the following vulnerabilities", regardless of what logic or reason say.
It happens to be true that predicting next tokens on your own output mocks actual reasoning fairly well. But it still just a simulacrum rather than the real thing, and when held as an abstraction it leaks everywhere.
Understanding this is critical to avoiding the common failure case of scope creep and code slop.
There’s enough experience with LLMs now and comparisons to our own thinking that we can say LLMs are thinking and doing so not so differently from us - at least for some aspects of reasoning.
Trying to pretend otherwise at this point doesn’t help us to utilize them or understand their limits. We’re better off accepting that intelligence isn’t unique to us as a species. Then we can see that modeling language as a tool for us as a vehicle for thinking and reasoning, which philosophers like Wittgenstein already landed upon before LLMs.
I only used "reason about" as shorthand for the computation that takes place between input text gets tokenized and output tokens gets translated back to text. There are a lot of layers in the middle where the tokens no longer represent text directly but rather a combination of the different tokens through the attention and MLP projections.
There's some interesting info here, but the presentation is pretty poor. The worst languages are where the AI gets confused and loops keeps repeating the same text a bunch of time.
From the table on page 6 [1]
> Token explosion. The sentence repeated 45 times.
Figure 4/table 3 is pretty interesting. It shows that essentially all languages clustered at 0.1J/token, with a few outshots up to in some cases 4.8J/token.
They have 900 runs per language, so that's a pretty good sample size.
I often get answers in non-english languages. I don't understand why they aren't forcing the language that the user is browsing with unless otherwise specified.
I think they’re probably just passing some context on the user (region) and instructing it to use that language. I assume many queries you cannot trace to the user language because they’re just some generic search terms (‘taj mahal’,’Donald Trump China’ or whatever) so they probably instruct the model to answer in a specific language and they overdo it a bit
English is not dense at all. I had to change my thread naming to Chinese because the side bar in codex is narrow and English just can’t be as expressive per word
Written English is still pretty good when compared to all languages. When considering UTF-8 bytes, a English is almost as good as Chinese, since Chinese uses about 3 bytes per character.
For spoken language, French and English are at the top, but most languages are about the same bits/second. Languages with less information density are just spoken faster to make up for it.
How fast people speak is more a cultural thing than a language thing. People in Texas, for example, speak more slowly than other speakers of American English.
Yeah I’d imagine having several thousand pictograms is more information dense, but there are huge drawbacks to using pictograms in computing that alphabets do not have, text input being a major one.
I read an article once that talk about this the first few layers are like a translation layers then middle is thinking and the output layers translate it back.
It’s pretty interesting actually, you can cut out the middle layers and stack them back to back and that increases the models performance without much memory increase.
If you want to try it out you can get a LLM with no tools and pass it base64 and it will understand what’s in the message
I don't think the concept of natural language reasoning tokens is here to stay
Current open source LLMs have an emergent structure where the first couple layers transform the input into some internal representation. Then the middle does multiple reasoning steps, and the end transforms it back to one output token. You can repeat the middle to get a stronger, slower model, so it is abstract reasoning. And Anthropic had a blog post about actually looking at what the middle does and mapping it back to concept for inspection
With that in mind, what's the point of reasoning tokens? They throw rich internal state away, for the benefit of inspectability and variable-length reasoning. But now that we understand the internal structure better we can dynamically vary how many loops the model spins through those middle layers, and still get enough insight into what's happening by decoding internal state. This takes some reengineering of model architecture and inference engines (especially to still get good batch performance), but the potential benefits seem to big to ignore
It seems a ton of the chain of thought tokens are English. It just processes weights, though, not tokens - maybe compute efficiency for using "cached" tokens/weights or it more efficiently learned to process English tokens by virtue of their greater weight in the corpus.
(Longer explanation though: I don't think german per se sucks, but
english is in many ways much simpler than other languages, excluding
the fact that english has more words, e. g. more expressions in
general. German is more convoluted, you have "der", "die", "das";
in english you just have "the". There are also super-long words
in german; you don't have that in english usually. One of my
favourite is "Donaudampfschiffahrtsgesellschaftskapitän" - the
semi-literal translation is, sort of, a "marine" captain on the
danube river, as part of an incorporation/company. See that
wikipedia article, but don't buy it 100%, since nobody is really
using that word anyway; it is just a demo of what monstrous
word combinations may exist in german: https://de.wikipedia.org/wiki/Donaudampfschiffahrtsgesellsch...)
So I think the title should be "more energy on more complex
languages", because I think this would be somewhat more correct;
as well as most likely the fact, that there are more english texts
than non-english (singular) texts as example. You can see this on
Wikipedia, where the english wikipedia is miles ahead of most other
languages too. More people use english and I bet there are more
english texts too. And Generative AI will prefer english over other
languages in general.
> One of my favourite is "Donaudampfschiffahrtsgesellschaftskapitän" - the semi-literal translation is, sort of, a "marine" captain on the danube river, as part of an incorporation/company. See that wikipedia article, but don't buy it 100%, since nobody is really using that word anyway; it is just a demo of what monstrous word combinations may exist in german
People keep saying that sort of thing but we have a similar thing in Norwegian where we are allowed to combine words to make a new “word”. But people who use other languages that don’t do this sort of combining of words misunderstand the (in)significance of combined words.
Take for example menneskerettighetserklæringen. This is the name we use in Norwegian for the Universal Declaration of Human Rights. Menneskerettighetserklæringen, wow what a long and strange word right? No. It is formed from three words in sequence, “menneske” (human), “rettighet” (right), “erklæring” (declaration).
It’s just a quirk of Norwegian and German and other languages that do the same, that we consider that we should removing spaces between words like this. But there is no actual meaningful difference between a Norwegian saying hey guys we came up with this thing and we’re gonna call it “menneskerettighetserklæringen”, and an English speaker saying the same thing about the “Universal Declaration of Human Rights”. Only that we remove the spaces and you don’t.
The named thing gets its dedicated Wikipedia article in both languages all the same:
I mean, this was pretty obvious once companies started saying that non-English text uses around a third more tokens. More tokens generally mean more energy used, so it's not exactly surprising.
The part that surprised me most was not the energy per token, which the paper says clusters around 0.1J across languages, but how much of the gap comes from output length: token explosion and models looping and repeating the same text, sometimes 45 times, in the worst cases. That makes this a fairness problem as well as an efficiency one, since per-token pricing means non-English users can pay more for a worse answer. I would like to see the benchmark rerun on current models, though, since the ones tested here are already about 18 months old.
I think that switching from fixed token sets to something like H-Nets [0] (discussed on HN [1]) can at least improve on this issue, if not solve it completely.
Because, right now the main issue is that some languages are tokenized much more efficiently than others and it is a zero sum game, since there are a limited number of tokens and by adding tokens that are needed for one language you will inevitably have to remove tokens for another language and therefore decrease efficiency for it.
[0] https://www.cartesia.ai/blog/from-tokens-to-bytes-part-1
[1] https://news.ycombinator.com/item?id=50278253
You have to read about half of the article before it mentions this, which should have been in the first paragraph:
> The researchers found the energy and accuracy disparity by measuring the amount of energy used by three open-source AI models, Alibaba’s Qwen 3, Google’s Gemma 3, and Meta’s Llama 3.1.
I feel like that info is placed appropriately. The abstract mentions that energy disparity persists across models, the rest follows the standard Intro -> Related Work -> Experiments structure
Edit: sorry, i went straight to the paper. I agree, the article itself (and the headline) are bad (as they tend to be on this topic)
How would one evaluate the energy consumption of models accessible only through Anthropic or OAI APIs? You can't fire them up on your own computers and measure power consumption.
The Anthropic and OAI APIs report token usage (input, output, and in some cases reasoning/thinking). The input tokens consist of the system prompt, the user input/messages, and any other context data.
Given the same model the only variable is the token count. The more tokens, the more passes through the LLM (to generate the next output token), the more energy used.
It's well known that non-English text consumes more tokens [1], [2] and thus more energy. This is due to two factors:
1. Latin script languages having different character pairings to English and modified characters, including language features like modifying verbs for person/tense/etc. (e.g. Spanish);
2. Non-Latin scripts that take more bytes to represent in UTF-8 and (esp. for ideographic and syllabic languages like Chinese, Japanese, and Korean) have lower common character pair sequences, meaning shorter tokens for those scripts and thus more tokens for a given text sequence.
[1] https://ikriv.com/blog/?p=5322 "LLM tokens and foreign languages" (5 Aug 2025)
[2] https://medium.com/@programmerraja/how-your-llm-costs-5x-mor...
> token usage
It is not exactly accurate, because it ignores speculative decoding which can often save a lot of time, compute and energy.
See this branch: https://news.ycombinator.com/item?id=50441444#50497854
So basically, the tokenizer is optimized for English and performs worse on other languages?
Look at the token usage counts.
It's explained too..
Is the publishing cycle this long? Those models were relevant 18 months ago
Working out the kinks in your methodology obviously takes some time. But is there any structural reason not to rerun the benchmarks on current models in the final stages before submission? Assuming the results generalize to those newer models
Unfortunately it is this long. I know people in academia studying human-AI interfaces and up until recently they were still using gpt-3.5.
I don’t think that’s true for SOTA models. The responses come just as fast in German as in English.
theoretically, if the model is properly trained, it should grok that влъхвъ and magician semantically are the seam thus must land relatively close in the latent space. the same way a cat from an image encodes the same way, so that it can decode later in arbitrary language.
I think that depends to the language not the LLM. they do not translate, but if a lnaguage express same thing with longer tokens that counts.
This reminded me of this link, which I've just posted to HN
https://partyfly.github.io/tokenizer-fairness-2026/
Already been using English to conserve tokens, and then translate the final result to other languages if necessary. I think this was already obvious to most international LLM users proficient in multiple languages.
I do use English for coding AIs. Most CS resources are in English so I thought that it should be far more efficient and accurate to use English. Some of my coworkers use our native language though.
I was able to pretty easily reproduce parts of this paper at Neuralwatt Cloud's SoA models where we do all our optimizations and pricing by energy.
Accuracy uses the paper's strict "#### N" parser and lenient fallback. acc is strict/lenient percent. len_cap is the number of responses truncated at the generation-length limit, not an energy cap. Each language has 50 responses.
Across the four low-resource versus four high-resource languages:Thanks for this table! Do you happen to have the fraction of accepted speculative decoding outputs as well? I know it is asking much, but do you possibly measure differences in different across speculation techniques (dspark vs dflash) and differences in framework/code used (unless you only use in house code for these and cannot run sglang or vllm)? Your table is more interesting than the blog for sure.
I think that the fact that TFA doesn't mention speculative decoding at all is not a good sign. It casts doubt both on the accuracy of the data and on the competence of whoever collected it.
I can imagine that in some cases the acceptance rate for non-English responses may be much higher, because while words on average need more tokens to spell, it is more "obvious" what the next token should be, so it is easier for the speculation model to predict it accurately. Information density per token is lower.
Or, maybe the acceptance rate is much lower, if the speculation model is tuned for English and doesn't work well with other languages.
We do use spec decode on all these models (I don't know if the paper is using that but I'm assuming not given the models listed) but not all these models have integrated spec decode stats in the responses. Turns out GLM 5.3 does though so here is a pass on that model.
Thanks. So the acceptance rates are rather similar across languages.
Yes it looks that way. I pulled the energy data for GLM 5.3 as well just to check the core findings are roughly the same and they do seem to be so spec decode doesn't seem to have the same divergence as the prefill/decode energy (which is rougly a function of thinking/decode lenghts and less a property of individual tokens).
I actually think beyond the large increase in energy noted in the paper the decrease in energy for Russian and Chinese was a bit unexpected. I would have actually expected the Chinese DBCS to use more energy but at least for Chinese there could be a training impact here given these are Chinese models.
I didn't read the full PDF, but at least in the linked article, I think the number of native speakers as a metric penalizes English (and French), making them look more efficient in the graphs. There are about 3x more non-native speakers than native, and vast amounts of training material has been written by them.
Doesn't this indicate that there's not a lot of transfer learning between languages? If the networks read a prompt, reason about it and then present the result, only the reading and writing part would differ between languages, and that should be a small part of the full energy use.
It would be interesting to see if it would make any difference if you posttrained with translations of the same texts, i.e. focusing on teaching the model to specifically translate concepts between languages.
My understanding is that it’s about tokenization. English input is converted into less tokens because tokenizer is optimized for English. This makes huge difference because all the output tokens are fed back into the input at each token generation. And attention blocks are generally quadratic with the input tokens. So foreign language means more tokens everywhere and more tokens mean more passes and much more compute and memory.
My understanding (which may be wrong) is that English is best but most of the Romance languages do okay because English has borrowed from them so much.
I think you only start getting seriously bad impacts in non-Romance languages where they have their own alphabet, or use the English alphabet phonetically causing letters to appear next to each other that never would in English. E.g. in English "qu" is a pretty obvious token, you rarely see a Q without a U after it. Happens all the time in anglicized Middle Eastern words, though. Quran is like a whole other thing, you'd almost always expect another vowel after qu, not a consonant.
This is absolutely correct.
But I would subtly add it's also about the training data, which is optimized towards English. Not intentionally, just incidentally. (And this is a really hot area of improvement with a long way to go.)
Which is largely why the tokenization part that you mentioned is what it is.
Can't you train a model to translate tokens from human languages to a separate NN token language with a compressed representation like a VAE? It seems like a leaky abstraction to have these trillion parameter models be dependant on the fact that English gets tokenized in a certain way, and I assume this has been researched a lot already.
"Reason about it" is where your abstraction has sprung a leak.
LLMs do what they say on the tin: they model language. If you ask an LLM "find security vulnerabilities in my code" for hello_world.rs, the next likely tokens are "I found the following vulnerabilities", regardless of what logic or reason say.
It happens to be true that predicting next tokens on your own output mocks actual reasoning fairly well. But it still just a simulacrum rather than the real thing, and when held as an abstraction it leaks everywhere.
Understanding this is critical to avoiding the common failure case of scope creep and code slop.
There’s enough experience with LLMs now and comparisons to our own thinking that we can say LLMs are thinking and doing so not so differently from us - at least for some aspects of reasoning.
Trying to pretend otherwise at this point doesn’t help us to utilize them or understand their limits. We’re better off accepting that intelligence isn’t unique to us as a species. Then we can see that modeling language as a tool for us as a vehicle for thinking and reasoning, which philosophers like Wittgenstein already landed upon before LLMs.
I only used "reason about" as shorthand for the computation that takes place between input text gets tokenized and output tokens gets translated back to text. There are a lot of layers in the middle where the tokens no longer represent text directly but rather a combination of the different tokens through the attention and MLP projections.
There's some interesting info here, but the presentation is pretty poor. The worst languages are where the AI gets confused and loops keeps repeating the same text a bunch of time.
From the table on page 6 [1]
> Token explosion. The sentence repeated 45 times.
Figure 4/table 3 is pretty interesting. It shows that essentially all languages clustered at 0.1J/token, with a few outshots up to in some cases 4.8J/token.
They have 900 runs per language, so that's a pretty good sample size.
[1]: https://aclanthology.org/2026.emnlp-main.2113.pdf
Google AI overviews refusing to answer in English unless explicitly instructed with "Answer in English only" is pretty annoying.
For me, it's always in the language I'm asking in, though my Google account language is set to English.
I often get answers in non-english languages. I don't understand why they aren't forcing the language that the user is browsing with unless otherwise specified.
I think they’re probably just passing some context on the user (region) and instructing it to use that language. I assume many queries you cannot trace to the user language because they’re just some generic search terms (‘taj mahal’,’Donald Trump China’ or whatever) so they probably instruct the model to answer in a specific language and they overdo it a bit
But my browser is sending my language.
When I was googling something about oil refining equipment it started answering in Arabic, obviously.
Training is what makes neural networks faster, language usage makes one fluent and English’s frequent usage speeds up generative translation.
why not to test different language with similar and repeated tasks?
The language inter-operability is improving every movement for many models.
Use different tokenizers for different languages. Or at least, use different tokenizers for different scripts.
I suspect the quality might be higher too when using more energy.
Isn't English one of the most information dense languages?
English is not dense at all. I had to change my thread naming to Chinese because the side bar in codex is narrow and English just can’t be as expressive per word
Written English is still pretty good when compared to all languages. When considering UTF-8 bytes, a English is almost as good as Chinese, since Chinese uses about 3 bytes per character.
For spoken language, French and English are at the top, but most languages are about the same bits/second. Languages with less information density are just spoken faster to make up for it.
How fast people speak is more a cultural thing than a language thing. People in Texas, for example, speak more slowly than other speakers of American English.
Here's a study on it, I don't know if they included Texans or not: https://verbavia.com/blog/which-language-is-spoken-the-faste...
Maybe Chinese / Japanese / Korean is more
Yeah I’d imagine having several thousand pictograms is more information dense, but there are huge drawbacks to using pictograms in computing that alphabets do not have, text input being a major one.
How does Multi-Token Prediction factor into this?
Clearly, to save water and the climate, international AI users must learn and use English.
Does AI process maybe in English?
Internally I don’t think anyone knows.
I read an article once that talk about this the first few layers are like a translation layers then middle is thinking and the output layers translate it back.
It’s pretty interesting actually, you can cut out the middle layers and stack them back to back and that increases the models performance without much memory increase.
If you want to try it out you can get a LLM with no tools and pass it base64 and it will understand what’s in the message
In Deepseek app you can see its reasoning tokens.
If I ask something in my native language (Korean), the model often switches into English in the middle and translate the answer into Korean.
But not sure about what happens below the token, in embedding space or per decoder layers.
Wonder if this is the final step in English becoming true lingua franca. It's what every single AI will be using internally.
I don't think the concept of natural language reasoning tokens is here to stay
Current open source LLMs have an emergent structure where the first couple layers transform the input into some internal representation. Then the middle does multiple reasoning steps, and the end transforms it back to one output token. You can repeat the middle to get a stronger, slower model, so it is abstract reasoning. And Anthropic had a blog post about actually looking at what the middle does and mapping it back to concept for inspection
With that in mind, what's the point of reasoning tokens? They throw rich internal state away, for the benefit of inspectability and variable-length reasoning. But now that we understand the internal structure better we can dynamically vary how many loops the model spins through those middle layers, and still get enough insight into what's happening by decoding internal state. This takes some reengineering of model architecture and inference engines (especially to still get good batch performance), but the potential benefits seem to big to ignore
It seems a ton of the chain of thought tokens are English. It just processes weights, though, not tokens - maybe compute efficiency for using "cached" tokens/weights or it more efficiently learned to process English tokens by virtue of their greater weight in the corpus.
Non-English or non-ascii?
Seems not to be the case given Arabic’s energy score.
Well. German sucks. :)
(Longer explanation though: I don't think german per se sucks, but english is in many ways much simpler than other languages, excluding the fact that english has more words, e. g. more expressions in general. German is more convoluted, you have "der", "die", "das"; in english you just have "the". There are also super-long words in german; you don't have that in english usually. One of my favourite is "Donaudampfschiffahrtsgesellschaftskapitän" - the semi-literal translation is, sort of, a "marine" captain on the danube river, as part of an incorporation/company. See that wikipedia article, but don't buy it 100%, since nobody is really using that word anyway; it is just a demo of what monstrous word combinations may exist in german: https://de.wikipedia.org/wiki/Donaudampfschiffahrtsgesellsch...)
So I think the title should be "more energy on more complex languages", because I think this would be somewhat more correct; as well as most likely the fact, that there are more english texts than non-english (singular) texts as example. You can see this on Wikipedia, where the english wikipedia is miles ahead of most other languages too. More people use english and I bet there are more english texts too. And Generative AI will prefer english over other languages in general.
> One of my favourite is "Donaudampfschiffahrtsgesellschaftskapitän" - the semi-literal translation is, sort of, a "marine" captain on the danube river, as part of an incorporation/company. See that wikipedia article, but don't buy it 100%, since nobody is really using that word anyway; it is just a demo of what monstrous word combinations may exist in german
People keep saying that sort of thing but we have a similar thing in Norwegian where we are allowed to combine words to make a new “word”. But people who use other languages that don’t do this sort of combining of words misunderstand the (in)significance of combined words.
Take for example menneskerettighetserklæringen. This is the name we use in Norwegian for the Universal Declaration of Human Rights. Menneskerettighetserklæringen, wow what a long and strange word right? No. It is formed from three words in sequence, “menneske” (human), “rettighet” (right), “erklæring” (declaration).
It’s just a quirk of Norwegian and German and other languages that do the same, that we consider that we should removing spaces between words like this. But there is no actual meaningful difference between a Norwegian saying hey guys we came up with this thing and we’re gonna call it “menneskerettighetserklæringen”, and an English speaker saying the same thing about the “Universal Declaration of Human Rights”. Only that we remove the spaces and you don’t.
The named thing gets its dedicated Wikipedia article in both languages all the same:
https://no.wikipedia.org/wiki/Menneskerettighetserkl%C3%A6ri...
https://en.wikipedia.org/wiki/Universal_Declaration_of_Human...
I mean, this was pretty obvious once companies started saying that non-English text uses around a third more tokens. More tokens generally mean more energy used, so it's not exactly surprising.
The part that surprised me most was not the energy per token, which the paper says clusters around 0.1J across languages, but how much of the gap comes from output length: token explosion and models looping and repeating the same text, sometimes 45 times, in the worst cases. That makes this a fairness problem as well as an efficiency one, since per-token pricing means non-English users can pay more for a worse answer. I would like to see the benchmark rerun on current models, though, since the ones tested here are already about 18 months old.