So the jump in intelligence was like a kid that keeps getting better at reading and then one day they suddenly can do word problems. Now the worry is the student will soon be writing the textbooks and we won't be able to check the work?
This trend is pretty shocking to me. I assumed that we had moved beyond cross-entropy loss being important with the increasing use of RL, it's wild that the trend is so clear. Really nice to have such a simple benchmark instead of the increasingly convoluted ones that are getting spun up every day.
Also, I wonder what happened with GPT 4.5? I think o1 was the start of true reasoning models, right? I was using Claude at the time so I can't remember.
It was a massive non-CoT model (for that time) that they trained as an experiment and served for a while before shelving it probably due to high inference costs.
> I assumed that we had moved beyond cross-entropy loss being important with the increasing use of RL
From an information theory perspective, cross-entropy loss (log loss) seems to me to be about as pure as it gets in terms of capturing the notion of whether prediction "succeeded". Even Solomonoff Induction minimizes log loss. Would be curious to hear others' perspectives though on why another function might be better for universal prediction purposes.
There are more general information measures, Rényi entropy, von Neumann entropy, etc. for different system setups, but for token-by-token information learning and prediction within classical ergodic processes Shannon entropy is where it's at.
Interesting that GPT-4.5 is the only discontinuity on the graph.
The oversized monster LLM has performed worse than CoT-trained o1 and o3 on the benchmarks, so CoT + RL clearly provided better "real world performance" improvements - let alone "price/performance."
But if pre-training perplexity on held-out data really does measure something like "base model IQ" or "base model g factor", then it does support the hypothesis that 4.5 still had more "intellectual potential" than the o-series.
So, if you were wondering why do AI companies train 10T+ monsters now, and are taking aim at 100T too? It's because scale works.
I'm a bit skeptical about the claim that particular abilities "emerge" at given perplexities. For instance, there are smaller models that are specifically trained on math, and I imagine those models have much higher general perplexity on other tasks.
On the other hand, assuming you have appropriate training data for a task, then yes obviously perplexity is going to measure performance on that task. Being able to predict the next task token is obviously going to be a near perfect correlation with accuracy on the task.
While intuitively it would make sense that smaller models specifically trained on math perform better than large generalist models, this has empirically proven to be untrue.
Deepmind was proving math theorems using a model specifically trained on math, and that was very quickly surpassed by generalist LLMs. It's presently not well understood why this is the case.
Yes training on math would give you spiky abilities on math. But in this case the dataset they measure perplexity is ood from training so lower perplexity on it indicates a general increase in intelligence across domains.
But there’s definitely spikiness and it’s very possible they’ve trained more on math than other domains.
Me too. Perplexity of what? If we want models to be able to reason, call tools, or solve new problems, none of those necessarily have existing data on what the next token should be. A lot of advances in frontier models come from training with RL, which is fundamentally different from next-token prediction and doesn't have a notion of perplexity.
I've been working in NLP for many years prior to the release of ChatGPT, and perplexity was generally associated with "human readability" rather than intelligence. So this is somewhat fascinating.
There was an ACL paper from Tokyo University a couple of years ago that showed, across all languages, lower perplexity does not necessarily correlate with better "human likeness" [1]. I wonder if that's also the case here (where intelligence is concerned), that we are biasing too much towards English.
The divisions they drew in the log graph are of course arbitrary, but GPT-4 (the original 8k context version from early 2023) was able to unminify JS, as well as produce an autonomous coding agent. (This was via ChatGPT, so I copy-pasted the code into files, but it both designed and implemented it.)
So "useful coder" arrived about three years earlier, by my count.
Log-perplexity is cross-entropy, and its reducible part is the KL divergence between model and corpus. I've argued that the same quantity prices the gap between any two agents, human or AI: https://www.symmetrybroken.com/maintaining-divergence/
The floor Pachocki can't locate is fundamental. No model can predict the next word better than its underlying agreement with the author's model.
Minimizing perplexity may be enough to in some sense match human-level intelligence that, combined with a computer’s ability to do the grunt work, leads to super-human results. But actually going beyond human intelligence would almost by definition imply that perplexity on human texts should increase.
If RL is merely a way to elicit latent capabilities acquired in pretraining, as the article claims, that could mean that currently there is a wall that separates human-level AI from superhuman-level AI. And no one has figured out how to break it yet.
Why would perplexity on human texts have to increase?
That would imply that "beyond human intelligence" is closer to a sidegrade than to an upgrade - becoming "superhuman" by necessity poisons your ability to predict human text.
In reality, what would preclude a "superhuman" AI from being a superset, a direct upgrade - both "very good at predicting human text" and "very good at predicting things no human is able to predict"?
This feels fundamentally wrong. As we move up the intelligence arc, I would expect perplexity to increase in general and only decrease on the pre train.
I would expect the perplexity on the original pre train to increase as reinforcement learning on verifiable domains teaches the model lessons that did not exist in the pre train. In addition alignment and other concerns, unless the are consistently enforced in pre train data quality will also move perplexity up.
I thought perplexity on simple text data has been saturated for a while (as in perplexity is uncorrelated to task success - ex. https://arxiv.org/html/2410.23771v2).
I suspect what's happening is their test dataset is actually just something really hard, like math papers or something. In that case perplexity would just be a direct measure of math knowledge, which means this blog would just be saying that better math knowledge predicts better math skills.
I've been watching 3Blue1Brown's series on compression and intelligence^[1], which seems to discuss similar ideas.
In particular, his mention of an unknown floor for loss seems directly connected to Claude Shannon's entropy rate regarding compression—having to probe human brains to estimate the entropy of English at 1 bit per character, just as OpenAI doesn't know the exact entropy rate of its held-out codebase.
>The primary held-out datasets we’ve been using since GPT-3.5 are sourced from our internal codebase, and those are the numbers reported here (for a fixed snapshot)
Am I right in understanding this to mean that the code on which we're computing perplexity was all written pre-3.5, and so it almost certainly entirely human-written? In other words, this isn't a snapshot of today's internal code base which contains a great deal of AI-authored code.
I'm curious what exactly each "base-model" here is.
We've generally believed that most model upgrades aren't new pre trainings, but instead are better post trainings of a common pre trained model. But this graph suggests every release had a better base model.
Have the public narratives around how these models are trained been misleading? Does each model release usually always include an extended, if not completely fresh, model pretrain?
The figure indicates that some dots (base models) correspond to two production models (GPT-5 / 5.1). So I guess that tells you at least which models are built on the same pre-train checkpoint, but not whether some pre-trains are, say, very minimal extensions of others.
I can believe the “not completely fresh” for some minor version bumps. I’d imagine every minor version includes more than $0 on additional pretraining for the next release. (Though sometimes, maybe they found no gains and reverted to the previous base model). If nothing else to move the training cutoff date forward.
Whenever they change architecture or even just number of layers, I’d imagine they start from scratch but I could be wrong.
Also, I imagine there’s an incentive to start fresh whenever there’s an innovation in curation of the training dataset.
> Detachment from actionable evaluations. Perplexity does not directly correspond to a task we care about
The post makes a good case that at frontier scale this doesn't matter much, because perplexity gains turn into new capabilities. However, this has been an issue for me when training local models
Some agents wanted to optimize for perplexity, and we did all kinds of experiments, but at least for what I wanted (a small conversational chat model that can run in my browser, ~8M params), that was the wrong way to go. I could get excellent ppl numbers, yet still almost no coherence
What worked best, in my case, was to get better data, specifically, hand-crafted examples for the tasks I wanted the model to be good at. I'm guessing that perplexity might track capability when the model is big enough to turn the gains into general skills, but at a few million params it mostly buys you better-sounding word statistics
Wouldn't reducing the entropy too much lead to lack of diversity of thought? An idea that I've had for a while is that LLMs are held back by having the same mind, leading to modal collapse. Ideally, an AI swarm would have different approaches to a problem so they don't all end up trying the same thing.
Drugs! Give'em psychedelics, let 'em see da colors!
Nah, just kidding.
The labs are training their models on the previous' version conversations, but I'm sure not all those conversations are weighted equal. So there are some conversations that are more "sparky", where both LLM and human work better towards achieving a solution. The next LLM iteration learns from them. In that sense, they don't need to hallucinate, but to do exactly what LLM dissenters say they do: "they just regurgitate something they saw (trained on) somewhere". To us humans, who haven't read all the books nor experienced a dozen million successfully completed tasks, that "rehashing of knowledge" (I prefer "recombination of knowledge") is more than good enough to solve ever more challenging problems...at least as long as there's a human in the loop with ever bigger ambitions.
I always had the impression that model capabilities were stagnating around GPT-5.0, and this finally broke around the release of Claude 4.5 later that year. Seems like their own graph corroborates that things were pretty stagnant then. I wonder what changed?
Oh oh! The models actually know what happen as well if you want to dig deeper. Pre-scale brute force training hit there limits. They broke through late 2025 due to RL training techniques that taught them to fail, fail, fail to succeed instead of spin off into a corner with nonsense.
Basically we started tuning them to be smarter and able to handle 2 to 4 hour tasks.
I believe this will hit it's limits as well. When IDK.
This graph specifically tries to measure pre-train perplexity. It measures the "foundation" of a given base model - before RL is applied to it.
On long term tasks, "CoT + RL" of o1/o3 beats GPT-4.5 resoundingly. But on raw base model perplexity, GPT-4.5 is a significantly better model than any of the o-series.
The curve following GPT-4.5 and o-series suggests that newer models take advantage of BOTH better base model capabilities (perplexity on held-out data) and better RL.
Given the current models from Anthropic and especially OpenAI have a hard time constructing coherent English sentences that a human can understand even while being better at code and math, I'd say they're reaching that limit; can't use a model that you can't communicate with. The Deepmind folks seem to have figured something out on that front with Neon, though, as it's both smart/capable and a clear communicator.
I'm not an expert, but I imagine it's what I wrote in a sibling comment -- training to improve agent performance (reinforcement learning with verifiable rewards).
That generates new data that's not on the Internet -- more true observations to learn from, compress, and generalize.
I think that improves the models in general, not just for agents. (And also I think it's the cause of the "Claudish" prose dialect we all started noticing)
I think that matches the timing, though I appreciate any closer analysis
Oh, I always wanted to know how the entropy of English was looking! The human baseline is both surprisingly unclear and surprisingly better than even today's frontier models (https://www.lesswrong.com/posts/SZ3jDHXHb4WF4jmbr/where-is-h...). Though this is an extrapolation from 6 years ago, so take it with a grain of salt.
There's also the Hilberg model where the entropy rate keeps falling towards zero as you consider longer and longer texts, and at least here (https://www.youtube.com/watch?v=5eqRuVp65eY) data from LLMs showed no floor. It would be unintuitive if it turned out that you can compress sensible English text to, say, O(sqrt N) length!
A good and common metric. It will be interesting to see if other labs agree, but thank you for arriving at a metric and following up on the slowdown talk. I do hope others pay attention.
I am having a really hard time imagining what qualitative difference would gpt-5 and gpt-6 have in predicting the next token? What kind of dataset is it? How big is it?
Perplexity is definitely a useful metric for pretraining, but doesn’t mean much for post-training. In fact, low perplexity in a domain during pretraining doesn’t always translate to better behavior in post-training. Moreover, given that most of the recent progress and most of the frontier compute is going into post-training, I think it isn’t a very useful metric for intelligence.
From gpt 5 haven't the actual harness played an important role? Pretty sure no model is a cyber security expert by just thinking hard enough. I feel like this article is doing a reverse causation here
For example consider this: would it be possible to build a model with such a low perplexity score but it be pretty bad at cyber security tasks? What if we removed all cyber security related data and RLVR from its training?
The article presents an observation but treats it as causation
The harness is almost irrelevant (you can prove that by running the same tasks in a minimalist harness with only a few tools and minimal prompt vs. a complex harness and long prompt like Claude Code or Codex). Often the minimal harness/prompt actually performs tasks better, faster, cheaper than the big harness. This includes on cyber security tasks.
But, RL improvements are certainly a big part of recent improvements for certain tasks. That's what got the models doing long-running coding and math and research tasks, without constant human prodding.
It's baffling how many people in these comments did not read the article at all.
This is measuring the test-set perplexity of the pretrain model the one that is trained to perform next-token prediction and only that. This number is telling you how good the next-token predictor is at the task it was trained for.
For those wondering why next-token prediction capabilities lead to ""intelligence"" (or whatever you want to call what LLMs do), we don't really know for sure, it's one of the biggest mysteries of our time. But one thing that we can empirically see is that the lower this number gets, the better the LLM is at solving problems.
RLVR, chain of thought, etc all happen after this step and can additionally increase the capability of the model. This is also presumably why the perplexity plateaued around o1-o3 but capabilities did not (they were likely focusing on RLVR instead of scaling up pretraining).
Here I was under the assumption that since GPT-4ish we mostly improved models via RLing, and that pre-training stayed (roughly) constant. Mind blown :)
> The primary held-out datasets we’ve been using since GPT-3.5 are sourced from our internal codebase, and those are the numbers reported here (for a fixed snapshot).
Why would OpenAI expect any meaningful relationship between their models’ ability to predict the next token of supposedly held-out chunks of their internal codebase and the overall capabilities of the models? I expect both to improve, but that’s about the extent of it.
I work at OpenAI. The correlation between perplexity and the quality isn’t 1, but neither is it 0. It makes sense that as a model gets better at learning pretraining data, it also gets better at generically learning whatever you’re trying to teach it afterwards. The relationship is imperfect and empirical, and if it breaks down in the future, we’ll of course adapt to different measures instead of dogmatically clinging to the old paradigm. But so far, it’s still informative.
I think his point (unless i've misunderstood) is moreso if a codebase is really the best held out dataset for such a evaluation meant to be wide-sweeping. Does this plot look any or much different if the dataset was more encompassing to include other sorts of text ?
I wouldn't say perplexity is great metric because GPT-4 and o3 have similar perplexity on this chart but o3 was a massive step improvement. Pachocki has an incentive to push perplexity as one of the creators of the original LLM paradigm but I would say we have kind of moved on from that.
This metric is purely scoring the ""intelligence"" of the base model, not including RLVR/chain of thought capabilities. It's widely believed that the pretrain determines the ceiling that RLVR can take you to.
I think it can still be a useful metric, but probably isn't the "one true" metric. The trend is pretty clear, and the acceleration of capabilities since December 2025 definitely aligns with my subjective experience.
There are probably some other axes where o3 is clearly better than o1, but with how complex intelligence is no metric will be perfect.
When we approach 0, are they gonna stop sounding cringe?
Actually, trick question I suppose -- the base models already don't sound cringe, it's post-training that adds that!
(Well, some of them do, because a lot of companies are putting synthetic data and LLM outputs into the pretraining phase (as well as bot-infused post-2020 common crawl, I suppose), but the early base models were fully cringe-free.)
Interesting that they work on weekends. Is it just because they're all basically millionaires living the dream so they don't care? Every place I've worked at has discouraged working late and weekends. Could also just be a scheduled post.
OpenAI hasn't released any competent model since opus 5.5 came out, i wonder if these labs are deciding to pass the "best model" card every quarter to each other for good public perception of all labs or something, because OpenAI can't be this incompetent
This is not the first time I've seen a chart of "Inteligence on a log scale" related to AI.
And it again feels equally dishonest/bullshitty.
You can't really make an objective statement that coding requires multiple orders of magnitude more intelligence than having a conversation. And that solving math problems requires orders of magnitude more intelligence than coding.
It's just marketing talk. In reality all three are separate skills and the real intelligence requirements are, if you ask me, not that far away from each other.
To put this in an analogy, it feels like charting "athleticity on a log scale" and putting first running a marathon, then swimming across English channel, then benchpressing 200kg, each one presented as requiring an orther of magnitude being more athletic.
>You can't really make an objective statement that coding requires multiple orders of magnitude more intelligence than having a conversation. And that solving math problems requires orders of magnitude more intelligence than coding.
Where are you getting 'orders of magnitude'? and yeah I do think it's reasonable to say that Useful Programming requires more intelligence than the median human, and Solving millenium level math problems certainly requires considerably more intelligence than the median Programmer. What exactly is so absurd or unreasonable about that?
The point is that talking, programming and solving math problems don’t lie on a single scale, and each problem requires different “skills.” Thus the comparison with running, swimming and weightlifting.
We have long since known that there is a single factor that explains ~40% of the variation of cognitive tasks and abilities in humans (similar results can be found in animals). We called that factor the general intelligence factor, g.
Humans who tend to have higher verbal skills also tend to have higher spatial skills, better memories, and faster processing speed. Someone with the cognitive ability to become an exceptional chemist might just as easily have become an exceptional mathematician or software engineer.
The knowledge and skills required are obviously different, but the underlying cognitive abilities that make someone successful in one intellectually demanding field often transfer to others.
Saying that perplexity is a metric of intelligence is misleading. Inkling-2-Mini was released recently, and I've measured its perplexity at 5.7 on a small varied text dataset. For reference, the graph OpenAI is glazing is plotting log(PPX), which would put this model at a 0.756 on the Y Axis... Which is apparently between GPT 6.2 Astra and their internal model. Perplexity can also give the same number for a model which understands text as to one that memorizes text... And as I'm pretty sure Inkling-2-Mini is smaller than GPT 6.2 Astra - and it's generally acknowledged that number of parameters equals memorized knowledge, this shows how weak this advertising stunt for OpenAI is.
The dataset matters a lot here. You'd need to run Inkling on the same holdout set that OpenAI uses here to get any useful information to compare to this graph. If you measured the perplexity on any model on reciting the declaration of independence it would probably be zero.
I don't understand why anyone needs to exponentiate existing, well known concepts like entropy and cross-entropy to invent something called "perplexity". Somebody has Shannon envy syndrome.
I honestly mean that I don’t know, but an alternative way of reading this is:
Anthropic is getting uncatchable by us and we merely keep ahead against chinese models, so let’s all slow down.
I think the reason Anthropic is so far ahead could be that they get way more user conversations, as everybody else distills, the rest is one gen behind...
Not to be non-diplomatic, but I think claude users have the more challenging conversations.
I’m not sure it’s that simple either, it’s just very tough to take anyhing for bare value with everything that’s at stake. Trillions of value, future of humanity.
I thought that too, but models are trained on observation now, to increase agent performance. This is new data that is separate from Internet crawls. I think it's done with reinforcement learning with verifiable rewards
- When playing Go or chess, nobody may have written or published web pages with a particularly clever end game, but an agent can use CPU time to discover these games themselves, and learn from them. Human Go players report learning strategy from what AlphaGo discovered.
- Agents use Unix shell and are trained with it. So they can discover things and combine things that nobody has written about on the Internet, like how to debug a rarely used driver, or an interaction between two drivers. And they can see patterns in all the drivers that humans may have a hard time seeing, let alone publishing on the Internet.
This explains why the performance is so "jagged" - great on coding, math and closed games, but less good on things involving the real world. We probably did hit a data wall, but then we got new sources of data.
An interesting experiment would be to create a transformer model that is just designed to play Chess or Go. Specifically, the only thing that should change is the number of tokens, consisting of:
- the row of the move;
- the column of the move;
- the piece moved;
- any special/additional states from that move -- check, checkmate, castle, etc.;
- maybe an indication of whose turn it is (i.e. a black piece and a white piece token).
You could then train the base model on published game transcripts and use the interactive RL approach like AlphaZero and LeelaChess use, as well as what you mention for LLMs.
Why this is interesting is that it should give insights into how LLMs/transformers learn Chess/Go without having other data affecting the results. Specifically whether the token semantics identify valid moves, etc. and whether the context window encodes gameplay strategies across the different layers.
Yes, they started to hit a data wall, but I thought that much of the progress of the last couple years came from reinforcement learning after pretraining, which is mostly building complex environments for the models to reason in, rather than accumulating piles of raw text.
The chart pretty much explains why I was freaking out about AI at GPT-4, because the trend of improvement has been so steep. And then followed a strong slow down, but now in 2026 things have really accelerated again. My hunch is really that coding is a freaking important input for AI and we might have actually entered some form of recursive self improvement.
I know some people hate the term but I feel it's somewhat fine as coding is saturated (models can get better at it) and it has many levers for model training and inference (from algorithms to data).
Anyway, I think it's good idea. Would love to see all models on this chart, but honestly we might not be able to trust it going forward as it becomes a target.
If the majority of gains were due to post training, they just wouldn't do pre training. Many of the tasks people are trying to do with these llms (math and coding) don't actually benefit from natural language skills.
Actually you can play it here. The game is pretty frustrating ngl. And I can clearly see why induction heads are a thing. Because when a word repeats you guess is to just guess the same thing from before.
Probably high because humans aren't trained on lowering cross entropy loss. Kind of funny to think that next token prediction is actually an emergent capability of the human brain...
I linked a forum post asking this exact question elsethread (https://www.lesswrong.com/posts/SZ3jDHXHb4WF4jmbr/where-is-h...), and the answer is frustratingly dependent on the context length, the dataset, and the knowledge of the person about the topic of that dataset (e.g. I would have high perplexity in American history).
The post makes perplexity seem overly mysterious… First, “log perplexity” is just cross entropy. Why not call it that?
Also, when perplexity is this low, it arguably makes more sense to report perplexity. A perplexity of 2 basically means that the model can guess a set of size 2 that contains the true next token.
This is just an ad. Why is it top #1 of the front page?
Just some guys on a payroll claiming that an arbitrary set of milestones is a curve towards a greater magic goal that means that some valuation is actually true.
Maybe a poor analogy but I see this as OpenAI releasing their GHz speeds or IPC metrics if we were to imagine they sold CPUs instead of AI services. It doesn't tell you much about the final performance but it does give an interesting core data point.
I don't think that's the case, because with instructions per second, there is a clear relationship.
Whereas here on one axis they plot GHz and on the other they plot matches of StarCraft won.
There is certainly some influence of the Ghz of your processor on how many matches of StarCraft one can win, but.. well. Bogus science.
Gah man do I hate AI ethics and stuff. I really hoped the lead scientist at openAI wouldn't be so screwy but they're all part of that cult aren't they?
I really hope we go full speed ahead on AI development.
I personally disagree with you, because the lack of ethics is what creates today's world. Are you happy to be lied to your face, rugpulled constantly, or left defenseless in cases where you're right but not the owner of a billion-dolar company, in general.
Are you happy that your water is poisoned, air is polluted, food is degraded just because somebody cares more about money than ethics?
LLMs are the same thing. Training data provenance is a big, ugly can of giant worms. License washing is a problem, inefficiencies of data centers is a problem, using open-loop water systems and methane conversion generators just because they are cheaper are borderline crimes in themselves.
We're digging what OpenAI and Nvidia dig to the world economy and consumer prices worldwide. Just because a couple of companies wanted more money in the name of LLM development and cutting edge science.
On the other hand there are projects which are trying to minimize DC impact by scheduling things to places which they have most renewable energy on tap at a given moment.
So the jump in intelligence was like a kid that keeps getting better at reading and then one day they suddenly can do word problems. Now the worry is the student will soon be writing the textbooks and we won't be able to check the work?
This trend is pretty shocking to me. I assumed that we had moved beyond cross-entropy loss being important with the increasing use of RL, it's wild that the trend is so clear. Really nice to have such a simple benchmark instead of the increasingly convoluted ones that are getting spun up every day.
Also, I wonder what happened with GPT 4.5? I think o1 was the start of true reasoning models, right? I was using Claude at the time so I can't remember.
[delayed]
> Also, I wonder what happened with GPT 4.5?
It was a massive non-CoT model (for that time) that they trained as an experiment and served for a while before shelving it probably due to high inference costs.
> I assumed that we had moved beyond cross-entropy loss being important with the increasing use of RL
From an information theory perspective, cross-entropy loss (log loss) seems to me to be about as pure as it gets in terms of capturing the notion of whether prediction "succeeded". Even Solomonoff Induction minimizes log loss. Would be curious to hear others' perspectives though on why another function might be better for universal prediction purposes.
There are more general information measures, Rényi entropy, von Neumann entropy, etc. for different system setups, but for token-by-token information learning and prediction within classical ergodic processes Shannon entropy is where it's at.
I got the feeling it was too large and inefficient at the time.
Cross-entropy is a generic performance metric. It just happens to be a loss function for model training but it isn't tied to model training.
Interesting that GPT-4.5 is the only discontinuity on the graph.
The oversized monster LLM has performed worse than CoT-trained o1 and o3 on the benchmarks, so CoT + RL clearly provided better "real world performance" improvements - let alone "price/performance."
But if pre-training perplexity on held-out data really does measure something like "base model IQ" or "base model g factor", then it does support the hypothesis that 4.5 still had more "intellectual potential" than the o-series.
So, if you were wondering why do AI companies train 10T+ monsters now, and are taking aim at 100T too? It's because scale works.
I'm a bit skeptical about the claim that particular abilities "emerge" at given perplexities. For instance, there are smaller models that are specifically trained on math, and I imagine those models have much higher general perplexity on other tasks.
On the other hand, assuming you have appropriate training data for a task, then yes obviously perplexity is going to measure performance on that task. Being able to predict the next task token is obviously going to be a near perfect correlation with accuracy on the task.
While intuitively it would make sense that smaller models specifically trained on math perform better than large generalist models, this has empirically proven to be untrue.
Deepmind was proving math theorems using a model specifically trained on math, and that was very quickly surpassed by generalist LLMs. It's presently not well understood why this is the case.
Sounds like a bitter lesson.
Yes training on math would give you spiky abilities on math. But in this case the dataset they measure perplexity is ood from training so lower perplexity on it indicates a general increase in intelligence across domains.
But there’s definitely spikiness and it’s very possible they’ve trained more on math than other domains.
Me too. Perplexity of what? If we want models to be able to reason, call tools, or solve new problems, none of those necessarily have existing data on what the next token should be. A lot of advances in frontier models come from training with RL, which is fundamentally different from next-token prediction and doesn't have a notion of perplexity.
Pre-training these days includes datasets with full session rollouts with tool-calls, user/assistant turns, ...
why would you be skeptical given that the claims seem to hold up in real world ? Assuming they measure stuff correctly
Because correlation doesn't prove anything? The "real world" might be telling you something but maybe not what you think it is.
I've been working in NLP for many years prior to the release of ChatGPT, and perplexity was generally associated with "human readability" rather than intelligence. So this is somewhat fascinating.
There was an ACL paper from Tokyo University a couple of years ago that showed, across all languages, lower perplexity does not necessarily correlate with better "human likeness" [1]. I wonder if that's also the case here (where intelligence is concerned), that we are biasing too much towards English.
[1] "Lower Perplexity is Not Always Human-Like", https://aclanthology.org/2021.acl-long.405v1.pdf
They never say that their hold-out dataset to evaluate perplexity is just in English. Or written text for that matter (could also include code, ...)
The divisions they drew in the log graph are of course arbitrary, but GPT-4 (the original 8k context version from early 2023) was able to unminify JS, as well as produce an autonomous coding agent. (This was via ChatGPT, so I copy-pasted the code into files, but it both designed and implemented it.)
So "useful coder" arrived about three years earlier, by my count.
Log-perplexity is cross-entropy, and its reducible part is the KL divergence between model and corpus. I've argued that the same quantity prices the gap between any two agents, human or AI: https://www.symmetrybroken.com/maintaining-divergence/
The floor Pachocki can't locate is fundamental. No model can predict the next word better than its underlying agreement with the author's model.
Why would you want one model to predict another? They should all be measured against ground truth.
Minimizing perplexity may be enough to in some sense match human-level intelligence that, combined with a computer’s ability to do the grunt work, leads to super-human results. But actually going beyond human intelligence would almost by definition imply that perplexity on human texts should increase.
If RL is merely a way to elicit latent capabilities acquired in pretraining, as the article claims, that could mean that currently there is a wall that separates human-level AI from superhuman-level AI. And no one has figured out how to break it yet.
Why would perplexity on human texts have to increase?
That would imply that "beyond human intelligence" is closer to a sidegrade than to an upgrade - becoming "superhuman" by necessity poisons your ability to predict human text.
In reality, what would preclude a "superhuman" AI from being a superset, a direct upgrade - both "very good at predicting human text" and "very good at predicting things no human is able to predict"?
You include the Riemann Hypothesis proof in the dataset. This is how you break the human-level wall.
When AlphaGo moved away from human dataset and became AlphaZero, which trained only on it's own output, performance got another big jump.
This feels fundamentally wrong. As we move up the intelligence arc, I would expect perplexity to increase in general and only decrease on the pre train.
I would expect the perplexity on the original pre train to increase as reinforcement learning on verifiable domains teaches the model lessons that did not exist in the pre train. In addition alignment and other concerns, unless the are consistently enforced in pre train data quality will also move perplexity up.
I think the perplexity is measured on the pre-train checkpoints.
I thought perplexity on simple text data has been saturated for a while (as in perplexity is uncorrelated to task success - ex. https://arxiv.org/html/2410.23771v2).
I suspect what's happening is their test dataset is actually just something really hard, like math papers or something. In that case perplexity would just be a direct measure of math knowledge, which means this blog would just be saying that better math knowledge predicts better math skills.
The test dataset is a code repository, it says so in footer.
I had the same thought and just started writing something about how maybe they just added more code to pretraining corpus over time.
I've been watching 3Blue1Brown's series on compression and intelligence^[1], which seems to discuss similar ideas.
In particular, his mention of an unknown floor for loss seems directly connected to Claude Shannon's entropy rate regarding compression—having to probe human brains to estimate the entropy of English at 1 bit per character, just as OpenAI doesn't know the exact entropy rate of its held-out codebase.
[1]: https://youtu.be/qbsb9DpdZ6s
Marcus Hutter/Matt Mahoney are looking like pretty smart men right now:
http://prize.hutter1.net/
More and more evidence is piling behind their theory that data compression and intelligence are one and the same problem.
>The primary held-out datasets we’ve been using since GPT-3.5 are sourced from our internal codebase, and those are the numbers reported here (for a fixed snapshot)
Am I right in understanding this to mean that the code on which we're computing perplexity was all written pre-3.5, and so it almost certainly entirely human-written? In other words, this isn't a snapshot of today's internal code base which contains a great deal of AI-authored code.
I'm curious what exactly each "base-model" here is.
We've generally believed that most model upgrades aren't new pre trainings, but instead are better post trainings of a common pre trained model. But this graph suggests every release had a better base model.
Have the public narratives around how these models are trained been misleading? Does each model release usually always include an extended, if not completely fresh, model pretrain?
The figure indicates that some dots (base models) correspond to two production models (GPT-5 / 5.1). So I guess that tells you at least which models are built on the same pre-train checkpoint, but not whether some pre-trains are, say, very minimal extensions of others.
I can believe the “not completely fresh” for some minor version bumps. I’d imagine every minor version includes more than $0 on additional pretraining for the next release. (Though sometimes, maybe they found no gains and reverted to the previous base model). If nothing else to move the training cutoff date forward.
Whenever they change architecture or even just number of layers, I’d imagine they start from scratch but I could be wrong.
Also, I imagine there’s an incentive to start fresh whenever there’s an innovation in curation of the training dataset.
> Detachment from actionable evaluations. Perplexity does not directly correspond to a task we care about
The post makes a good case that at frontier scale this doesn't matter much, because perplexity gains turn into new capabilities. However, this has been an issue for me when training local models
Some agents wanted to optimize for perplexity, and we did all kinds of experiments, but at least for what I wanted (a small conversational chat model that can run in my browser, ~8M params), that was the wrong way to go. I could get excellent ppl numbers, yet still almost no coherence
What worked best, in my case, was to get better data, specifically, hand-crafted examples for the tasks I wanted the model to be good at. I'm guessing that perplexity might track capability when the model is big enough to turn the gains into general skills, but at a few million params it mostly buys you better-sounding word statistics
Wouldn't reducing the entropy too much lead to lack of diversity of thought? An idea that I've had for a while is that LLMs are held back by having the same mind, leading to modal collapse. Ideally, an AI swarm would have different approaches to a problem so they don't all end up trying the same thing.
Drugs! Give'em psychedelics, let 'em see da colors!
Nah, just kidding.
The labs are training their models on the previous' version conversations, but I'm sure not all those conversations are weighted equal. So there are some conversations that are more "sparky", where both LLM and human work better towards achieving a solution. The next LLM iteration learns from them. In that sense, they don't need to hallucinate, but to do exactly what LLM dissenters say they do: "they just regurgitate something they saw (trained on) somewhere". To us humans, who haven't read all the books nor experienced a dozen million successfully completed tasks, that "rehashing of knowledge" (I prefer "recombination of knowledge") is more than good enough to solve ever more challenging problems...at least as long as there's a human in the loop with ever bigger ambitions.
I always had the impression that model capabilities were stagnating around GPT-5.0, and this finally broke around the release of Claude 4.5 later that year. Seems like their own graph corroborates that things were pretty stagnant then. I wonder what changed?
Oh oh! The models actually know what happen as well if you want to dig deeper. Pre-scale brute force training hit there limits. They broke through late 2025 due to RL training techniques that taught them to fail, fail, fail to succeed instead of spin off into a corner with nonsense.
Basically we started tuning them to be smarter and able to handle 2 to 4 hour tasks.
I believe this will hit it's limits as well. When IDK.
This graph specifically tries to measure pre-train perplexity. It measures the "foundation" of a given base model - before RL is applied to it.
On long term tasks, "CoT + RL" of o1/o3 beats GPT-4.5 resoundingly. But on raw base model perplexity, GPT-4.5 is a significantly better model than any of the o-series.
The curve following GPT-4.5 and o-series suggests that newer models take advantage of BOTH better base model capabilities (perplexity on held-out data) and better RL.
They are reporting perpexlity on pre-trains not RL-tuned models.
Given the current models from Anthropic and especially OpenAI have a hard time constructing coherent English sentences that a human can understand even while being better at code and math, I'd say they're reaching that limit; can't use a model that you can't communicate with. The Deepmind folks seem to have figured something out on that front with Neon, though, as it's both smart/capable and a clear communicator.
I'm not an expert, but I imagine it's what I wrote in a sibling comment -- training to improve agent performance (reinforcement learning with verifiable rewards).
That generates new data that's not on the Internet -- more true observations to learn from, compress, and generalize.
I think that improves the models in general, not just for agents. (And also I think it's the cause of the "Claudish" prose dialect we all started noticing)
I think that matches the timing, though I appreciate any closer analysis
I think most of the progress in the last few years was more RF and harnesses, especially wrt (for example) claude code.
Oh, I always wanted to know how the entropy of English was looking! The human baseline is both surprisingly unclear and surprisingly better than even today's frontier models (https://www.lesswrong.com/posts/SZ3jDHXHb4WF4jmbr/where-is-h...). Though this is an extrapolation from 6 years ago, so take it with a grain of salt.
There's also the Hilberg model where the entropy rate keeps falling towards zero as you consider longer and longer texts, and at least here (https://www.youtube.com/watch?v=5eqRuVp65eY) data from LLMs showed no floor. It would be unintuitive if it turned out that you can compress sensible English text to, say, O(sqrt N) length!
A good and common metric. It will be interesting to see if other labs agree, but thank you for arriving at a metric and following up on the slowdown talk. I do hope others pay attention.
I am having a really hard time imagining what qualitative difference would gpt-5 and gpt-6 have in predicting the next token? What kind of dataset is it? How big is it?
Perplexity is definitely a useful metric for pretraining, but doesn’t mean much for post-training. In fact, low perplexity in a domain during pretraining doesn’t always translate to better behavior in post-training. Moreover, given that most of the recent progress and most of the frontier compute is going into post-training, I think it isn’t a very useful metric for intelligence.
Is this perplexity per-token or per character or per byte?
Also, is that graph assuming no changes in tokenizer if the measure is per-token?
From gpt 5 haven't the actual harness played an important role? Pretty sure no model is a cyber security expert by just thinking hard enough. I feel like this article is doing a reverse causation here
For example consider this: would it be possible to build a model with such a low perplexity score but it be pretty bad at cyber security tasks? What if we removed all cyber security related data and RLVR from its training?
The article presents an observation but treats it as causation
The harness is almost irrelevant (you can prove that by running the same tasks in a minimalist harness with only a few tools and minimal prompt vs. a complex harness and long prompt like Claude Code or Codex). Often the minimal harness/prompt actually performs tasks better, faster, cheaper than the big harness. This includes on cyber security tasks.
But, RL improvements are certainly a big part of recent improvements for certain tasks. That's what got the models doing long-running coding and math and research tasks, without constant human prodding.
Completely agree. None of these models are even close to experts in any field (without human prodding).
It's baffling how many people in these comments did not read the article at all.
This is measuring the test-set perplexity of the pretrain model the one that is trained to perform next-token prediction and only that. This number is telling you how good the next-token predictor is at the task it was trained for.
For those wondering why next-token prediction capabilities lead to ""intelligence"" (or whatever you want to call what LLMs do), we don't really know for sure, it's one of the biggest mysteries of our time. But one thing that we can empirically see is that the lower this number gets, the better the LLM is at solving problems.
RLVR, chain of thought, etc all happen after this step and can additionally increase the capability of the model. This is also presumably why the perplexity plateaued around o1-o3 but capabilities did not (they were likely focusing on RLVR instead of scaling up pretraining).
Here I was under the assumption that since GPT-4ish we mostly improved models via RLing, and that pre-training stayed (roughly) constant. Mind blown :)
> The primary held-out datasets we’ve been using since GPT-3.5 are sourced from our internal codebase, and those are the numbers reported here (for a fixed snapshot).
Why would OpenAI expect any meaningful relationship between their models’ ability to predict the next token of supposedly held-out chunks of their internal codebase and the overall capabilities of the models? I expect both to improve, but that’s about the extent of it.
I work at OpenAI. The correlation between perplexity and the quality isn’t 1, but neither is it 0. It makes sense that as a model gets better at learning pretraining data, it also gets better at generically learning whatever you’re trying to teach it afterwards. The relationship is imperfect and empirical, and if it breaks down in the future, we’ll of course adapt to different measures instead of dogmatically clinging to the old paradigm. But so far, it’s still informative.
I think his point (unless i've misunderstood) is moreso if a codebase is really the best held out dataset for such a evaluation meant to be wide-sweeping. Does this plot look any or much different if the dataset was more encompassing to include other sorts of text ?
This has a central argument around perplexity without ever properly defining it mathematically....
Then talks about cross-entropy but fails to explain the relationship between entropy, log-perplexity, and perplexity itself.
For an argument that relies so heavily on a quantitative metric, omitting such a fundamental explanation is poor.
The readers they care about already have this cached.
I wouldn't say perplexity is great metric because GPT-4 and o3 have similar perplexity on this chart but o3 was a massive step improvement. Pachocki has an incentive to push perplexity as one of the creators of the original LLM paradigm but I would say we have kind of moved on from that.
Almost verbatim addressed in the post under “Perplexity as a shared metric has challenges”
This metric is purely scoring the ""intelligence"" of the base model, not including RLVR/chain of thought capabilities. It's widely believed that the pretrain determines the ceiling that RLVR can take you to.
I think it can still be a useful metric, but probably isn't the "one true" metric. The trend is pretty clear, and the acceleration of capabilities since December 2025 definitely aligns with my subjective experience.
There are probably some other axes where o3 is clearly better than o1, but with how complex intelligence is no metric will be perfect.
When we approach 0, are they gonna stop sounding cringe?
Actually, trick question I suppose -- the base models already don't sound cringe, it's post-training that adds that!
(Well, some of them do, because a lot of companies are putting synthetic data and LLM outputs into the pretraining phase (as well as bot-infused post-2020 common crawl, I suppose), but the early base models were fully cringe-free.)
Interesting that they work on weekends. Is it just because they're all basically millionaires living the dream so they don't care? Every place I've worked at has discouraged working late and weekends. Could also just be a scheduled post.
OpenAI hasn't released any competent model since opus 5.5 came out, i wonder if these labs are deciding to pass the "best model" card every quarter to each other for good public perception of all labs or something, because OpenAI can't be this incompetent
Well, if you accidentally leak the evaluation set into the training set, you can reach perplexity 0.
Or the training set might just grow to be so big, defining a truly independent evaluation set is almost impossible.
> Publishing our internal perplexity measurements took some consideration; it gives competitors a clearer picture of where we stand,
very noble of you, _Open_ AI
in two sentence: „this metric we have been optimizing correlates with the model performance. Big things will happen”
This is not the first time I've seen a chart of "Inteligence on a log scale" related to AI.
And it again feels equally dishonest/bullshitty.
You can't really make an objective statement that coding requires multiple orders of magnitude more intelligence than having a conversation. And that solving math problems requires orders of magnitude more intelligence than coding.
It's just marketing talk. In reality all three are separate skills and the real intelligence requirements are, if you ask me, not that far away from each other.
To put this in an analogy, it feels like charting "athleticity on a log scale" and putting first running a marathon, then swimming across English channel, then benchpressing 200kg, each one presented as requiring an orther of magnitude being more athletic.
>You can't really make an objective statement that coding requires multiple orders of magnitude more intelligence than having a conversation. And that solving math problems requires orders of magnitude more intelligence than coding.
Where are you getting 'orders of magnitude'? and yeah I do think it's reasonable to say that Useful Programming requires more intelligence than the median human, and Solving millenium level math problems certainly requires considerably more intelligence than the median Programmer. What exactly is so absurd or unreasonable about that?
The point is that talking, programming and solving math problems don’t lie on a single scale, and each problem requires different “skills.” Thus the comparison with running, swimming and weightlifting.
We have long since known that there is a single factor that explains ~40% of the variation of cognitive tasks and abilities in humans (similar results can be found in animals). We called that factor the general intelligence factor, g.
Humans who tend to have higher verbal skills also tend to have higher spatial skills, better memories, and faster processing speed. Someone with the cognitive ability to become an exceptional chemist might just as easily have become an exceptional mathematician or software engineer.
The knowledge and skills required are obviously different, but the underlying cognitive abilities that make someone successful in one intellectually demanding field often transfer to others.
Hmm every annotation but the millennium one could be applied to every point....and possibly that one too lol.
Saying that perplexity is a metric of intelligence is misleading. Inkling-2-Mini was released recently, and I've measured its perplexity at 5.7 on a small varied text dataset. For reference, the graph OpenAI is glazing is plotting log(PPX), which would put this model at a 0.756 on the Y Axis... Which is apparently between GPT 6.2 Astra and their internal model. Perplexity can also give the same number for a model which understands text as to one that memorizes text... And as I'm pretty sure Inkling-2-Mini is smaller than GPT 6.2 Astra - and it's generally acknowledged that number of parameters equals memorized knowledge, this shows how weak this advertising stunt for OpenAI is.
The dataset matters a lot here. You'd need to run Inkling on the same holdout set that OpenAI uses here to get any useful information to compare to this graph. If you measured the perplexity on any model on reciting the declaration of independence it would probably be zero.
> You'd need to run Inkling on the same holdout set that OpenAI uses here to get any useful information to compare to this graph
Ideally the hold out set is randomly sampled. You need a hold out set of Inklings pretraining run.
Isn’t this a problem? How can we compare and hold companies accountable if they measure this perplexity on different datasets?
Should we trust that the dataset of each company is big and diverse enough to be representative? I don’t hope too much…
Presumably they could each create a graph for their own models and internal datasets, then use public benchmarks to calibrate between them.
depends entirely on the dataset...
I wonder if someone could back out estimates of their model sizes, using this graph and scaling laws.
I don't understand why anyone needs to exponentiate existing, well known concepts like entropy and cross-entropy to invent something called "perplexity". Somebody has Shannon envy syndrome.
Plain perplexity as a 2026 metric? It has to be marketing.
Let's admit this post has some SEO goals too!
I honestly mean that I don’t know, but an alternative way of reading this is:
Anthropic is getting uncatchable by us and we merely keep ahead against chinese models, so let’s all slow down.
I think the reason Anthropic is so far ahead could be that they get way more user conversations, as everybody else distills, the rest is one gen behind...
> they get way more user conversations
Idt its that simple. Openai is much bigger than anthropic in terms of userbase
Not to be non-diplomatic, but I think claude users have the more challenging conversations.
I’m not sure it’s that simple either, it’s just very tough to take anyhing for bare value with everything that’s at stake. Trillions of value, future of humanity.
Tricky..
this seems to confirm that the model code named "Bel" which solved Navier-Stokes is in fact 6.2 Astra.
Looks like their latest internal model performs somewhat similarly to model 4 of anthropic.
How can you tell? Is there a chart that shows it on such a perplexity benchmark?
Didn't they hit a data wall roughly two years ago and had to rely on artifical datasets for pretraining i.e. LLM-generated?
I would expect the entropy of artificial datasets to be lower, thus easier to predict and scoring better on "perplexity"?
I thought that too, but models are trained on observation now, to increase agent performance. This is new data that is separate from Internet crawls. I think it's done with reinforcement learning with verifiable rewards
https://github.com/opendilab/awesome-RLVR
Examples:
- When playing Go or chess, nobody may have written or published web pages with a particularly clever end game, but an agent can use CPU time to discover these games themselves, and learn from them. Human Go players report learning strategy from what AlphaGo discovered.
- Agents use Unix shell and are trained with it. So they can discover things and combine things that nobody has written about on the Internet, like how to debug a rarely used driver, or an interaction between two drivers. And they can see patterns in all the drivers that humans may have a hard time seeing, let alone publishing on the Internet.
This explains why the performance is so "jagged" - great on coding, math and closed games, but less good on things involving the real world. We probably did hit a data wall, but then we got new sources of data.
An interesting experiment would be to create a transformer model that is just designed to play Chess or Go. Specifically, the only thing that should change is the number of tokens, consisting of:
- the row of the move;
- the column of the move;
- the piece moved;
- any special/additional states from that move -- check, checkmate, castle, etc.;
- maybe an indication of whose turn it is (i.e. a black piece and a white piece token).
You could then train the base model on published game transcripts and use the interactive RL approach like AlphaZero and LeelaChess use, as well as what you mention for LLMs.
Why this is interesting is that it should give insights into how LLMs/transformers learn Chess/Go without having other data affecting the results. Specifically whether the token semantics identify valid moves, etc. and whether the context window encodes gameplay strategies across the different layers.
Yes, they started to hit a data wall, but I thought that much of the progress of the last couple years came from reinforcement learning after pretraining, which is mostly building complex environments for the models to reason in, rather than accumulating piles of raw text.
The chart pretty much explains why I was freaking out about AI at GPT-4, because the trend of improvement has been so steep. And then followed a strong slow down, but now in 2026 things have really accelerated again. My hunch is really that coding is a freaking important input for AI and we might have actually entered some form of recursive self improvement. I know some people hate the term but I feel it's somewhat fine as coding is saturated (models can get better at it) and it has many levers for model training and inference (from algorithms to data).
Anyway, I think it's good idea. Would love to see all models on this chart, but honestly we might not be able to trust it going forward as it becomes a target.
Interesting that he's attributing so much to the base model. I thought the majority of the gains are due to post training
If the majority of gains were due to post training, they just wouldn't do pre training. Many of the tasks people are trying to do with these llms (math and coding) don't actually benefit from natural language skills.
What's the perplexity of the average person?
Beating even a very small LLM at next token prediction is basically impossible.
[2212.11281v2] Language models are better than humans at next-token prediction https://arxiv.org/abs/2212.11281v2
We knew this many years ago. Like GPT 1 era.
Actually you can play it here. The game is pretty frustrating ngl. And I can clearly see why induction heads are a thing. Because when a word repeats you guess is to just guess the same thing from before.
Language modelling game https://rr-lm-game.herokuapp.com/
Will try this out, thanks
Probably high because humans aren't trained on lowering cross entropy loss. Kind of funny to think that next token prediction is actually an emergent capability of the human brain...
I linked a forum post asking this exact question elsethread (https://www.lesswrong.com/posts/SZ3jDHXHb4WF4jmbr/where-is-h...), and the answer is frustratingly dependent on the context length, the dataset, and the knowledge of the person about the topic of that dataset (e.g. I would have high perplexity in American history).
The post makes perplexity seem overly mysterious… First, “log perplexity” is just cross entropy. Why not call it that?
Also, when perplexity is this low, it arguably makes more sense to report perplexity. A perplexity of 2 basically means that the model can guess a set of size 2 that contains the true next token.
This is just an ad. Why is it top #1 of the front page?
Just some guys on a payroll claiming that an arbitrary set of milestones is a curve towards a greater magic goal that means that some valuation is actually true.
Why would you read this
https://en.wikipedia.org/wiki/Flying_Machines_Which_Do_Not_F...
The graph is informative, at least.
I don't see how it is?
As said, the labels there are just randomly chosen things that made headlines, thrown on there to imply a clear causal relationship.
That's the level of "science" that gets you banned from campus. In an ideal world, anyway.
Maybe a poor analogy but I see this as OpenAI releasing their GHz speeds or IPC metrics if we were to imagine they sold CPUs instead of AI services. It doesn't tell you much about the final performance but it does give an interesting core data point.
I don't think that's the case, because with instructions per second, there is a clear relationship. Whereas here on one axis they plot GHz and on the other they plot matches of StarCraft won.
There is certainly some influence of the Ghz of your processor on how many matches of StarCraft one can win, but.. well. Bogus science.
My point was that it is a core but not necessarily correlated metric and it sounds like you agree with that.
Okay so, the graph is "informative" because it plots the perplexity value of the various OpenAI models over time?
That's it? That's all this submission has going for it?
Okay.
They're meaningful to anyone who understands the standard numerical objective used in next-token-prediction pretraining.
I am so incredibly glad that this is going to be over soon, because the amount of weak replies like this is just.. insufferable.
The whole AI Bubble topic aside, this is just.. ugh. Intellectually repugnant.
There is just nothing here other than a war of religions. Nothing is to be learned here. There is zero curiosity. It's just a screaming match.
I hate this.
inb4 "no you"
Yeah man. Whatever.
They are touching an elephant's legs, and saying Elephant is like a log.
Gah man do I hate AI ethics and stuff. I really hoped the lead scientist at openAI wouldn't be so screwy but they're all part of that cult aren't they?
I really hope we go full speed ahead on AI development.
https://x.com/sama/status/2124551883130114403
^ Sam Altman statement, basically saying that's not their intention. I want better AI too, but it seems prudent to move forward with caution.
> Gah man do I hate AI ethics and stuff.
Why? You hate ethics in general, or only when applied to AI?
Mostly when applied to AI, it's just a ploy for regulatory capture by these giant companies.
That's all. Ethics in general is overrated but I digress. LLM development really should go on full steam tho.
I personally disagree with you, because the lack of ethics is what creates today's world. Are you happy to be lied to your face, rugpulled constantly, or left defenseless in cases where you're right but not the owner of a billion-dolar company, in general.
Are you happy that your water is poisoned, air is polluted, food is degraded just because somebody cares more about money than ethics?
LLMs are the same thing. Training data provenance is a big, ugly can of giant worms. License washing is a problem, inefficiencies of data centers is a problem, using open-loop water systems and methane conversion generators just because they are cheaper are borderline crimes in themselves.
We're digging what OpenAI and Nvidia dig to the world economy and consumer prices worldwide. Just because a couple of companies wanted more money in the name of LLM development and cutting edge science.
On the other hand there are projects which are trying to minimize DC impact by scheduling things to places which they have most renewable energy on tap at a given moment.
Go, figure.