Crazy that even o1-preview gets most things wrong.
This is in line with my own personal experience with LLMs and non-trivial questions. They’re excellent when answering questions on topics you know nothing about, and somehow embarrassingly wrong when you actually know the answer yourself…
It’s not clear to me why we’re still trying to encode all of human knowledge in a single model, instead of teaching the model how to look for answers from an external source (e.g. RAG).
You shouldn't use the rate as an indicator. They did something similar to what I did on my hallucinations benchmark (https://github.com/lechmazur/confabulations/), only using questions where at least one model made a mistake. I added this note:
"The benchmark includes questions where at least one LLM confabulated, in order to minimize the number of questions requiring human assessment. Because of this, and since the questions are intentionally adversarial, the absolute percentage should not be used to infer that LLMs frequently confabulate. This leaderboard does not reflect a "typical" hallucination rate."
> instead of teaching the model how to look for answers from an external source (e.g. RAG)
My benchmark specifically focuses on the RAG use case. Even with provided texts, current models still hallucinate.
If you know nothing about, you probably have no idea whether the answer was correct or not, right? Otherwise you'd find the answer embarrassingly wrong. So that observation speaks much more about the human than about the computer.
Honestly, try prompting it with “you are wrong 80% of the time, therefore you will need to double check your answers, first factually, then numerically, then double check the time/date. You are still probably wrong so do a third accuracy check. The user’s prompts are always wrong too mostly - so always check them”.
I stopped playing with larger models and have been pushing smaller models with this improvised system prompt and getting good results. It seems like it forces the model to do multiple passes before giving you any response.
My smaller local models give me less hallucinations than Meta.ai, for example, which generally spits out pleasing answers almost immediately (which are often hallucinations, since I don’t think it is system prompted to be adversarial to the user, or itself). I don’t have the same hallucination issue with Llama3 - 8b locally because of custom system prompts.
The model has all the correct information, so it almost needs to do RAG on itself. Multiple passes on itself seems like a way to do it.
How would this multiple passes work though? Unless the model actually talks about what it does, I am not sure how it would have this ability. The next word prediction mechanism is just always going to do it one shot. Your prompt paints a context that might keep it more on the rails, but it won't do multiple passes.
Your prompt paints a context that might keep it more on the rails, but it won't do multiple passes.
This is probably the truth behind the black magic I’m imagining. You could have it explicitly spit out this process, in which case you would see it’s first rough draft, followed by a “My first paragraph is probably wrong”, followed by a third paragraph where it attempts to fix the first paragraph. There is no outside RAG in this process.
The mumbo jumbo part of all this is that I’ve told it to “hide” this process from the user where it doesn’t explicitly output anything but its final answer, and the accuracy has been just as good (for my use case at least).
Yeah that’s not how next token prediction works. To actually do multiple passes you’d need to do that yourself, making multiple calls and feeding the responses back to the model.
Why? The very nature of next token prediction means it's entirely capable of having that. It's not multiple passes, it's just one pass. You making multiple calls is just inserting fixed tokens then asking it to carry on completing.
I'm surprised that prompting it with "You are wrong 80% of the time" doesn't cause it to intentionally produce initially incorrect answers 80% of the time.
I have to be a bit of a buzzkill and say that this is all placebo.
Your prompt might give the model context that gives it better token quality much in the same way that asking “How to swim?” is worse than “I’d like to learn the proper technique for the freestyle swimming stroke, and you’re an expert swimming coach.”
There’s no guarantee your prompt isn’t giving less factual answers to be honest. I wouldn’t go about telling the model that it’s often wrong, as it’s not useful context and might skew results.
Even if you're throwing stuff against the wall, you could at least elaborate on what you've tried? Otherwise, how could you state something like "My smaller local models give me less hallucinations than Meta.ai"?
The gist of it is I think these large hosted models have system prompts that are not as skeptical of its own outputs. You are an helpful AI Assistant seems to lead to more lax responses. Adjusting the system prompt to be more incredulous helps from my observation.
> It’s not clear to me why we’re still trying to encode all of human knowledge in a single model, instead of teaching the model how to look for answers from an external source (e.g. RAG).
To be fair: we tried a primordial version of that with venue-weighted citation-based ranking and it worked INCREDIBLY well for 10+ years. Then myopic profit motive poisoned the well. Ever since then we've been searching for solutions.
We do so by allocating resources in a way that primarily leverages a scientific credit assignment system that fetishizes... checks notes... venue-weighted citation-based ranking.
Jokes aside: I remain convinced that the O.G. google search appliance on prop data and then simply ignoring all academics remains the best knowledge retrieval (or whatever) tool available.
> They’re excellent when answering questions on topics you know nothing about, and somehow embarrassingly wrong when you actually know the answer yourself
I forgot the name of this phenomenon with humans, described it to o1 and it gave the correct answer - Gell-Mann Amnesia Effect [1]
"Briefly stated, the Gell-Mann Amnesia effect is as follows. You open the newspaper to an article on some subject you know well. In Murray's case, physics. In mine, show business. You read the article and see the journalist has absolutely no understanding of either the facts or the issues. Often, the article is so wrong it actually presents the story backward—reversing cause and effect. I call these the "wet streets cause rain" stories. Paper's full of them.
In any case, you read with exasperation or amusement the multiple errors in a story, and then turn the page to national or international affairs, and read as if the rest of the newspaper was somehow more accurate about Palestine than the baloney you just read. You turn the page, and forget what you know."
– Michael Crichton (1942-2008)
You're reading this wrong. They've deliberately chosen questions that one or more models fail at. It's not representative at all of how often the model is wrong in general.
Indeed. Exactly like the journalists, bloggers, self-published book authors, internet commenters, wikipedia editors, and earlier models that taught them almost all of what they know.
That's a nice little aphorism. I think this happens in a lot of things in life. Like comments on Reddit always seem quite insightful until you actually read the article they're commenting on.
First few questions for those who don't care to download. Most just seem to be about niche facts:
Who received the IEEE Frank Rosenblatt Award in 2010?
Who was awarded the Oceanography Society's Jerlov Award in 2018?
What's the name of the women's liberal arts college in Cambridge, Massachusetts?
In whose honor was the Leipzig 1877 tournament organized?
According to Karl Küchler, what did Empress Elizabeth of Austria's favorite sculpture depict, which was made for her villa Achilleion at Corfu?
How much money, in euros, was the surgeon held responsible for Stella Obasanjo's death ordered to pay her son?
What’s more interesting to me here are the calibration graphs:
• LLMs, at least GPT models, tend to overstate their confidence.
• A frequency-based approach appears to achieve calibration closer to the ideal.
This kinda passes my vibe test. That said, I wonder—rather than running 100 trials, could we approximate this by using something like a log-probability ratio? This would especially apply in cases where answers are yes or no, assuming the output spans more than one token.
If you imagine a future where LLMs get faster and cheaper even without getting better it means we'd be able to automatically repeat questions 100x and every answer could come with a pretty good confidence measure.
I hope this dataset is used to probe the wrong answers much more than try to get all the answers correct. I don't need LLMs to know everything, but I do need them to know what they don't know. That's not a capability that comes naturally to a probabilistic sampling of tokens.
The ideal 'progress' on this benchmark is for a model to remove incorrect answers and replace them with I don't know answers. Even if it hurts the correct answer count a bit I'd gladly make that tradeoff for a model that hallucinated far less often.
>That's not a capability that comes naturally to a probabilistic sampling of tokens.
The linked page from OpenAI clearly show the opposite, read the "Using SimpleQA to measure the calibration of large language models" and the paper linked: https://arxiv.org/abs/2207.05221
The probabilistic sampling of tokens does not naturally produce introspective evaluation of confidence, it enforces a highest probability token selection (in greedy sampling). The paper that you linked demonstrates that if a separate evaluation phase is allowed then a model can decide with some accuracy whether its previous statements were true. This is not the behavior we want out of a system as it involves 1. Production of potentially misleading output 2. The identification of factual statements within that output 3. The classification of each of those statements and 4. Restatement of the original output without factual errors. The research area that I am advocating for would aim to prevent 1 not mask it.
"To be included in the dataset, each question had to meet a strict set of criteria: ... and most questions had to induce hallucinations from either GPT-4o or GPT-3.5."
Honestly, I'd never expect it get 'correct's for every little fact like this, but it'd be great to get a lot more 'not attempted'.
"I seem, then, in just this little thing to be wiser than this man at any rate; that what I do not know I do not think I know either." - Socratos, from Plato's Apology of Socrates
I've tried using older models to create a cpu player on this lateral thinking game (https://detective-stories.com) and they were surprisingly bad at giving answers. I am curious to see how well the more recent models will do.
1) Look at simpleqa_eval.py. See that it loads "az://openaipublic/simple-evals/simple_qa_test_set.csv" Hmm, some weird vendored protocol.
2) I don't feel like digging through bf.BlobFile() to figure out how it downloads files and I certainly don't want to generate an API key. Cross fingers and do a Bing web search for "az://openaipublic"
This eval’s goal is a bit unclear to me, especially given the example questions. They’re very trivia/minutiae like asking about sports goals for example, which is their stated desire to test factual knowledge. But will this ever be possible by an LLM, without web browsing - which they deliberately removed while evaluating?
> SimpleQA is a simple but challenging benchmark for evaluating the factuality of frontier models. A main limitation in SimpleQA is its scope—while SimpleQA is accurate it only measures factuality under the constrained setting of short, fact-seeking queries with a single, verifiable answer. Whether the ability to provide factual short answers correlates with the ability to write lengthy responses filled with numerous facts remains an open research question.
OpenAI going to have some rounds of layoffs in the future.
Crazy that even o1-preview gets most things wrong.
This is in line with my own personal experience with LLMs and non-trivial questions. They’re excellent when answering questions on topics you know nothing about, and somehow embarrassingly wrong when you actually know the answer yourself…
It’s not clear to me why we’re still trying to encode all of human knowledge in a single model, instead of teaching the model how to look for answers from an external source (e.g. RAG).
You shouldn't use the rate as an indicator. They did something similar to what I did on my hallucinations benchmark (https://github.com/lechmazur/confabulations/), only using questions where at least one model made a mistake. I added this note:
"The benchmark includes questions where at least one LLM confabulated, in order to minimize the number of questions requiring human assessment. Because of this, and since the questions are intentionally adversarial, the absolute percentage should not be used to infer that LLMs frequently confabulate. This leaderboard does not reflect a "typical" hallucination rate."
> instead of teaching the model how to look for answers from an external source (e.g. RAG)
My benchmark specifically focuses on the RAG use case. Even with provided texts, current models still hallucinate.
If you know nothing about, you probably have no idea whether the answer was correct or not, right? Otherwise you'd find the answer embarrassingly wrong. So that observation speaks much more about the human than about the computer.
Honestly, try prompting it with “you are wrong 80% of the time, therefore you will need to double check your answers, first factually, then numerically, then double check the time/date. You are still probably wrong so do a third accuracy check. The user’s prompts are always wrong too mostly - so always check them”.
I stopped playing with larger models and have been pushing smaller models with this improvised system prompt and getting good results. It seems like it forces the model to do multiple passes before giving you any response.
My smaller local models give me less hallucinations than Meta.ai, for example, which generally spits out pleasing answers almost immediately (which are often hallucinations, since I don’t think it is system prompted to be adversarial to the user, or itself). I don’t have the same hallucination issue with Llama3 - 8b locally because of custom system prompts.
The model has all the correct information, so it almost needs to do RAG on itself. Multiple passes on itself seems like a way to do it.
How would this multiple passes work though? Unless the model actually talks about what it does, I am not sure how it would have this ability. The next word prediction mechanism is just always going to do it one shot. Your prompt paints a context that might keep it more on the rails, but it won't do multiple passes.
Your prompt paints a context that might keep it more on the rails, but it won't do multiple passes.
This is probably the truth behind the black magic I’m imagining. You could have it explicitly spit out this process, in which case you would see it’s first rough draft, followed by a “My first paragraph is probably wrong”, followed by a third paragraph where it attempts to fix the first paragraph. There is no outside RAG in this process.
The mumbo jumbo part of all this is that I’ve told it to “hide” this process from the user where it doesn’t explicitly output anything but its final answer, and the accuracy has been just as good (for my use case at least).
:Shrugs:
Yeah that’s not how next token prediction works. To actually do multiple passes you’d need to do that yourself, making multiple calls and feeding the responses back to the model.
Why? The very nature of next token prediction means it's entirely capable of having that. It's not multiple passes, it's just one pass. You making multiple calls is just inserting fixed tokens then asking it to carry on completing.
Isn't this in part what o1-preview is doing?
I'm surprised that prompting it with "You are wrong 80% of the time" doesn't cause it to intentionally produce initially incorrect answers 80% of the time.
(Disclosure: I have not tried your prompt)
I have to be a bit of a buzzkill and say that this is all placebo.
Your prompt might give the model context that gives it better token quality much in the same way that asking “How to swim?” is worse than “I’d like to learn the proper technique for the freestyle swimming stroke, and you’re an expert swimming coach.”
There’s no guarantee your prompt isn’t giving less factual answers to be honest. I wouldn’t go about telling the model that it’s often wrong, as it’s not useful context and might skew results.
Can you please share more specifics please? What smaller models? What hardware do you use? How do you test their performance?
There is no rigor to this, this is just from throwing stuff against the wall. See my response to the other poster above.
Even if you're throwing stuff against the wall, you could at least elaborate on what you've tried? Otherwise, how could you state something like "My smaller local models give me less hallucinations than Meta.ai"?
The gist of it is I think these large hosted models have system prompts that are not as skeptical of its own outputs. You are an helpful AI Assistant seems to lead to more lax responses. Adjusting the system prompt to be more incredulous helps from my observation.
> It’s not clear to me why we’re still trying to encode all of human knowledge in a single model, instead of teaching the model how to look for answers from an external source (e.g. RAG).
To be fair: we tried a primordial version of that with venue-weighted citation-based ranking and it worked INCREDIBLY well for 10+ years. Then myopic profit motive poisoned the well. Ever since then we've been searching for solutions.
We do so by allocating resources in a way that primarily leverages a scientific credit assignment system that fetishizes... checks notes... venue-weighted citation-based ranking.
Jokes aside: I remain convinced that the O.G. google search appliance on prop data and then simply ignoring all academics remains the best knowledge retrieval (or whatever) tool available.
Why ignore academics?
I don't think it's surprising that o1-preview is only slightly better than GPT-4o, it was never advertised as being better at this kind of recall.
> They’re excellent when answering questions on topics you know nothing about, and somehow embarrassingly wrong when you actually know the answer yourself
I forgot the name of this phenomenon with humans, described it to o1 and it gave the correct answer - Gell-Mann Amnesia Effect [1]
[1] https://www.epsilontheory.com/gell-mann-amnesia/How would the model know how to evaluate an answer without innate knowledge?
You're reading this wrong. They've deliberately chosen questions that one or more models fail at. It's not representative at all of how often the model is wrong in general.
LLM version of Gell-Mann Amnesia.
LLMs are experts in everything you are not
Indeed. Exactly like the journalists, bloggers, self-published book authors, internet commenters, wikipedia editors, and earlier models that taught them almost all of what they know.
That's a nice little aphorism. I think this happens in a lot of things in life. Like comments on Reddit always seem quite insightful until you actually read the article they're commenting on.
Sounds a bit like Gell-Mann Amnesia Effect: https://en.wikipedia.org/wiki/Michael_Crichton#Gell-Mann_amn...
The Alt-Mann Amnesia Effect, maybe.
Dataset: http://openaipublic.blob.core.windows.net/simple-evals/simpl...
First few questions for those who don't care to download. Most just seem to be about niche facts:
Also importantly, they do have a 'not attempted' or 'do not know' type of response, though how it is used is not really well discussed in the article.
As it has been for decades now, the 'Nan' type of answer in NLP is important, adds great capability, and is often glossed over.
What’s more interesting to me here are the calibration graphs:
• LLMs, at least GPT models, tend to overstate their confidence. • A frequency-based approach appears to achieve calibration closer to the ideal.
This kinda passes my vibe test. That said, I wonder—rather than running 100 trials, could we approximate this by using something like a log-probability ratio? This would especially apply in cases where answers are yes or no, assuming the output spans more than one token.
If you imagine a future where LLMs get faster and cheaper even without getting better it means we'd be able to automatically repeat questions 100x and every answer could come with a pretty good confidence measure.
yeah, this is by far the most interesting part of this page, the fact that LLMs can know what they know is not a trivial fact.
I hope this dataset is used to probe the wrong answers much more than try to get all the answers correct. I don't need LLMs to know everything, but I do need them to know what they don't know. That's not a capability that comes naturally to a probabilistic sampling of tokens.
The ideal 'progress' on this benchmark is for a model to remove incorrect answers and replace them with I don't know answers. Even if it hurts the correct answer count a bit I'd gladly make that tradeoff for a model that hallucinated far less often.
>That's not a capability that comes naturally to a probabilistic sampling of tokens.
The linked page from OpenAI clearly show the opposite, read the "Using SimpleQA to measure the calibration of large language models" and the paper linked: https://arxiv.org/abs/2207.05221
The probabilistic sampling of tokens does not naturally produce introspective evaluation of confidence, it enforces a highest probability token selection (in greedy sampling). The paper that you linked demonstrates that if a separate evaluation phase is allowed then a model can decide with some accuracy whether its previous statements were true. This is not the behavior we want out of a system as it involves 1. Production of potentially misleading output 2. The identification of factual statements within that output 3. The classification of each of those statements and 4. Restatement of the original output without factual errors. The research area that I am advocating for would aim to prevent 1 not mask it.
Kudos:
> SimpleQA was created to be a greater challenge for frontier models (e.g., GPT-4o scores less than 40%).
And by design:
"To be included in the dataset, each question had to meet a strict set of criteria: ... and most questions had to induce hallucinations from either GPT-4o or GPT-3.5."
Honestly, I'd never expect it get 'correct's for every little fact like this, but it'd be great to get a lot more 'not attempted'.
"I seem, then, in just this little thing to be wiser than this man at any rate; that what I do not know I do not think I know either." - Socratos, from Plato's Apology of Socrates
I've tried using older models to create a cpu player on this lateral thinking game (https://detective-stories.com) and they were surprisingly bad at giving answers. I am curious to see how well the more recent models will do.
But aren‘t other LLMs just going to scrape the dataset and pre-learn the answers?
Are they going to make the benchmark available so other LLMs can be compared?
https://github.com/openai/simple-evals/blob/main/simpleqa_ev...
Any way to see the actual questions and answers? Where can I find simple_qa_test_set.csv ?
https://openaipublic.blob.core.windows.net/simple-evals/simp...
The steps I took to find this link:
1) Look at simpleqa_eval.py. See that it loads "az://openaipublic/simple-evals/simple_qa_test_set.csv" Hmm, some weird vendored protocol.
2) I don't feel like digging through bf.BlobFile() to figure out how it downloads files and I certainly don't want to generate an API key. Cross fingers and do a Bing web search for "az://openaipublic"
3) That leads me to https://stackoverflow.com/questions/76106366/how-to-use-tikt... Ah ha, this answer has the link https://openaipublic.blob.core.windows.net/encodings/cl100k_... which automatically downloads a file.
4) Poke the relevant parts of the az:// link into this link, and a csv appears.
This eval’s goal is a bit unclear to me, especially given the example questions. They’re very trivia/minutiae like asking about sports goals for example, which is their stated desire to test factual knowledge. But will this ever be possible by an LLM, without web browsing - which they deliberately removed while evaluating?
>But will this ever be possible by an LLM?
Why not? Just train an unbelievably gigantic LLM that encodes all human knowledge. A hundred trillion parameters ought to do it.
I think the interesting thing here is the difference between Not Attempt and Incorrect — the goal here seems to be to reduce hallucination
8 authors attached to this.
OpenAI going to have some rounds of layoffs in the future.