SQL functions can be non-deterministic just fine. E.g. SQL:2003 grammar defines DETERMINISTIC | NOT DETERMINISTIC characteristic for CREATE FUNCTION. Or, e.g. PostgreSQL has IMMUTABLE | STABLE | VOLATILE clauses.
Not at all. Even the ones that provide a "seed" parameter don't generally 100% guarantee you'll get back the same result.
My understanding is that this is mainly down to how floating point arithmetic works. Any performant LLM will be executing a whole bunch of floating point arithmetic in parallel (usually on a GPU) - and that means that the order in which those operations finish can very slightly affect the result.
Classic implementations of LLMs (like llama.cpp) and diffusion image models allow to specify seed, and as long as it runs the same code on the same hardware with the same parallelism level the result will be the same. This is even checked in autotests[1]. The thing that produces randomized results in floating point operations (excluding bugs) is known as "stochastic rounding": it is pretty novel (from implementations standpoint) and it also can be controlled by seed. Other than that I've never seen hardware that has non-deterministic (maybe stochastic) output, but maybe we will see it in the next few years.
Do you know why OpenAI are unable to provide a "seed" parameter that's fully deterministic? I had assumed it was for the reason I described, but I'm not confident in my assertion there.
The original question was about LLMs, and what OpenAI provides is out of LLM definition (they have caches, "memory", undocumented o1 Chain-of-Thought and so on). Also I am pretty sure that they collected a zoo of hardware with autoscaling and per-client parallelism. And if you change number of threads even basic bricks like matrix multiplication produce different result.
Funny wrinkle here: unless I’ve misread the OpenAI API docs[1], the recently added prompt caching feature cannot be explicitly disabled and automatically applies to all input prompts over 1024 tokens for a ~few minutes.
It seems to be possible to work around it by mixing up the very start of your prompt (e.g., with an iteration number), but it’s messed up some of our workflows which rely on running multiple hits with the same prompt to gather a consensus output.
They are not, necessarily. Especially when using commercial providers who may change models, finetunes, privacy layers, and all kinds of other non-foundational-model things without notice.
I love the simplicity of this. Hurray for small models for small tasks.
open source version: https://github.com/dsg-polymtl/large-flock
Interesting -- is there any impact from LLM outputs not being deterministic?
SQL functions can be non-deterministic just fine. E.g. SQL:2003 grammar defines DETERMINISTIC | NOT DETERMINISTIC characteristic for CREATE FUNCTION. Or, e.g. PostgreSQL has IMMUTABLE | STABLE | VOLATILE clauses.
That's how something like `clock_timestamp()` can be supported. It returns the timestamp at time of execution, so it can change even within the same query if called multiple times: https://www.postgresql.org/docs/current/functions-datetime.h...
Nice, TIL. Thanks!
Aren't LLM outputs deterministic given the same inputs?
Not at all. Even the ones that provide a "seed" parameter don't generally 100% guarantee you'll get back the same result.
My understanding is that this is mainly down to how floating point arithmetic works. Any performant LLM will be executing a whole bunch of floating point arithmetic in parallel (usually on a GPU) - and that means that the order in which those operations finish can very slightly affect the result.
Classic implementations of LLMs (like llama.cpp) and diffusion image models allow to specify seed, and as long as it runs the same code on the same hardware with the same parallelism level the result will be the same. This is even checked in autotests[1]. The thing that produces randomized results in floating point operations (excluding bugs) is known as "stochastic rounding": it is pretty novel (from implementations standpoint) and it also can be controlled by seed. Other than that I've never seen hardware that has non-deterministic (maybe stochastic) output, but maybe we will see it in the next few years.
[1] https://github.com/ggerganov/llama.cpp/blob/master/examples/...
Do you know why OpenAI are unable to provide a "seed" parameter that's fully deterministic? I had assumed it was for the reason I described, but I'm not confident in my assertion there.
The original question was about LLMs, and what OpenAI provides is out of LLM definition (they have caches, "memory", undocumented o1 Chain-of-Thought and so on). Also I am pretty sure that they collected a zoo of hardware with autoscaling and per-client parallelism. And if you change number of threads even basic bricks like matrix multiplication produce different result.
Funny wrinkle here: unless I’ve misread the OpenAI API docs[1], the recently added prompt caching feature cannot be explicitly disabled and automatically applies to all input prompts over 1024 tokens for a ~few minutes.
It seems to be possible to work around it by mixing up the very start of your prompt (e.g., with an iteration number), but it’s messed up some of our workflows which rely on running multiple hits with the same prompt to gather a consensus output.
Would be great if they let us disable it.
[1]: https://platform.openai.com/docs/guides/prompt-caching
They are not, necessarily. Especially when using commercial providers who may change models, finetunes, privacy layers, and all kinds of other non-foundational-model things without notice.