Classic implementations of LLMs (like llama.cpp) and diffusion image models allow to specify seed, and as long as it runs the same code on the same hardware with the same parallelism level the result will be the same. This is even checked in autotests[1]. The thing that produces randomized results in floating point operations (excluding bugs) is known as "stochastic rounding": it is pretty novel (from implementations standpoint) and it also can be controlled by seed. Other than that I've never seen hardware that has non-deterministic (maybe stochastic) output, but maybe we will see it in the next few years.

[1] https://github.com/ggerganov/llama.cpp/blob/master/examples/...

Funny wrinkle here: unless I’ve misread the OpenAI API docs[1], the recently added prompt caching feature cannot be explicitly disabled and automatically applies to all input prompts over 1024 tokens for a ~few minutes.

It seems to be possible to work around it by mixing up the very start of your prompt (e.g., with an iteration number), but it’s messed up some of our workflows which rely on running multiple hits with the same prompt to gather a consensus output.

Would be great if they let us disable it.

[1]: https://platform.openai.com/docs/guides/prompt-caching