Reminds me of the largest AJAX app I worked on, back when jquery was still hot and IE6 still existed as a problem.
The landing page in our app used jqueryUI’s drag and drop support, back around the time they declared bankruptcy on the confusing buggy code and wouldn’t even accept bug fixes because they were replacing it component by component (which was taking almost 3x as long as predicted). We had columns you could drag items between but they had a max height and scroll bars and it turned out jqueryUI would let you drag items into different rows if the overflow area for adjacent drag targets overlapped your row.
The person who found it couldn’t fix it. The other fixer couldn’t fix it. I diagnosed it but the spaghetti code was a recursive mess and I could not find a spot where I could fix it. Especially given I couldn’t send in a patch to them.
So I spent half of my free time the last day of every (2 week) sprint for almost six months before I finally found a small function I could monkey patch to wrap it in a short circuit check for clipping region. I spent maybe 20,30 hours on this, a lot of it just getting back to the same situation to debug. But it felt like it took forever to fix it.
The short circuit also made drag and drop faster, which was just getting in the edge of distracting. Particularly on a crowded page.
I remember many similar cycles of having different browsers open side-by-side, and trying to pinpoint (without the developer tools we know and love today) the exact reason why one border was one pixel in one browser, and two pixels in the other, throwing the whole layout off.
Also remembering when Firebug for Firefox appeared, and made so many things so much easier. Suddenly things that took hours took days, and it was so much easier when you had some introspection tools.
* { border: red 1px solid }
Remember when IE6 was a thing? The kids today are angry at chrome for good reasons and yet, there was a time in which the most popular browser didn't implement jack shit from the specs. And it was the kind of browser that ships with the OS.
God the bad karma for working with this crap. I'm glad it's over.
I had to do a reflow reordering trick on a sibling page in that app and it doubled or tripled the speed on FF and safari, but on IE6 the test case went from 30s to 3.5s. Good Christ.
Incorrect Pytorch gradients with Apple MPS backend...
Yep this kind of thing can happen. I found and reported incorrect gradients for Apple's Metal-backed tensorflow conv2d in 2021 [1].
(Pretty sure I've seen incorrect gradients with another Pytorch backend, but that was a few years ago and I don't seem to have raised an issue to refer to... )
One might think this class of errors would be caught by a test suite. Autodiff can be tested quite comprehensively against numerical differentiation [2]. (Although this example is from a much simpler lib than Pytorch, so I could be missing something.)
Yeah, luckily, you can unit tests these and fix them. They are not concurrency bugs (again, luckily).
BTW, numeric differentiation can only be tested very limitedly (due to algorithmic complexity when you doing big matrix). It is much easier / effective to test against multiple implementations.
Not that I understand much of what they say, but it appears there are a lot of correctness bugs in pytorch that are flying under the radar, probably having a measurable impact on the results of model quality.
It would be interesting to see model weights comparison of the same model trained with the two to see if they exhibit meaningfully different behavior.
When we update Torch versions, we're required to run a test where the only change is the library change and compare the outputs. We saw a measurable improvement in accuracy by upgrading from torch 2.4.x to 2.7.x.
> Not that I understand much of what they say, but it appears there are a lot of correctness bugs in pytorch that are flying under the radar, probably having a measurable impact on the results of model quality.
Do you have any links to public thoughts about this? As if it was true, could mean a lot of research could be invalidated, so obviously would make huge news.
Also feels like something that would be relatively easy to make reproducible test cases from, so easy to prove if that's true or not.
And finally if something is easy to validate, and would make huge news, I feel like someone would already have attempted to prove this, and if it was true, would have published something a long time ago.
That's why project like nanochat are really cool, you can get around the limitations of such gigantic libraries, while at the same time understanding the underlying architecture.
They might be referring to Karpathy's earlier micrograd tutorial, where the whole thing is built from scratch. That was how I learned the basics myself.
I thought that the effect of these compiler flags was widely known in numerical computing. It allows e.g., reordering of floating point computations and in general disregards IEEE 754. As such, these results are expected, I’d think.
The unexpected thing in that particular case is that even if you were well aware and avoided the flag when building your numeric code, the way some other non-numeric-computing person compiled some unrelated non-numeric module like "gevent" could result in the fast-math behaviour being applied to your code too. (Happily gcc has now fixed this.)
Widely know amongst very niche groups, most of whom have either been burnt by the issue or heard about someone who has and have it ingrained in their mind out of fear of debugging such a thing.
I’d bet the majority of ML people are unaware, including those doing lower level stuff.
This is a great write up and I’d love to see more like it. Debugging this sort of thing in the megatron->pytorch->CUDA stack is what my team spends more than half of their time on as an ML research team.
Only slightly related, but how common are bugs in GPUs and/or CUDA? I'm currently on Day 5 of trying to debug why my GPT-OSS implementation (not using PyTorch) I've made from scratch isn't working correctly, and while I have it somewhat working with some naive and slow methods, I'm now doing an implementation of the tensor cores and have been just stuck for 2-3 days because of some small numerical difference I can't understand why it's happening.
Every day I'm getting closer to believing this is some sort of hardware bug in Blackwell or in CUDA itself, but as we know, the bug is (almost) never in the compiler or in the hardware. Until it is...
They exist, but they're not that common (give or take the "expected" numerical deviations based on the order of summation and whatnot, which can both be nontrivial and propagate error further).
Something I recommend doing, the best time being the start of the project and the second best time being now, is adding numerical gradient checking tests to all operations. You will make mistakes in your kernels from time to time, and it's valuable to know at a glance where those mistakes are.
Mind you, it's possible to write both the forward pass and the backward pass in a way that's wrong but compatible. An additional layer of checks I like to add is a dead-simple implementation of all algorithms -- no vectorization, no fancy blocking or re-orderings, nothing. Compare results to the simple implementation.
It sounds like a lot of work, but writing an optimized kernel is much slower than the numerical gradient checking and the simple kernel, and given how in numerical code it's basically impossible to identify the source of a bug without doing the equivalent of all of those checks, it only takes one bug in the whole project for the effort to pay off.
Thanks a lot for the pointers, I think I've done a similar approach to what you suggest, lots of tiny (relative) tests for each step in the process, and doing sort of sanity checking between the naive stuff I first wrote which works and which does inference correctly, and the new kernel which is a lot more performant, but currently incorrect and produces incoherent outputs.
I'll try to replace bits by simplified versions though, probably could help at least getting closer to knowing where the issue is.
Anyone have more debugging tips I'd greatly appreciate it! Nothing is too small or "obvious", as I'm about to lose my mind more or less.
Would make sense I suppose if I was using two different GPUs for the same thing and get two different outcomes. But instead I have two implementations (one naive, one tensor cores) running on the same GPU, but getting different outcomes, where they should be the same.
But then this joke might be flying above my head as well.
Consumer-visible hardware bugs are extremely uncommon nowadays. There's approximately 10x as many people working in design verification as actual hardware design.
I say "consumer-visible" because the bugs still exist and people who can catch them early get promoted quickly and paid a lot. It's very exciting work if you can get it, since you really have to understand the full GPU to break it.
Magnitudes away (maybe "small numerical difference" was an understatement), my current hypothesis is that I'm doing scaling wrong somewhere, but I can't help but sometimes slide into the "maybe there is something deeper wrong" territory in the evening after another day...
Sounds like Placeholder should somehow be split into InputPlaceholder and OutputPlaceholder, based on the usage.
Even identical classes could help future folks know copying back is platform specific: “hm, we wrote to an OutputPlaceholder but didn’t read back from it, that seems wrong”.
Great write-up, but I admit that I found the interweaving of human and AI-written content/headlines/summaries pretty distracting. I kept on wanting to scroll past, but had to keep on backtracking to find the human thread again.
I think if you want to give your reader a quick intro to, e.g., what is the Adam optimizer, a simple link to Wikipedia is fine. No need to copy-paste an AI tutorial on Adam into the blog post.
To be fair, you can easily click to hide those expanded sections. I found it a neat compromise between "Link to (usually) obtuse Wikipedia article" which aren't usually written for laypersons, and forcing me to read through stuff I already know about, I just hid the sections I already understood but found value in the others.
Apple used to contribute to the PyTorch MPS backend, but decided to create their own framework (MLX) instead, fragmenting the ecosystem for very little gain. (MLX is basically PyTorch, but invented-at-apple)
Meta, the creator and main contributor to PyTorch, does not use Macs for their day-to-day ML work (they focus on GPUs and CPUs), so the MPS backend is sadly incomplete and has errors like the one you see here.
MLX and MPS are 2 completely different teams within Apple. It's more like MPS team doesn't have control or visibility into PyTorch roadmap and can only contribute so much from their side.
none of this is correct (except the part where FB doesn't use apple in prod).
EDIT: for the downvoters - i'll repeat, this is not a correct assessment of the relationship between Apple and PyTorch. but you can keep downvoting if you want <shrug>
Apple Engineering's *actual* time spent on PyTorch support has't given me confidence that PyTorch Mac experience would get anywhere close to NVIDIA's any time soon, if ever.
The Meta engineers continue to do a huge amount of heavy-lifting for improving the MPS backend, including feeling the responsibility for the Mac experience. Apple's priorities keep changing, the number of engineering hours they contribute keeps changing and their interest in actually and wholly owning the PyTorch MPS backend keeps varying.
If Apple wants MacStudio to become an actual AI devbox, and not just an AI inference machine, then prioritizing software support for PyTorch (>90% marketshare in AI) would probably be a good idea."
This is a minor quibble but I don't really like the author calling Placeholder a leaky abstraction. It's just straight up an incomplete abstraction that only handles inputs but not outputs. As the author says, Placeholder should know about the difference and do the copy-back itself.
haha oops yeah the other comment is correct- that was just a mistake
I originally wrote "vanilla" there but didn't want to repeat that word twice in a row so swapped it for "standard" without realizing it now looked like the SGD acronym
just fixed that to avoid confusion- thanks for pointing it out!
Another reason people use Nvidia. You know that Nvidia is the most used backend and the most likely to have this kind of bug found and fixed before you encounter it.
Reminds me of the largest AJAX app I worked on, back when jquery was still hot and IE6 still existed as a problem.
The landing page in our app used jqueryUI’s drag and drop support, back around the time they declared bankruptcy on the confusing buggy code and wouldn’t even accept bug fixes because they were replacing it component by component (which was taking almost 3x as long as predicted). We had columns you could drag items between but they had a max height and scroll bars and it turned out jqueryUI would let you drag items into different rows if the overflow area for adjacent drag targets overlapped your row.
The person who found it couldn’t fix it. The other fixer couldn’t fix it. I diagnosed it but the spaghetti code was a recursive mess and I could not find a spot where I could fix it. Especially given I couldn’t send in a patch to them.
So I spent half of my free time the last day of every (2 week) sprint for almost six months before I finally found a small function I could monkey patch to wrap it in a short circuit check for clipping region. I spent maybe 20,30 hours on this, a lot of it just getting back to the same situation to debug. But it felt like it took forever to fix it.
The short circuit also made drag and drop faster, which was just getting in the edge of distracting. Particularly on a crowded page.
I remember many similar cycles of having different browsers open side-by-side, and trying to pinpoint (without the developer tools we know and love today) the exact reason why one border was one pixel in one browser, and two pixels in the other, throwing the whole layout off.
Also remembering when Firebug for Firefox appeared, and made so many things so much easier. Suddenly things that took hours took days, and it was so much easier when you had some introspection tools.
That bug took me on a whirlwind tour of that code and I understand why they wanted to start over. Woof.
* { border: red 1px solid } Remember when IE6 was a thing? The kids today are angry at chrome for good reasons and yet, there was a time in which the most popular browser didn't implement jack shit from the specs. And it was the kind of browser that ships with the OS.
God the bad karma for working with this crap. I'm glad it's over.
I had to do a reflow reordering trick on a sibling page in that app and it doubled or tripled the speed on FF and safari, but on IE6 the test case went from 30s to 3.5s. Good Christ.
Incorrect Pytorch gradients with Apple MPS backend...
Yep this kind of thing can happen. I found and reported incorrect gradients for Apple's Metal-backed tensorflow conv2d in 2021 [1].
(Pretty sure I've seen incorrect gradients with another Pytorch backend, but that was a few years ago and I don't seem to have raised an issue to refer to... )
One might think this class of errors would be caught by a test suite. Autodiff can be tested quite comprehensively against numerical differentiation [2]. (Although this example is from a much simpler lib than Pytorch, so I could be missing something.)
[1] https://github.com/apple/tensorflow_macos/issues/230
[2] https://github.com/sradc/SmallPebble/blob/2cd915c4ba72bf2d92...
Yeah, luckily, you can unit tests these and fix them. They are not concurrency bugs (again, luckily).
BTW, numeric differentiation can only be tested very limitedly (due to algorithmic complexity when you doing big matrix). It is much easier / effective to test against multiple implementations.
The tinygrad folks talk about this a lot.
Not that I understand much of what they say, but it appears there are a lot of correctness bugs in pytorch that are flying under the radar, probably having a measurable impact on the results of model quality.
It would be interesting to see model weights comparison of the same model trained with the two to see if they exhibit meaningfully different behavior.
When we update Torch versions, we're required to run a test where the only change is the library change and compare the outputs. We saw a measurable improvement in accuracy by upgrading from torch 2.4.x to 2.7.x.
> Not that I understand much of what they say, but it appears there are a lot of correctness bugs in pytorch that are flying under the radar, probably having a measurable impact on the results of model quality.
Do you have any links to public thoughts about this? As if it was true, could mean a lot of research could be invalidated, so obviously would make huge news.
Also feels like something that would be relatively easy to make reproducible test cases from, so easy to prove if that's true or not.
And finally if something is easy to validate, and would make huge news, I feel like someone would already have attempted to prove this, and if it was true, would have published something a long time ago.
Check their Twitter, I saw something either yesterday or earlier today iirc
That's why project like nanochat are really cool, you can get around the limitations of such gigantic libraries, while at the same time understanding the underlying architecture.
Nanochat is using PyTorch under the hood. I don’t understand your point.
They might be referring to Karpathy's earlier micrograd tutorial, where the whole thing is built from scratch. That was how I learned the basics myself.
https://moyix.blogspot.com/2022/09/someones-been-messing-wit...
TLDR: Python gevent compiled with -Ofast messes up x87 floating point unit state. Bad for PyTorch.
I thought that the effect of these compiler flags was widely known in numerical computing. It allows e.g., reordering of floating point computations and in general disregards IEEE 754. As such, these results are expected, I’d think.
The unexpected thing in that particular case is that even if you were well aware and avoided the flag when building your numeric code, the way some other non-numeric-computing person compiled some unrelated non-numeric module like "gevent" could result in the fast-math behaviour being applied to your code too. (Happily gcc has now fixed this.)
Widely know amongst very niche groups, most of whom have either been burnt by the issue or heard about someone who has and have it ingrained in their mind out of fear of debugging such a thing.
I’d bet the majority of ML people are unaware, including those doing lower level stuff.
The tinygrad folks talk too much
I see another commenter highlighted this:
> The exact same float32 code updates weights on CPU but fails on MPS
It's MPS... Exactly zero research is being impacted. Why doesn't the $3.9T corporation contribute more to torch?
I mean, some researchers clearly use Apple Silicon for their "cheap and cheerful" runs.
This is a great write up and I’d love to see more like it. Debugging this sort of thing in the megatron->pytorch->CUDA stack is what my team spends more than half of their time on as an ML research team.
Only slightly related, but how common are bugs in GPUs and/or CUDA? I'm currently on Day 5 of trying to debug why my GPT-OSS implementation (not using PyTorch) I've made from scratch isn't working correctly, and while I have it somewhat working with some naive and slow methods, I'm now doing an implementation of the tensor cores and have been just stuck for 2-3 days because of some small numerical difference I can't understand why it's happening.
Every day I'm getting closer to believing this is some sort of hardware bug in Blackwell or in CUDA itself, but as we know, the bug is (almost) never in the compiler or in the hardware. Until it is...
They exist, but they're not that common (give or take the "expected" numerical deviations based on the order of summation and whatnot, which can both be nontrivial and propagate error further).
Something I recommend doing, the best time being the start of the project and the second best time being now, is adding numerical gradient checking tests to all operations. You will make mistakes in your kernels from time to time, and it's valuable to know at a glance where those mistakes are.
Mind you, it's possible to write both the forward pass and the backward pass in a way that's wrong but compatible. An additional layer of checks I like to add is a dead-simple implementation of all algorithms -- no vectorization, no fancy blocking or re-orderings, nothing. Compare results to the simple implementation.
It sounds like a lot of work, but writing an optimized kernel is much slower than the numerical gradient checking and the simple kernel, and given how in numerical code it's basically impossible to identify the source of a bug without doing the equivalent of all of those checks, it only takes one bug in the whole project for the effort to pay off.
Thanks a lot for the pointers, I think I've done a similar approach to what you suggest, lots of tiny (relative) tests for each step in the process, and doing sort of sanity checking between the naive stuff I first wrote which works and which does inference correctly, and the new kernel which is a lot more performant, but currently incorrect and produces incoherent outputs.
I'll try to replace bits by simplified versions though, probably could help at least getting closer to knowing where the issue is.
Anyone have more debugging tips I'd greatly appreciate it! Nothing is too small or "obvious", as I'm about to lose my mind more or less.
You may be running into jensen (huang)’s inequality,
E(loss).cuda() <= E(loss.cuda())
Would make sense I suppose if I was using two different GPUs for the same thing and get two different outcomes. But instead I have two implementations (one naive, one tensor cores) running on the same GPU, but getting different outcomes, where they should be the same.
But then this joke might be flying above my head as well.
Tensor cores use lower precision, so small numerical differences should be expected.
Consumer-visible hardware bugs are extremely uncommon nowadays. There's approximately 10x as many people working in design verification as actual hardware design.
I say "consumer-visible" because the bugs still exist and people who can catch them early get promoted quickly and paid a lot. It's very exciting work if you can get it, since you really have to understand the full GPU to break it.
Good luck!!
How big is the numerical difference? If it's small it might be within the precision of the operation itself.
Magnitudes away (maybe "small numerical difference" was an understatement), my current hypothesis is that I'm doing scaling wrong somewhere, but I can't help but sometimes slide into the "maybe there is something deeper wrong" territory in the evening after another day...
Sounds like Placeholder should somehow be split into InputPlaceholder and OutputPlaceholder, based on the usage.
Even identical classes could help future folks know copying back is platform specific: “hm, we wrote to an OutputPlaceholder but didn’t read back from it, that seems wrong”.
Apps Hungarian v. System Hungarian: https://herbsutter.com/2008/07/15/hungarian-notation-is-clea...
What a fantastic way to write a post mortem, pedagogically very useful.
Great write-up, but I admit that I found the interweaving of human and AI-written content/headlines/summaries pretty distracting. I kept on wanting to scroll past, but had to keep on backtracking to find the human thread again.
I think if you want to give your reader a quick intro to, e.g., what is the Adam optimizer, a simple link to Wikipedia is fine. No need to copy-paste an AI tutorial on Adam into the blog post.
To be fair, you can easily click to hide those expanded sections. I found it a neat compromise between "Link to (usually) obtuse Wikipedia article" which aren't usually written for laypersons, and forcing me to read through stuff I already know about, I just hid the sections I already understood but found value in the others.
I came here to say the same thing. Claude’s voice was pretty evident, but became actually grating when the header was “The Fix”.
I too have been insanely burned by an MPS bug. I wish Apple would throw an engineer or two at making sure their hardware works with PyTorch.
Apple used to contribute to the PyTorch MPS backend, but decided to create their own framework (MLX) instead, fragmenting the ecosystem for very little gain. (MLX is basically PyTorch, but invented-at-apple)
Meta, the creator and main contributor to PyTorch, does not use Macs for their day-to-day ML work (they focus on GPUs and CPUs), so the MPS backend is sadly incomplete and has errors like the one you see here.
MLX and MPS are 2 completely different teams within Apple. It's more like MPS team doesn't have control or visibility into PyTorch roadmap and can only contribute so much from their side.
none of this is correct (except the part where FB doesn't use apple in prod).
EDIT: for the downvoters - i'll repeat, this is not a correct assessment of the relationship between Apple and PyTorch. but you can keep downvoting if you want <shrug>
Please be specific if you have anything to say. By the way, the co-creator and core maintainer of PyTorch has the same opinion as me.
https://x.com/soumithchintala/status/1978848796953161754
"MacStudio you ask?
Apple Engineering's *actual* time spent on PyTorch support has't given me confidence that PyTorch Mac experience would get anywhere close to NVIDIA's any time soon, if ever.
The Meta engineers continue to do a huge amount of heavy-lifting for improving the MPS backend, including feeling the responsibility for the Mac experience. Apple's priorities keep changing, the number of engineering hours they contribute keeps changing and their interest in actually and wholly owning the PyTorch MPS backend keeps varying.
If Apple wants MacStudio to become an actual AI devbox, and not just an AI inference machine, then prioritizing software support for PyTorch (>90% marketshare in AI) would probably be a good idea."
Everyone I know at Meta uses a Mac
No one at Meta runs local inference on a Mac, unless its for fun.
awesome read!
this is a great writeup! methodical without being pedantic.
Nice work, surprising, I'd imagine implementations are cross tested all the time and this kind of bugs have no way of appearing?
Naive question: ML tensor libraries don’t use a Z-order memory layout like textures do? It’s not beneficial like it is for textures?
This is a minor quibble but I don't really like the author calling Placeholder a leaky abstraction. It's just straight up an incomplete abstraction that only handles inputs but not outputs. As the author says, Placeholder should know about the difference and do the copy-back itself.
Dumb question: why isn't there some kind of assertion to sanity-check some bits of the GPU results against CPU's?
This is the first time I see "SGD" to mean "standard gradient descent" and not "stochastic gradient descent".
haha oops yeah the other comment is correct- that was just a mistake
I originally wrote "vanilla" there but didn't want to repeat that word twice in a row so swapped it for "standard" without realizing it now looked like the SGD acronym
just fixed that to avoid confusion- thanks for pointing it out!
Presumably that's just a mistake. The author calls it "stochastic gradient descent" correctly elsewhere in the article
Another reason people use Nvidia. You know that Nvidia is the most used backend and the most likely to have this kind of bug found and fixed before you encounter it.
Non-contiguous tensors have to be the #1 source of bugs in PyTorch lol