No mention of the venerable Tesla P4. 75W peak, 8GB VRAM, about $80 (£60).
I have 6x P4s, a Xeon E5 2696v3 (36 threads, 3.8ghz peak but all core turbo unlocked, so 6 cores at 3.8Ghz - about 8 cores at 3.5ghz, or all cores at 3.1ghz), 48GB DDR4, all fit into a micro atx case running on a 650W MSI psu.
This gives me a virtual 48GB GPU (llama.cpp ftw) to backup that 48GB of RAM.
I typically see scores of at least 7-12t/s on 20-30B Q4KM size dense models, on a 32K/48K/64K context, adequate for modern inference.
The pain point is the prompt loading, it is far far slower, minutes not seconds, than modern tensor core 8GB 5060s (my other machine's 2x GPUs) but is quite similar in regular inference speed once it has loaded.
Yes! P4 and the other small cards are fantastic. I've been more focused on developing a cooling system around the 2 slot sized cards so I don't have any of these lying around. Are you using P4 for anything outside of LLM work? I'm interested in seeing their image processing capabilities.
When I wanted to tinker with self-hosted models, I bought a couple of Radeon Pro V620 GPUs, because they're 32GB, still supported by current ROCm releases, and a few years newer than the similar-priced 32GB Nvidia cards (which are all EOL). They're a little faster than the old Tesla stuff, as well. 64GB is enough to run Gemma 4 31b 4-bit QAT with pretty big context at a respectable interactive speed (30+ tokens per second sustained).
That said, even the old Radeon Pro stuff has gotten more expensive on eBay, so I'm not necessarily recommending cheap old server cards that need custom-printed fan shrouds to operate in a consumer PC. Probably better to buy the Radeon AI Pro R9700 for $1400, which will be faster, supported for many years, and has a fan already. Or, maybe even the Intel ARC B70 for $1000.
Thanks! Yeah this is a major consideration. I have looked at power consumption throughout runs in the past (https://esologic.com/gpu-server-benchmark/#gpu-box-benchmark) and found that for many of these enterprise class cards, they're happy to slam right into the max TDP. So, for doing actual work, you'll be living up closer to the rated TDP of the cards. Recording power consumption is easy on nvidia and I'll likely add this to future versions of the benchmarking tool.
For an easier change than HTML or SVG, try running them though pngcrush to make the graph images much smaller. Won't give you the lossless vector quality, but you should be able to keep these image files visually indistinguishable at much lower size.
This initial round of benchmarking was to understand if there was any usecase here at all and I think there is. In a follow up, I'll be trying to answer questions like this. How big of a model can you fit on 4x M60, 4x P100, 4x V100? What are the tok/second when varying context length?
Do you have a set of models you'd like me to look at?
That's great. Personally, I'd interested in Qwen3.6-27B and deepseek V4 flash (or pro), with contexts above 60k. They seem to be popular and have good coding performance. I'd appreciate numbers on a single or two GPUs where a quantized version fits reasonably into the VRAM (Qwen in 16 or 24GB). 4 older GPUs approach a used 3090 in price, and the 3090 has better support for speedups like MTP. So cheaper but slower looks like a reasonable target to me.
No problem. Varying context size is a common request I've been getting as well. Personally I'm looking forward to seeing how much we can cram into the ancient K80's 24GB of VRAM :0
Similar interest here, possibly including if qwen 3.6, Gemma4 or DiffusionGemma (with the largest quants that will fit in a single card) will offer, say,
50 tokens-per-second (fast enough for interactive human-in-the-loop code research, print-f iterations on code to debug things, etc; or let the LLM churn on a problem for a minute while I step out to handle something else), context of up to 200k preferred.
Also if nothing else the below project lets you use an NVidia graphics card as low-latency swap, which has been nice as a buffer as RAM prices remain high and leaves me eyeing that 24GB card you mentioned as an alternative...
Darn, I was hoping to see bc-250's (aka PS5 chips) in there. They've recently become popular for inference and they are only about $200 on ebay. They hold a special place in my heart because I deployed 20k of them and I'm glad to see they are finding a purpose now and not just e-waste.
Yea, I'm bummed I didn't know about the 40-CU unlock, although it probably wouldn't have had much impact on mining performance. It still would have been neat to test. I did build a whole automated solution for auto-tuning each individual board. It would start at the "best" settings and then downgrade every time there was a crash. If it wasn't crashing, then those were the new "best" settings for that individual chip.
16 GPUs would require one or more 220V breaker panels, more akin to an EV charger than a computer. You would also quickly run out of PCIe lanes. My goal with this benchmarking is to think about what is the most cost effective way to fill 4U.
A few years ago we got rid of a bunch of K80s at work, they were not only obsolete but had gotten glitchy as hell. I suspect this is from the many heat/cool cycles they went through. When they were running flat out the exhaust air felt like a hair dryer.
Do you have any #s on how old they were at decomm time? There's some suspicion that part of the AI bubble is companies playing games with depreciation, eg assuming that H100/H200s will survive for 5 years.
They were probably about 8 years old. They were well past reasonable EOL, but they were used for teaching, so performance was not a primary concern, as long as they worked. They had reached the point of not working often enough that we finally scrapped them.
Depends on the use case, as for hardware h265 codecs a rtx 5070 Ti works just as well as the rtx 6000 gpu. Legacy GPU don't support modern codecs, but modern Intel chips have h265 HDR hardware support. Lower <16GB VRAM GPU are not really useful for "AI" model labs, so are often far more economical for rendering media.
One metric that isn't considered is VRAM, as some rendering pipelines still rely on composited baked-scenes to reduce each areas memory requirements.
In general, the $/performance unit will depend on what you are doing, but there is 1 more thing to consider... Old GPU use mystery binary BLOB drivers no longer maintained on modern kernels. You might get the software to work with a legacy Windows GPU driver, but the key takeaway concept here is "might". =3
"The results showed that these GPUs can still deliver significant compute power at a fraction of the cost of newer models, making them attractive for budget-conscious users." -Mistral AI
The rules for comments are now so lengthy that probably 90% of comments would violate one. It's like how police can pull you over for any reason and justify it by picking a law you've unwittingly violated
No mention of the venerable Tesla P4. 75W peak, 8GB VRAM, about $80 (£60).
I have 6x P4s, a Xeon E5 2696v3 (36 threads, 3.8ghz peak but all core turbo unlocked, so 6 cores at 3.8Ghz - about 8 cores at 3.5ghz, or all cores at 3.1ghz), 48GB DDR4, all fit into a micro atx case running on a 650W MSI psu. This gives me a virtual 48GB GPU (llama.cpp ftw) to backup that 48GB of RAM.
I typically see scores of at least 7-12t/s on 20-30B Q4KM size dense models, on a 32K/48K/64K context, adequate for modern inference.
The pain point is the prompt loading, it is far far slower, minutes not seconds, than modern tensor core 8GB 5060s (my other machine's 2x GPUs) but is quite similar in regular inference speed once it has loaded.
Yes! P4 and the other small cards are fantastic. I've been more focused on developing a cooling system around the 2 slot sized cards so I don't have any of these lying around. Are you using P4 for anything outside of LLM work? I'm interested in seeing their image processing capabilities.
> about $80 (£60)
Man, I wish I lived where you guys lived.
How the hell did you fit 6 P4s in a mATX case?
When I wanted to tinker with self-hosted models, I bought a couple of Radeon Pro V620 GPUs, because they're 32GB, still supported by current ROCm releases, and a few years newer than the similar-priced 32GB Nvidia cards (which are all EOL). They're a little faster than the old Tesla stuff, as well. 64GB is enough to run Gemma 4 31b 4-bit QAT with pretty big context at a respectable interactive speed (30+ tokens per second sustained).
That said, even the old Radeon Pro stuff has gotten more expensive on eBay, so I'm not necessarily recommending cheap old server cards that need custom-printed fan shrouds to operate in a consumer PC. Probably better to buy the Radeon AI Pro R9700 for $1400, which will be faster, supported for many years, and has a fan already. Or, maybe even the Intel ARC B70 for $1000.
Intriguing. I should benchmark my dust-gathering-stack of Titan V's, unless someone already has?
Great read. I'd love to know more about how power consumption changes as cards get newer too!
Thanks! Yeah this is a major consideration. I have looked at power consumption throughout runs in the past (https://esologic.com/gpu-server-benchmark/#gpu-box-benchmark) and found that for many of these enterprise class cards, they're happy to slam right into the max TDP. So, for doing actual work, you'll be living up closer to the rated TDP of the cards. Recording power consumption is easy on nvidia and I'll likely add this to future versions of the benchmarking tool.
for some of these gpus you can set a very reduced power limit for modest reduction in performance, tdp is not the full story
This site does not like being on the front page of HN. ~7MB for pictures of graphs that probably should have html or svg.
This is an interesting article though. Bookmarking since my dual e5-v4 system is unplugged until summer is over.
I'm trying bruh fuck!
For an easier change than HTML or SVG, try running them though pngcrush to make the graph images much smaller. Won't give you the lossless vector quality, but you should be able to keep these image files visually indistinguishable at much lower size.
Lesson learned for real, and TIL matplotlib is happy to export SVG. Good to know for next time. I upgraded my lightsail instance size in the meantime.
Have you tried 27B class models like qwen3.6?
This initial round of benchmarking was to understand if there was any usecase here at all and I think there is. In a follow up, I'll be trying to answer questions like this. How big of a model can you fit on 4x M60, 4x P100, 4x V100? What are the tok/second when varying context length?
Do you have a set of models you'd like me to look at?
That's great. Personally, I'd interested in Qwen3.6-27B and deepseek V4 flash (or pro), with contexts above 60k. They seem to be popular and have good coding performance. I'd appreciate numbers on a single or two GPUs where a quantized version fits reasonably into the VRAM (Qwen in 16 or 24GB). 4 older GPUs approach a used 3090 in price, and the 3090 has better support for speedups like MTP. So cheaper but slower looks like a reasonable target to me.
No problem. Varying context size is a common request I've been getting as well. Personally I'm looking forward to seeing how much we can cram into the ancient K80's 24GB of VRAM :0
Similar interest here, possibly including if qwen 3.6, Gemma4 or DiffusionGemma (with the largest quants that will fit in a single card) will offer, say, 50 tokens-per-second (fast enough for interactive human-in-the-loop code research, print-f iterations on code to debug things, etc; or let the LLM churn on a problem for a minute while I step out to handle something else), context of up to 200k preferred.
Also if nothing else the below project lets you use an NVidia graphics card as low-latency swap, which has been nice as a buffer as RAM prices remain high and leaves me eyeing that 24GB card you mentioned as an alternative...
https://github.com/c0deJedi/nbd-vram
I get 14-16 t/s on Qwen 3.6 - 27B Q4 MTP with a combination of P4000 + P5000.
Darn, I was hoping to see bc-250's (aka PS5 chips) in there. They've recently become popular for inference and they are only about $200 on ebay. They hold a special place in my heart because I deployed 20k of them and I'm glad to see they are finding a purpose now and not just e-waste.
Interesting! I had only heard of them as cheap gaming boxes. Didn't know they were being used for cheap inference, too, but it makes sense.
> They hold a special place in my heart because I deployed 20k
Sounds like something I'd love to hear more about if you can share
ethereum mining, long shut down...
Cool stuff. Just read: https://github.com/akandr/bc250
Yea, I'm bummed I didn't know about the 40-CU unlock, although it probably wouldn't have had much impact on mining performance. It still would have been neat to test. I did build a whole automated solution for auto-tuning each individual board. It would start at the "best" settings and then downgrade every time there was a crash. If it wasn't crashing, then those were the new "best" settings for that individual chip.
Wow holy crap this is news to me! I will have to consider picking some of these up for testing, what is it like working with them?
there is a discord server for fans of bc-250... lots of information there.
Would it possible to stack up to 16x32GB VRAM, and test the performance of a MOE model such as Deepseek-v4-flash?
16 GPUs would require one or more 220V breaker panels, more akin to an EV charger than a computer. You would also quickly run out of PCIe lanes. My goal with this benchmarking is to think about what is the most cost effective way to fill 4U.
A few years ago we got rid of a bunch of K80s at work, they were not only obsolete but had gotten glitchy as hell. I suspect this is from the many heat/cool cycles they went through. When they were running flat out the exhaust air felt like a hair dryer.
Do you have any #s on how old they were at decomm time? There's some suspicion that part of the AI bubble is companies playing games with depreciation, eg assuming that H100/H200s will survive for 5 years.
They were probably about 8 years old. They were well past reasonable EOL, but they were used for teaching, so performance was not a primary concern, as long as they worked. They had reached the point of not working often enough that we finally scrapped them.
This is a great datapoint. Someone else brought up that I should be memory checking the GPUs to understand if things are breaking down.
Depends on the use case, as for hardware h265 codecs a rtx 5070 Ti works just as well as the rtx 6000 gpu. Legacy GPU don't support modern codecs, but modern Intel chips have h265 HDR hardware support. Lower <16GB VRAM GPU are not really useful for "AI" model labs, so are often far more economical for rendering media.
https://www.pugetsystems.com/pugetbench/creators/davinci-res...
https://www.pugetsystems.com/pugetbench/creators/premiere-pr...
In some cases it is better to have lower passmark scores:
https://www.videocardbenchmark.net/gpu.php?gpu=RTX+PRO+6000+...
Blender is heavily bottle-necked by ray-tracing and de-noising operations:
https://opendata.blender.org/benchmarks/query/?compute_type=...
One metric that isn't considered is VRAM, as some rendering pipelines still rely on composited baked-scenes to reduce each areas memory requirements.
In general, the $/performance unit will depend on what you are doing, but there is 1 more thing to consider... Old GPU use mystery binary BLOB drivers no longer maintained on modern kernels. You might get the software to work with a legacy Windows GPU driver, but the key takeaway concept here is "might". =3
"The results showed that these GPUs can still deliver significant compute power at a fraction of the cost of newer models, making them attractive for budget-conscious users." -Mistral AI
"Don't post generated text" - https://news.ycombinator.com/newsguidelines.html
The rules for comments are now so lengthy that probably 90% of comments would violate one. It's like how police can pull you over for any reason and justify it by picking a law you've unwittingly violated
Trying to follow all the rules is fun for its own sake.
Most of them are just restatements of each other, with the occasional pet peeve thrown in.