I haven’t been directly involved in networking for a while, so it was funny to tune back in and see that now not only is 10Gbps apparently no big deal, they even put it on the management interface!
Essentially, yes. The DGX Sparks, at 128 gigs for ~$4700 are one of the cheapest ways to get enough high-ish speed memory cobbled together to run one of the more capable open weight models to run at home/for a small biz. This switch lets you combine the memory of 2-4 of them.
I purchased my SSD based on price and I have a AM4 APU so I'm limited to pci-e 3, but a quick check with crystaldiskmark says I'm hitting about 3.6 GB/s on sequential reads in seq1m q8t1 ... That would saturate 25gbps (assuming you have the right access patterns). seq1m q1t1 does 2.3 GB/s on my system, close enough to 25 gbps.
By my math, pci-e 5.0 should do 4x the throughput, so if you had 4 pci-e ssds you wanted to serve over the network and you had the pci-e lanes, you could saturate 400 gbps.
I'm pretty happy with my little baby network with a handful of 10G and 2.5G ports, and I'm cheap so if I made a big upgrade it'd probably be something used that has more 10g ports plus a 40G I could DAC to the adjacent home server ... but my home servers are barely powerful enough to push 10g anyway. (Something like a DCS-7050TX-48 goes for ~ $100 and would be a big upgrade in capacity I don't use over the S3300-28X I currently run)
DGX Sparks have 200Gbps interlinks for clustering. Apparently switches are the only practical way to scale such clusters beyond 2 nodes even though each Spark has 2 such ports.
Well, the problem is that you want each node to be able to directly connect to each other, so you run into a mesh situation. https://www.youtube.com/watch?v=QJqKqxQR36Y shows 8 running at 100gbps and clustered.
(tl;dw - guy connects first four DGX and then another 4 "DGX-alikes". Uses 400gbps breakout cables (you can split interfaces, TIL) so he has 2 ports on the first switch, and then gets this one, I believe. You can buy cables that are 1x400 to 4x100gig, etc.
I mean at lot of what a homelab is experimenting. For example I have an absurd amount of communication 56GbE Mellanox switches but never really saturate it.
I haven’t been directly involved in networking for a while, so it was funny to tune back in and see that now not only is 10Gbps apparently no big deal, they even put it on the management interface!
Noting actually says those ports have to be for management.
Most people probably would use them for something else.
They're not connected to the ASIC so anything on those ports is taking the slow path.
Nic needs pcie5 x16 slot, so on am5, take out your gpu
> While $1100 is not a low number
It really is when you're talking 400G.
OK there's very few ports on this, but in general if you're looking at 400g interfaces. $1100 won't even get you a genuine Arista 400G passive DAC.
On their video Patrick points out that this is now lower than $1 per Gbps, cheaper than any of the cheapest AliExpress switches.
Is there an actual use for this in a homelab? Isn't 25gbps enough that no SSD will saturate it?
Please be careful with units. There are high-end consumer SSDs that will deliver data at 14+GB/s, which is much higher than 25Gb/s.
The second sentence of the article:
> We have a number of these now and have been running our RDMA network backend for the 8x NVIDIA GB10 cluster for several months now, using one.
I have a friend who uses something similar for his 4 NVIDIA GDX Spark cluster.
I see, so only if you are running an AI cluster. And what is it transferring between the nodes? Partial computations?
Essentially, yes. The DGX Sparks, at 128 gigs for ~$4700 are one of the cheapest ways to get enough high-ish speed memory cobbled together to run one of the more capable open weight models to run at home/for a small biz. This switch lets you combine the memory of 2-4 of them.
4-8 of them. 2 Sparks can operate without a switch.
I purchased my SSD based on price and I have a AM4 APU so I'm limited to pci-e 3, but a quick check with crystaldiskmark says I'm hitting about 3.6 GB/s on sequential reads in seq1m q8t1 ... That would saturate 25gbps (assuming you have the right access patterns). seq1m q1t1 does 2.3 GB/s on my system, close enough to 25 gbps.
By my math, pci-e 5.0 should do 4x the throughput, so if you had 4 pci-e ssds you wanted to serve over the network and you had the pci-e lanes, you could saturate 400 gbps.
I'm pretty happy with my little baby network with a handful of 10G and 2.5G ports, and I'm cheap so if I made a big upgrade it'd probably be something used that has more 10g ports plus a 40G I could DAC to the adjacent home server ... but my home servers are barely powerful enough to push 10g anyway. (Something like a DCS-7050TX-48 goes for ~ $100 and would be a big upgrade in capacity I don't use over the S3300-28X I currently run)
DGX Sparks have 200Gbps interlinks for clustering. Apparently switches are the only practical way to scale such clusters beyond 2 nodes even though each Spark has 2 such ports.
Well, the problem is that you want each node to be able to directly connect to each other, so you run into a mesh situation. https://www.youtube.com/watch?v=QJqKqxQR36Y shows 8 running at 100gbps and clustered.
Yes, AI... this video does a good job walking through: https://www.youtube.com/watch?v=QJqKqxQR36Y
(tl;dw - guy connects first four DGX and then another 4 "DGX-alikes". Uses 400gbps breakout cables (you can split interfaces, TIL) so he has 2 ports on the first switch, and then gets this one, I believe. You can buy cables that are 1x400 to 4x100gig, etc.
I mean at lot of what a homelab is experimenting. For example I have an absurd amount of communication 56GbE Mellanox switches but never really saturate it.