Would this approach be possible to extend downmarket, to older consumer cards? For a lot of LLM use cases we're constrained by memory and can tolerate lower compute speeds so long as there's no swapping. ELI5, what would prevent a hundred 1060-level cards from being used together?
> ELI5, what would prevent a hundred 1060-level cards from being used together?
In this case, you'd only have a single PCIe (v3!) lane per card, making the interconnect speed horribly slow. You'd also need to invest in thousands of dollars of hardware to get all of those cards connected and, unless power is free, you'd outspend any theoretical savings instantly.
In general, if you go back in card generations, you'll quickly hit so low memory limits amd slower compute that a modern CPU-based setup is better value.
