> You can get that much ddr5 today for like $400

Not comparable. M2 Ultra with 128GB RAM has 800 GB/s bandwidth.

Maximum bandwidth for a DDR5 Intel 14900K system is 89.6 GB/s. https://www.intel.com/content/www/us/en/products/sku/236773/...

You can only get 2tb of storage max i believe, you are probably bottlenecked elsewhere if you are actually needing to move that much data that fast into memory in practice. I wonder what the temps would even be doing something that maxes that ram bandwidth

> I wonder what the temps would even be doing something that maxes that ram bandwidth

The temps will be completely normal since that is normal operation. TDP is only 80 watts for M2 Ultra.

> You can only get 2tb of storage max i believe, you are probably bottlenecked elsewhere

No. Just no. This is not the issue. The models are loaded into GPU memory.

yeah but you can't get an nvidia gpu with at least 128gb (H200) for less than $40k

You can set how much of your system ram your gpu gets in windows and linux

Useless for LLMs. Limited system memory bandwidth kills it. GPU will be mostly idle waiting for memory access. The H200 has 4.8 TB/s bandwidth. System memory is below 90 GB/s

If you have a use case for an h200 class gpu you are probably making enough money off of it to not care about $40k and the 900gb/s bandwidth mentioned in another comment would seem absolutely pedestrian in comparison. If these things were really that compelling of a performance per dollar ratio you wouldn’t be able to buy them. They’d be sold out because people would be filling datacenters with them.

M2 Ultra sits between a lower end Intel setup and an H200 for this purpose.

I reject this all-or-nothing black and white childish framing of it.