The memory unit economics of Apple Silicon are insane.
When you realise LLM inference is massively memory-bound (even at high concurrency!), you find that $ per memory bandwidth ($/GB/s) is the most important metric for inference.
M5 Ultra: $5499, 96GB @ 1.2TB/s, $4.5/GB/s
DGX Spark: $6950, 128GB @ 273GB/s, $25.5/GB/s
DGX Station: ~$150K, 252GB @ 7.1TB/s (HBM3e), $21.1/GB/s
M5 Ultra has 5.6x better $/GB/s than DGX Spark and 4.7x better $/GB/s than DGX Station.
I am surprised Apple didn't increase their prices more (maybe they will so might be a good time to buy...)
I expect 512GB M5 Ultra Mac Studio to start at $15899 (following the usual +$25/GB incremental memory price). That's a crazy good price for that much high speed memory.
With RDMA over Thunderbolt 5 you get single digit microsecond latency and you can connect up to 5 Thunderbolt 5 cables between Macs for 400Gb/s network bandwidth (same as ConnectX-7). Thunderbolt 5 cables are ~$25 each.
The main problem is the software on Apple Silicon is not great. There are about 30 competing inference engines, each supporting a small set of models, and quite unstable. The ecosystem around MLX is immature and limited (e.g. no Dynamo, LMCache, llm-d, ...). In our experience, more serious customers like Governments and Enterprises are put off by that and want something stable with an ecosystem of tools and software to run inference in production.
My mind is blown by how far ahead Apple is with local LLMs. The new Mac Studio M5 Ultra will have a 512 GB model for around 15-20k. You can chain 6 of them together using thunderbolt 5 and this have over 3 TB of ram and a massive amount of CPU and GPU cores. This means you can run a frontier model with over 2 trillion parameters locally for around 100kish. Nivida is around 500k for the same capability.