From rumours of Apple buying lots of GPUs from Nvidia not that long ago I think management got a nice little scare when OpenAI released GPT-3.5 and then GPT-4. It takes several years to bring a CPU to market. Apple probably realised far too late that they needed specific features in their SOCs to handle the new AI stuff, so it wasn’t included in anything before the A17 Pro. For the M1, M2 and M3 I believe that Apple is willing to sacrifice heat and battery to achieve what they want. The A17 Pro is probably very efficient at running LLMs so it can do so in a phone with a small battery and terrible thermal performance. For their Macs and iPads with M1, M2, M3 they will just run the LLMs on the AMX or the GPU cores and use more power and produce more heat.
Could also be a memory problem. The A17 Pro in the iPhone 15 Pro comes with 8 GB of memory while everything before that has 6 GB or less. All machines with the M1 or newer come with at least 8 GB of memory.
PS: The people who bought the Intel Mac Pro after the M1 was released knew very well what they were getting into.
It worth noting that the power of the neural engine doubled between the A16 and A17 chips (17 vs 35 TOPS, according to Wikipedia), while the A15 to A16 was a much more modest increase (15.8 to 17 TOPS). So it does seem like they started prioritizing AI/ML performance with the A17 design.
And at .6 TOP/second of performance, that Neural Engine is practically useless today. You can go buy a $50 Rockchip board with an NPU 10x faster.
Which introduces a funny aspect, of the whole NPU/TPU thing. There's a constant stairstepping in capability; the newer models improving only obsoletes older ones faster. It's a bit of a design paradox.
Yes. But I was responding to "Apple probably realised far too late". I think they were in fact way ahead of everyone else, it's just that the hardware of 2017 can't keep up with the demands of today.
It was specifically the LLM stuff. Their neural engines were never designed for running LLMs. The question is if the new neural engine in the A17 Pro and M4 actually have the required features to run LLMs or not. That’s at least what I suspect.
> I think they were in fact way ahead of everyone else,
This would be a lot easier to argue if they hadn't gimped their Neural Engine by only allowing it to run CoreML models. Nobody in the industry uses or cares about CoreML, even now. Back then, in 2017, it was still underpowered hardware that would obviously be outshined by a GPU compute shader.
I think Apple would be ahead of everyone else if they did the same thing Nvidia did by combining their Neural Engine and GPU, then tying it together with a composition layer. Instead they have a bunch of disconnected software and hardware libraries; you really can't blame anyone for trying to avoid iOS as an AI client.
I'm genuinely wondering why the NU was added in first place, I can't think of any app that made egregious use of that outside of the Gallery and Photo App. They didn't even allow any access from third parties in a few initial iterations.
On-device voice to text for Siri. Facial recognition in the photos app. Text recognition in photos. Scene recognition and adaptation in the camera app. And FaceID.
Not all those were available from day, except FaceID.
Could also be a memory problem. The A17 Pro in the iPhone 15 Pro comes with 8 GB of memory while everything before that has 6 GB or less. All machines with the M1 or newer come with at least 8 GB of memory.
PS: The people who bought the Intel Mac Pro after the M1 was released knew very well what they were getting into.