A few months ago I bought a Mac with a M2 Pro 32G, and it is great for LLMs. I never thought that I would be running models like mixtral-8-7B-q3 locally on my own computer. I think the M2 Pro has 200GB/sec internal bandwidth, about 25% faster than the M3.
I think smaller models will keep getting better. For privacy and economic reasons, it makes sense to do as much AI on-device as possible.
But is it stable? I've seen a video on YouTube [0] where someone was showing LLM inferencing on a MacBook compared to an NVIDIA card, and it was really good, but before the video was over, everything on the Mac stopped working (he was also recording via OBS I think and it went to a halt).
This was a MacBook, but since you're just saying "Mac", I don't know what you're referring to.
I think that the MacBook couldn't cope with the heat generated so it went to its knees. I've read about it on Reddit as well.
I set my kids up with Faraday.dev on their M2 Airs and it does just fine. They will have chatbots up for hours to help them with schoolwork or when playing Minecraft to give them ideas.
The big thing is just don’t lock the models into memory on lower RAM systems. That gets you into trouble when some of the unified RAM is needed for something else and you can soft lock the system. The heat is not an issue though in my experience.
I bought a Mac Mini specifically for experimenting with LLMs. On the rare times I want to try fine tuning, I use Colab Pro or rent a server on Lambda Labs. I just use the M2 for inferencing, so far. So, no heat problems.
EDIT: maybe the YouTube demo was using 2 bit quantization?
I've heard multiple reports that you can hard-lock a M1/2 Mac simply by using OpenGL in harsh ways, so I wouldn't be surprised that an LLM + OBS can crash it.
I use my MBP for local ML and then CUDA in production. So far so good. We run YOLO inference models and Open CLIP models. Soon adding SD1.5 to the mix!
I tried everything for running small models locally, and finally settled on ollama.ai - good command line and REST API functionality. If you like Lisp, recent updates to my Racket and Common Lisp books have examples using the Ollama REST APIs (Can be read online).
Are PC laptops going to catch up? I wonder if there is some kind of conflict of interest with Nvidia holding back more capable PC chips or software integration.
x86 has to scale from embedded boards to laptops to servers. AMD made different trade offs than Apple.
Apple chose to achieve a level of performance by keeping the frequency low and having a high IPC. High IPC means that cores consume a larger size of the die. That helped with lower power consumption.
x86 makers like AMD, did the other way around. They dedicated a smaller portion of the die to a core. That makes it cheaper. To make up for that, they used a higher frequency, which also means a higher power consumption.
Apple also has a node advantage, which also makes their CPUs more expensive.
AMD could totally do what Apple did: have faster IPC at lower power consumption if they would have release more expensive CPUs targeted at the mobile market.
But AMD and Apple have different goals.
If we stick to the mobile world, Qualcomm is doing with their Orion cores just what Apple did with their M cores. In the first half of this year we will see laptops with Orion cores that will have both strong IPC and low power consumption.
> Apple also has a node advantage, which also makes their CPUs more expensive.
M2 is on TSMC N5. AMD's Zen 4 mobile processors are made with TSMC N4. From die shots, I believe the P-cores in the M2 are about the same size as Zen 4 cores. And yet Apple's M2 laptops are a good deal more power efficient than Zen 4 laptops.
> a good deal more power efficient than Zen 4 laptops
Zen4 is competitive when it comes to actual computing. When doing (almost) nothing, the Apple system comes out on top, but that also has a lot to do with its software.
I think macOS is also just better at actually being idle when the user isn’t doing anything. Windows seemingly always has something shuffling around in the background.
This is definitely part of it, as anyone who’s run Linux and Windows on the same machine can probably attest Windows is constantly doing… something? It is quite odd as it doesn’t seem to offer much in exchange for all these wasted cycles.
Yes, though Arm is probably a bit more robust at the lower end (think microcontrollers and very low-power industrial cores), and a little weaker at the high end (per-core performance on Neoverse still doesn't seem to match AMD/Intel).
To me the more interesting aspect is how memory architecture will be approached going ahead by both parties. Apple can afford to go more bespoke and integrated, but how can x86 maintain a good mix between performance and design flexibility.
>AMD could totally do what Apple did: have faster IPC at lower power consumption if they would have release more expensive CPUs targeted at the mobile market.
Doubtful. I'm not sure they have the talent/knowledge to pull it off while Apple Silicon is a moving target with a new generation introduced in each new iPhone.
Processor design isn't black voodoo. Its all configuration, most of which is now designed with CAD, and targeting particular use cases.
Its kinda like the evolution of bitcoin mining-everyone was doing it with GPUs, AMDs were better because of some hardware optimization where they could hash something in less cycles, then people started making ASIC which are purpose built and the fastest possible way to run the necessary compute. There were no technical advancements here, just economic forces driving the design and manufacture of components that made making ASICs profitable.
Apple did the same thing except they designed the chip and the computer as one, and they succeeded because it allowed them to optimize stuff further, and you see this in results where they are the leader but not by much, and often behind systems that are overclocked in single core performance. Power consumption on macs, also varies greatly depending on what you are running, which is again a result of optimization for particular use cases.
The iPhone is the most profitable product in the world. It gets massive resources to design a new SoC every single year. That resource spills over to the M series which is just based on whatever the A series does that year.
And yet, despite the advantages if all the Apple silicon, its not being used to run the backbone of the web, while AMD and Intel are. Making something work well in general purpose is just as "advanced" as making something super optimized.
Apple Silicon isn't designed to be server chips. The M1 team thought it was a good idea but management did not want to do it. So some of the team broke off and started Nuvia to design ARM chips for servers. Qualcomm acquired Nuvia and that became the Qualcomm Oryon SoCs.
Anyways, ARM chips are being used for the backbone of the internet. They're called Amazon Graviton chips. Ampere Computing also has ARM server chips. So do many Chinese hyperscalers.
AMD and Apple are in a different business. What makes sense for Apple doesn't make sense for AMD.
And AMD employed some of the most talented people in the area like Jim Keller. They totally can do it. Intel, too. Qualcomm is doing it with Orion and Qualcomm has less expertise and resources.
Qualcomm literally bought the team if Apple engineers who led the M1 design in Nuvia. That’s Oryon.
For the last 10 years, Qualcomm couldn't come close to Apple designed SoCs. Now they might come close after buying a team of ex-M1 designers.
It sounds ridiculous to say that AMD, Intel, and Qualcomm can do what Apple is doing. If they could, they'd be able to compete with Apple Silicon now. It's been 3 years. It's like saying AMD, Intel, Qualcomm can do what Nvidia is doing if they want to.
sometimes I do think about my first contributions. how i could have done this and that completely differently. kind of like going to first grade again with what you know (and are) as an adult.
yeah but look at freaking chromium, 3.2GB compressed archive. That is absolutely insane, and cannot be defended in ANY WAY.
look at what they started with with webkit? chromium is an abomination the likes of which definitively proves there is no god, or that god HATES humanity, as god would have 100% for sure schmited us all down for the atrocity that is chromium.
Chromium is one of the most amazing fully featured bundles of computing the planet has ever built. We should be in awe. But, "this has made a lot of people very angry and been widely regarded as a bad move." Contempt is rampant.
Yes, and people often forget that web browsers provide 100% backwards-compatibility to 100s of legacy web standards. Of course that bloats things up, see also Windows.
Also, parent is maybe refering to the source code size? Which also includes a huge chunk of ChromeOS code.
One doesn’t even need to go as far as C++ or Rust… many best in class Mac apps were written in Objective-C (which is notably slower than either) and were plenty responsive on single core PowerPC G3s, G4s, and G5s.
I think a great deal of the issue comes down to bloated frameworks, bad application code, and a prevalent attitude of nonchalance towards optimization that compounds with dev machines being powerful enough to mask most performance problems.
I think it's probably more productive to think of this in economic terms; "nonchalance toward optimization" <-> "I can write this code in an hour instead of three hours". And, from a company's perspective: "I can get this code running acceptably at 1/3 the dev cost".
Surely we’ve crossed a threshold of diminishing returns on this by now though? There has to be a point where the gains in convenience/DX/speed are too tiny for the costs involved and the weight of the average app ceases to grow.
Macs have Cocoa which is a good UI framework. There's your single good UI framework for them.
iOS has UIKit, which is a good UI framework. There's your single good UI framework for them.
Linux has GTK and Qt, though I'm not sure which one is in vogue these days, but both of which seemed fine when I used them in the past.
etc
But it sounds like what you're actually saying is "I only want one OS to exist".
Every OS has its own UI API, because every perform has different UI design language, UI interaction, and UI behavior. You cannot make one UI that is good on every platform because every platform is different and those differences are what make people prefer one over the others, but a single UI is (as Java and Chrome wrappers demonstrate) going to produce a bunch of exceedingly mediocre apps that fail numerous basic UI tools on their respective platforms.
If you want a cross platform apps you have cross platform core logic, and per platform UI.
If you want a single UI framework across platforms, it cannot be good, and it will inherently at best produce the kind of mediocrity we get with chrome apps, or you only have a single platform and have the same UI toolkit across all of them: desktop, phone, tablet, etc. I'm going to guess that's going to produce a fairly terrible UX on at least one of the options if it's good on others, or maybe it will just be mediocre across the board.
Yes, the problem is easy crappy apps are cheaper for developers even if they're worse for users.
The alternative "single UI framework" is "everyone must run the same OS".
As long as people keep buying/using crapware apps that's all that will happen. The fact that so many people are ok with that mediocrity is depressing, but that they do so while then complaining about things being slow is actively annoying.
It is totally possible to have a "runs on anything" UI framework; it simply has to have bindings for every applicable platform. SDL does this, for example, in 2D Graphics. Java has a few "run anywhere" UI frameworks, and Microsoft has .Net Maui (as well as the 3rd party avalonia UI). The problem with most native frameworks, however, is they are nowhere near as flexible as HTML and CSS (which many pre-teens taught themselves as a way to customize their tumblr and myspace pages). Other frameworks, by comparison, require programming knowledge or more setup than just a text editor (usually requiring an IDE and the rest of your toolchain).
I can see native software making a comeback if it is:
1. Declarative: only software engineers like imperative UI systems
2. Flexible: some frameworks (like any of the XAML-based ones) are strongly typed and you can only combine certain XAML elements with other certain XAML elements, and will fail to compile entirely if you mess something up
3. Unopinionated: HTML, CSS, and Javascript don't require you to implement any particular paradigm for getting data on the screen; that is left up to the coder
4. Dead ass simple: the litmus test is that a 7th grader should be able to learn the basics of it on their own, and the tooling should be ideally no more complicated than a text editor
5. Totally sandboxed: You can't trust 99% of frontend devs with security
The people you need to appeal to are not hardcore software engineers, but hobbyists and newbies, since they end up being the majority of frontend devs anyways.
Honestly I think you could get pretty far with just adding declarative languages (read: XML) to existing native frameworks, like gtk and Win32.
The problem is not putting bits onto the screen, that's all SDL does. It does not integrate at all with the host platform, to an even greater extent that chrome based crapware.
1. That's an opinion, not a fact, and most "declarative" UI frameworks have poor performance and can't do anything not explicitly models by the declarative framework. The fact that people want to just draw a UI doesn't mean that the resulting UI is either good, or works correctly, and the idea that it's "easier" simply implies lower understanding of what is being done and that you aren't considering any of the edge cases you still need to handle.
2. Not sure what to say here, as I don't know your point? it seems more about the type system of your language and not at all related to the ui toolkit
3. This is BS. html, css, and the DOM all have a very clear model for how and what a UI is, claiming it isn't means that you haven't apparently spent much time working in any other system. Much like I keep meeting people who say that they don't have an accent, everyone else does.
4. Application development is conceptually complicated unless your application is not doing anything interesting. I'm also curious what makes a browser ui more "simple" than say Cocoa?
5. Sandboxing is a part of the OS, not the UI framework. All applications on iOS are sandboxed, that vast majority on macOS are.
Application development is hard, and it's not because of the UI framework you're using. The overwhelming amount of any actual application is the application logic, not the UI.
Chrome based apps are all abysmal because to make an actual application using a browser engine, you need to reimplement, in html+JS, a huge amount of work the host OS would otherwise just provide for you.
If all you want is a webpage, just make a webpage, don't force everyone to download a 600Mb copy of chrome.
> It does not integrate at all with the host platform
SDL does in fact make system calls and have OS-specific code (DirectX implementation, Metal implementation), even if most of them don't directly manipulate OS UI controls, though it can initialize a window with a rendering context on any of its OSs.
I feel like you're missing the point that performance matters less than accessibility for UI frameworks. HTML and CSS are the most popular choices because a child can learn them quickly. Native apps aren't as popular because all the frameworks out there require you to be an engineer to use them.
> 1. the idea that it's "easier" simply implies lower understanding of what is being done and that you aren't considering any of the edge cases you still need to handle.
I don't see how that follows at all. HTML is declarative and powers most applications at this point.
> 2. Not sure what to say here, as I don't know your point? it seems more about the type system of your language and not at all related to the ui toolkit
Web browsers are very forgiving with HTML errors, native UI frameworks (especially those which simply convert the markup to imperative statements) often refuse to run entirely if there's a single markup error. A native framework which seeks to become as popular as HTML would need to be as flexible as HTML
> 3. html, css, and the DOM all have a very clear model for how and what a UI is, claiming it isn't means that you haven't apparently spent much time working in any other system. Much like I keep meeting people who say that they don't have an accent, everyone else does
You're not understanding what I'm saying. You can run an ajax request and get some data, but to put that data into the UI would require you to run a query on the dom and manually set the innerHTML of the element to the new data. Some other UI frameworks will force you to bind specific variables or lists to a control in order to auto refresh when the bound variable changes. This is an opinionated framework: one which tells you how to manage updating the UI rather than allowing the developer to do it themselves. Such an opinionated framework would become more difficult for people to learn
> 4. Application development is conceptually complicated unless your application is not doing anything interesting. I'm also curious what makes a browser ui more "simple" than say Cocoa?
I have no experience with cocoa because I don't use a mac, but from what I can tell it requires you to know Objective C and run XCode. HTML requires, at the very least, notepad to write. It is extremely easy to get setup with HTML and start learning it as a lay person.
> 5. Sandboxing is a part of the OS, not the UI framework. All applications on iOS are sandboxed, that vast majority on macOS are.
You would want to make sure that programs written in this framework are sandboxed away from any resources. On Windows this means AppContainers and on Linux there are different implementations (snap, docker). But the host which runs and loads the UI should run all applications sandboxed by default (i.e: like a web browser)
Your post is thoughtful. Why do you care so much about declarative? I tried SwiftUI last summer to see what it was about and I find the whole style rather awkward. Writing a desktop app in XML sounds even worse. And I'm not seeing the benefit to the user. With an imperative coding style, you can more directly control the user experience and give them the best one.
With an imperative style, you need to learn a programming language before you can start making applications. With HTML, you can start making applications immediately by just writing HTML, and then you can learn the programming language (javascript) later. It's a barrier to entry for non-engineers
Who will develop a cross-platform UI system as capable, flexible and well-embedded as HTML/CSS and as responsive as native UI? Google did its shot with Flutter, it's not exactly eating the world. The performance issue will long be solved before somebody comes up with a capable contender.
> Who will develop a cross-platform UI system as capable, flexible and well-embedded as HTML/CSS and as responsive as native UI?
History shows that you can only choose one. If you're not using the platform's native UI stack, you're going to end up with a slower, less capable solution that will look and feel foreign and clumsy in comparison. What the industry really needs is a React Native-type solution that isn't tied specifically to React.
Trying to do that with Slint: https://slint.dev
No need to use compiled language like C++, still possible to use a language like JavaScript/TypeScript, just no need to bring a browser.
This is what I'm trying to achieve. I'm going to build a cross-platform ecosystem of high quality apps (in terms of UX and performance) based on Qt C++ and QML. Starting with an alternative to the resource-hag Electron app: Notion, I'm building Plume[1]. Got some plans for what will come next.
Yes, but this seems fairly off topic when my complaint is about Slack and general software bloat. I know there exists responsive software, but much of it isn't.
Nah, my company full of people who can't even open an excel file are definitely going to be magically able to use an IRC client as their main form of communication.
if you are suggesting you have employees who are unable to be using an irc client, perhaps you should suggest to management that a round of layoffs are in order?
how hard can it be? spend 5 minutes and anyone thats worth keeping can figure it out.
There will always exist (always have) bloated and non-bloated programs. I don’t think anyone here knows anything about Slack’s particular future, but I bet it will not be getting any better.
Maybe if AI can code a native interface quickly on whatever platform you are on today. The problem with human programming is it takes time so these cross-platform abominations are the current solution. We need a hands-off, useable conversion amongst native platform toolkits.
Stop accepting these BS "apps" that are just web pages being loaded in a full copy of chrome. It's mind blowing to me that so many people accept these "cross platform" apps that are just as bad today as java "cross platform" apps were a few decades ago (if not actually worse).
Just a plethora of enshittified apps developed to be cheap rather than good.
Why would it? People are voting with their wallets that this is okay, that they'd rather have Slack and all that it has instead of some competing solution that's more responsive but lags in other areas, so there is no market pressure to improve this.
If I recall correctly, these integrated "AI accelerators" from both Intel and AMD are actually rather underpowered in comparison to an actual GPU (even the iGPU), at least in their current forms. So they don't have any performance advantage, but was put in place for power efficiency.
I think most chip manufacturers have plans to do that. But it's going to take some time for that to get to market. AMD, Nvidia, Samsung, and a few others are working on arm based chips for laptops. It will be interesting to see if Intel adapts. These things don't happen instantly and Apple seems to have caught most of the industry off guard by doing such a good job with the m1 and its successors. They put a stake in the ground for high performance computing on Arm and raised the ambition level quite a bit.
Nvidia of course has the added complexity that they were going to buy Arm and then that got blocked. They had to adjust to that as well. And of course they are the GPU provider of choice for GPUs to many AI developers and researchers. On paper they have all the right tech and ideas here.
AMD, Samsung and a few others have similar plans. My guess is there will be several options in the market in a few years. We'll have to see how good those are.
You can buy a laptop with a 4060 with 8 GB of VRAM for under $1500. For many ML tasks that’s plenty of memory for inference and training (assuming you’re willing to deal with a lower batch size).
Qualcomm has not given much details but it will be interesting to see performance numbers for 23Watt profile. Even if you can match M1/M2 performance that would be big step up from x86
It's difficult for them to catch up when the market for Windows laptops can't bear a 2k+ price tag. Expensive Windows laptops do exist and are pretty good, but they're few and far between.
I know giant gaming laptops exist, but in order to keep that power and go smaller, the price is going to jump up in the moon.
With that being said, the base-model Macbook Air is absolutely designed to leave you unsatisfied. 256gb of storage is a decade-running joke, and 8gb base RAM in an age of unified memory might as well be planned obsolescence.
The upgraded models feel like a different beast entirely. Apple wants you to feel like you're justified paying for 16gb of memory and 1tb of storage because... you are. Those upgraded machines run smooth and stay running nice after multiple years of software updates. MacOS is a storage-hungry OS, 3 years of daily-driving a base-model Macbook Air will make you wish you spent $2k on your laptop.
While I think it’s dumb to sell 8GB devices, I’m coming to the conclusion 256GB may not be a problem for normal people.
If your pictures are in iCloud or Google photos, your music is streamed, and other stuff is in iCloud or Google drive or whatever so most people need a ton of space these days?
I’m not arguing Apple’s RAM/storage prices are sane. They’re not. But with so much in the cloud or just being done in the browser instead of a large downloaded app i’m not sure normal users really benefit.
Paying for cloud storage may be a better deal, especially since I suspect the cloud is less likely to randomly lose data than if you forget your laptop somewhere or break it without a backup.
If you want to do video editing or a lot of programming that’s probably not gonna work. But if you’re mostly going to do “office task“ kind of things of typing and surfing and listening to music or watching YouTube maybe 256 GB isn’t so bad.
Base configuration should still be 16/512 in my opinion. But I’m not sure it’s as bad as it would’ve been 10 years ago to go low on storage.
The Air is designed to satisfy the target market for the Air, which it does, and is why they sell in large numbers. It runs Safari, MS Word, and TurboTax just fine.
For comparison, a base Surface Pro has even lower specs than a basic MacBook Air. (8GB/128GB)
But both of these devices are absolutely useable for general tasks.
With that being said, the base-model Macbook
Air is absolutely designed to leave you unsatisfied.
256gb of storage is a decade-running joke
I spent a couple of years doing Rails and Python work on a 2018 MBP (local dev) with 256GB. It's doable, barely, if you are willing to leave a low-profile thumbdrive permanently inserted. Or if you are willing to velcro an external SSD to your laptop lid =)
8GB+256GB is fine for a lot of use cases though. If you have a remote development setup, or if you are just kind of using your laptop as kind of a glorified $1099 chromebook.
I think the problem is Microsoft and Intel. Intel isn't too interested in making an ARM chip like the M2/M3, and Windows only runs on x86. If Linux had a significant marketshare on the desktop/laptop, this wouldn't be so much of a problem, and it'd be possible for someone to challenge Intel/AMD in non-Apple CPUs, but that's not the world we live in.
Snapdragon is making big claims about the Snapdragon X Elite. It is an ARM chip instead of x86, meant for the laptop market. I haven't delved deep into it yet so I can't make any comparisons or give my own take.
People seem to have taken "Apple made a very performant laptop" and factored it down to "the M1/2 series of chips beats everything every time" - which is obviously not the case.
For the particular use case of "a laptop on battery that doesn't die in ten minutes" Apple is decently far ahead. But everything else you can easily beat it if you make different trade-offs.
Which laptop with AMD CPU I can use to watch movies for 20 hours or browse the Internet for the same amount of time without recharging?
I am not an Apple or MacOs fan but I recently bought a MacBook Pro just for battery life. So I can watch movies and browse the Internet on the sofa without having to plug the laptop.
The ASUS ZenBook 14 has been tested to have about 18 hours of battery life for basic web browsing [1]. In the same test the 2020 MacBook Air 13" only scored about 14 hours. The Zenbook also easily beat the M1 in multicore performance, though single core is still where Apple shines.
Unfortunately, this test didn't include a more recent Macbook, so I can't tell how it stacks up against the M3. Still, for about €500 less than the M2 Air, I know what I would pick. The 1600p screen and the soldered-on memory is a bit of a downside, but the thing sells for less than half the price of a new M2 Macbook Air when you actually configure it with 16GB of RAM and 512GB of storage; whatever improvements the M2 chip made don't seem all that interesting for that price difference.
Based on their historical data [2] for another guide, the Zenbook also seems to beat the M2 in terms of multi core performance and browsing battery life when recorded in another test, but I can't tell if those numbers are representative.
Before buying a MacBook Pro 13", I've seen lots of reviews on notebookcheck. I bought my MacBook used and it really gives me 20 hours of battery life like in their test.
Every website tests different settings. There's a certain standard for brightness, the websites browsed, the software used, and so on. You can't compare battery tests unless all the test parameters are the same.
For instance, Tweakers uses a screen brightness of 180 cd/m² while testing, whereas NotebookCheck uses 150 cd/m². That alone should account for a sizeable difference in test results. I find it hard to tell what 30 cd/m² means in practice.
> The ASUS ZenBook 14 has been tested to have about 18 hours of battery life for basic web browsing. In the same test the 2020 MacBook Air 13" only scored about 14 hours.
Because the ZenBook has a 75 Wh battery, in comparison to the MBA's 52.6 Wh battery. That's one of the reasons it's heavier, too.
If you benchmark the AMD 7xxx series aganst the Apple chips while considering their power budget, AMD is on par or on top.
Battery life is strongly tied to software though, and here Apple has the upper hand. However, if you use Chrome vs Safari, you will notice degraded battery life on Mac OS too.
> However, if you use Chrome vs Safari, you will notice degraded battery life on Mac OS too.
Still way better than non-M chips.
I've done the runaround and used Safari, Orion, Chrome, Arc, and back to Firefox on my M1 MBP. Some are better than others, but the machine overall is still superior.
At both ends of the spectrum? My intuition is that on the low end they’d take a hit in efficiency by not having low-power cores to schedule background tasks onto like Intel and ARM have been pushing
Fair point. They do have efficiency cores (Zen4c) in some of the latest mobile and server chips. However, even when they do have dedicated efficiency cores, it's not clear that this would be a major advantage in the Windows software landscape.
Any modern CPU, made within the past 5 years or so (perhaps longer), is plenty fast, but the software either doesn't take advantage of that speed, or is does so in a very wrong way.
There is no reason for Windows to be sluggish on any mid-range CPU or up, yet here we are. I should not, I haven't used Windows for a good while, but I see demos and people are amazed by the speed of which windows open and coming from the Mac or a light weight Linux desktop, I can't say that I'm impressed.
Apple still holds the advantage that they just need to target a very small set of hardware configurations, all of which they have designed and control. It does still allow them to build the most pleasant desktop experience of all the current offering, even with less amazing hardware (mostly their GPU).
But this is HN, what people probably mean is: LLM are still better with an Nvidia GPU and neither Apples nor AMDs APIs and libraries are as good as Nvidias CUDA.
I don’t experience this difference with Windows vs Mac performance. Actually my desktop machine is much faster in most tasks than the similar vintage M1 Macbook Pro I use for work. Probably mostly due to faster storage in the desktop.
Personally, I won't. I'm far more interested in buying a 14700K to upgrade my 12700K daily driver desktop than I am replacing my M2 Macbook Air that spends most of its time as a portable jukebox.
Oh, I should have clarified - could one start with a bfloat16 on software-side, convert to float16 (so that e.g a 3.4E38 float16 becomes a 65504 float16), then do any "heavy math" in fast hardware float16 instructions, and then convert back at the end?
Nothing necessarily wrong with that code, but it also kinda smells. Why even store it as bfloat16 at all? You risk getting the numerical disadvantages of both float16-representation, and none of the advantages.
Oh, I should have clarified - could one start with a bfloat16 on software-side, convert to float16 (so that e.g a 3.4E38 float16 becomes a 65504 float16), then do any "heavy math" in fast hardware float16 instructions, and then convert back at the end?
No, you can't. The exponent in a float16 is too small. You'd rather convert back to a 32-bit float, do your operations, and then throw away the surplus precision and convert back to bfloat16.
This makes me wonder whether on-device AI will ever have a real chance against cloud based AI except for extremely latency sensitive tasks.
If the technology advances at such a pace that almost everyone is sitting on obsolete or far too weak hardware then the economics of on-device AI processing may not work out for app developers.
If it's just a matter of running a little bit slower or a little bit faster then it won't be a problem. But if people miss out on important capabilities altogether unless they spend thousands on hardware at least every other year then AI will be almost exclusively a cloud thing.
I expect to see dynamic, hybrid AI frameworks that let the same application run anywhere, with differing amounts of local compute depending on device capabilities.
Its the only way huge AI models can be latency and cost efficient across massively heterogeneous execution environments.
Multi-layer caches are also complex, but the benefits are worth it.
We already see NPUs spanning 3 orders of magnitude in TOPs in consumer devices. In the next few years that will probably be eight or ten orders of magnitude. I don’t think a LCD approach is going to work.
I would say that everything gets pushed to the cloud, but the economics are brutal. All of those companies spending billions a year on GPUs have very strong incentives to push as much inference to the edge as possible.
I'm considering whether I should trade in my M1 MBP for either an M2 15" MBA or M3 MBP today, or wait until M3 MBAs become a thing. I'm also wary of the trade-in value of the M1 MBP, which I guess will go down the longer I wait... But I suppose the value of jumping to a more powerful chip may outweigh the trade-in value of jumping to an M2 today...
> M1 MBP for either an M2 15" MBA or M3 MBP today, or wait until M3 MBAs become a thing
My 2 cents, since I am also looking to replace the original M1 MBP with something.
- Apple's official refurbished laptops are a thing, M2 already can be found on it. They have warranty.
- Think what you need, RAM vs CPU. If you run Docker containers, hundreds of browser tabs, work with heavy multimedia etc. and your budget is limited (or you wouldn't be asking) then you may want to consider M1/M2 with specced up RAM (you'd need to get Pro or Max to unlock higher RAM) instead of a basic M3. Yes, basic M3 is faster than basic M1, but it won't make your RAM issues go away so if you are at the limit of RAM capacity then you'll shoot yourself in the foot.
- If you do go for a basic model still consider upping at least storage because at the low end (less than 1TB) SSDs in M2 and M3 have been nerfed and are slower than SSDs in M1.
Oh good to know, thanks. The online quote they gave me was really decent and they didn't ask about keycaps, but I've never done a trade-in before so maybe they'd back out at the last minute after seeing my poor N key...
I’ve traded in more MacBooks than I care to count, almost always via mail (the method with the most complaints because it’s asynchronous).
Never had any issues and my keycaps have almost always been worn out.
As long as your device meets the condition as answered in the questionnaire, you should get what is quoted. Those are the only things they care about, they don’t even want the charger for example.
If you’re truly anxious about it, you can also walk into an Apple Store and you’ll get the decision on the spot.
Thanks, I guess you are the only one with valid experience here.
> As long as your device meets the condition as answered in the questionnaire
> If you’re truly anxious about it, you can also walk into an Apple Store and you’ll get the decision on the spot.
That's exactly what I did. The questionnaire was slightly different than for mail-in. The in-store questionnaire asked for "cracked keycaps" (online didn't) and the employee proceeded to answer yes saying that worn-out paint on keys is "cracked keycaps". Result: $0 value.
You can trade in by mail only for store credit, right? Do you know what happens if some employee uses the same stupid logic when handling mail-in? I suppose they just keep the device and give no store credit?
> Thanks, I guess you are the only one with valid experience here.
No I’m not, but I am the one who points it out when someone is spreading misinformation about Apple’s trade-in policy.
> That's exactly what I did. The questionnaire was slightly different than for mail-in. The in-store questionnaire asked for "cracked keycaps" (online didn't)
This makes me wonder which country you were in. I’ve done in-store trade ins both in the US and two EU countries, in neither of those cases was this part of the in-store questionnaire.
Keycaps are captured under the general functionality question both on the in-store as well as the online questionnaire.
> and the employee proceeded to answer yes saying that worn-out paint on keys is "cracked keycaps".
Sounds like you were dealing with an employee who didn’t know what they were doing. I wouldn’t have accepted that assessment to begin with, whether or not it was part of the questionnaire, and asked for a supervisor.
> Result: $0 value
At which point you of course would thank them kindly and keep your MacBook.
> You can trade in by mail only for store credit, right?
A gift card or value towards a pending order, so essentially store credit. Same as in-store.
> Do you know what happens if some employee uses the same stupid logic when handling mail-in? I suppose they just keep the device and give no store credit?
Same as in store. You get an email with an adjusted value offer (adjusted from the value based on the answers on the questionnaire) and if you don’t agree with that they’ll send it back to you at no cost.
I'm sure the policy is reasonable but if employees have overriding power over policy then it doesn't matter how good the policy is and saying it's misinformation implies I lie
> This makes me wonder which country you were in.
No comment:)
By the way, I did look it up after my failed attempt and on Reddit people in the US have different luck with in store and mail in. Apparently in store is stricter and people get turned away. So looks like misinformation was on your part.
> I wouldn’t have accepted that assessment to begin with, whether or not it was part of the questionnaire
Ever tried arguing with an Apple Store employee? How far did it get you?
> At which point you of course would thank them kindly and keep your MacBook.
Yes except for the thank you part. Mail-in offers around 300 USD or so voucher so I may try that and can always just sell it myself. (Second market is probably what trade-in was intended to reduce...)
I have an M2 for my work. MacOS still gives me random lag spikes sometimes. But it's probably because the OS is bad and not the hardware. This is never an issue on win/linux. idk. im not a fan. but.. im just saying.. if the hardware is so superior, why does it feel so inferior? Like.. my $500 linux laptop feels smoother and faster (GUI wise)
I’ve had a few corporate MacBooks sent to me by clients for security testing and they always seem much slower than my own. It’s normally down to security tooling they install on them.
Careful, your homedir has a CloudStorage folder and if you are using, say, Dropbox or Google Drive then that find will be incredibly slow (in addition to security software possibly slowing it down).
On the one hand, a lot of security software is poorly written, eats resources like it's Chrome, and introduces all kinds of microstutters through (exclusive) locks all over the place.
On the other hand, many operating systems don't provide a (reliable) API to design decent security software against. Log collection is all over the place, even on Windows, which traditionally had Event Log as a well concentrated logging destination. There's no way to write good security software for an operating system that's written without security software in mind.
If I were handling important secrets, I wouldn't want a fleet of machines out there with just the basic antivirus that came preinstalled with the OS (if it came with one at all). On the other hand, so many pieces of "enterprise" security management are absolutely terrible, and require one (or more) full-time employee(s) to constantly configure them, communicate with users, and solve problems, just to keep software in check.
I think both operating systems and security management software need to listen to each other, and change. Operating systems need to be written with security stuff in mind, and security software needs to focus on a good user experience rather collecting than shiny buzzwords to sell to management.
> Operating systems need to be written with security stuff in mind, and security software needs to focus on a good user experience rather collecting than shiny buzzwords to sell to management.
They won't because by the time the users get fed the "food" the contract is long since signed and valid for a few years, and the competition isn't better so even if management could be arsed to vote with their wallet, they couldn't.
Is there an actual term for "the entire industry is bullshit, but can get away with it because the cost to entry is so high that new, less bullshit competitors can't even enter the industry"?
Yeah. At my previous job we had all kinds of JAMF management software and Crowdstrike (??) and it was a massive performance killer.
Particularly, it seemed to be configured to scan every file on disk access, which was a performance nightmare for things like that involved "accessing lots of files" like Git on our large repo. Spotlight indexing seemed to cause some pretty big random lag spikes as well.
Do you work for a big corporation? They tend to install a ton of spyware on work machines they issue. Lots of flakey security and monitoring software with overlapping functionality.
Is this a work laptop at a large corporation? I’ve found things my company installs can bring any system to its knees. The M1 was the first time my fans didn’t kick on randomly due to some background process, which has been nice. Windows or macOS, they found ways to screw up both. I’m sure they’d do it to Linux too, if it was supported.
I get this on my Mac Studio when I do anything CPU-intensive, the UI becomes jittery. macOS on M CPUs seems to be particularly sensitive to this because I can push a Ryzen much harder and not notice anything in Windows.
Windows XP used to have that issue until they released some patch that prioritized UI when CPU was under stress. Linux had the same issue some time ago as well. I guess Apple will need to do something similar...
When you say lag spikes what kind of thing are you talking about? I have an M1 iMac I run pretty’s hard and I’m not sure what kind of behaviour you’re seeing.
Anything weird showing in Activity Monitor or Console?
If it refers to random beach balls for 30 seconds when accessing disk ether directly or through apps... yeah I get that. I found it disappointing, a flaw in an otherwise great machine (M1 Mac mini).
I have 16BG too. I have a Time Machine backup set up for a disk plugged into an AirPort Extreme, could that be the issue? (The machine connected to the network using WiFi if that matters.)
I used to hit this on an M1 with low RAM while absolutely assaulting it with a project that used docker compose to run far too many resources.
These days I can't seem to hit my M2 Max hard enough. Nothing slows it down. Just about anything will have lag spikes if you crank too much stuff through it, so I assume that (like me) you're experiencing this with devices running MacOS with around 8-16GB RAM, M1 processor at best?
>Like.. my $500 linux laptop feels smoother and faster (GUI wise)
Which Linux do you use? We've had fast enough machines for fast and smooth GUI rendering for decades now. It's just that most software isn't optimized any farther than a few years back.
Linux tends to be quite good on resource consumption. I can still run smooth and fast Linux on my 15 year old laptop. I can't do the same with Windows.
IME, Intel’s Clear Linux is unbelievably good. Most people never give it a chance, but it runs great on core i3 from 2017 with just 8GB of RAM… and this is an OS that uses containers for almost everything. Just wonderful work by Intel’s team and contributors.
One thing to check: are you using iterm? iterm, by default (or at least this used to be the default; not sure if they fixed it) keeps indefinite scrollback history, in memory. It's perfectly possible for this to consume all your memory. It's worth checking memory usage in activity monitor, anyway.
That said, "terrible corporate spyware" would be my first guess here.
Same - I've used multiple ARM Macs and they just freeze up for seconds sometimes. At this point I'm pretty sure its related to their swapping behavior.
The OS really has zero hesitation to swap to sometimes insane amounts and by the time you feel it in the UI responsiveness you might already be swapping 10-20GB.
8GB is just too little for the base model. From what I've seen the OS alone will use 20-50% of that, and if I do anything productive with a couple applications and browser tabs I'm soon at 100% memory usage. Which can cause lags or short freezes when switching between applications. But on 16GB it's a very different experience, it gives decent breathing room.
I think Apple really shot themselves in the foot with this, anyone using even half the CPU's potential is guaranteed to have significantly degraded performance and a worse experience.
The processes of training and running use similar low-level operations, but training obviously requires a lot more of them. So it’s relevant to both, but one is much more computation-heavy to begin with.
The point is that at the moment only a few of us are interested in training our own models, but this will change gradually as Apple introduces more machine learning models trained on our own local data behind the scenes. At that point, Macs with less support for this will start to slow down (well, or they’ll just not support those features). And the M1 chip will be worse in this regard than the M2.
As long as you don't mind your machine using lots of power when plugged in overnight (and training then), the old machines should be able to keep up for quite a while?
I'd love to see training working on arm + mps macs. Hopefully someone knowledeable chimes in, that it's actually possible.
I tried running some mpx supported conda/pytorch inference (whisper to be precise) on a workplace issued M1 mac and it did fall back to cpu. I encountered:
- `NotImplementedError: The operator 'aten::_fft_r2c' is not currently implemented for the MPS device`
- `TypeError: Trying to convert ComplexFloat to the MPS backend but it does not have support for that dtype`.
EDIT: just read up it might have more to do with audio processing.
Fingers crossed for better mac support in ML in general! Competition is a great thing
Yes, and the article writer is suggesting that more AI training will be done on the client computer, and gave spelling and grammar checking as an example.
I don't know much about AI and stuff, but I know Apple's been doing machine learning / image recognition and the like for a long time now, so that the Photos app on your phone can recognize the same people in photos or categorise them - all done on device instead of uploading your photos.
I think smaller models will keep getting better. For privacy and economic reasons, it makes sense to do as much AI on-device as possible.