So, what's the question if the M1/M2 Ultra was limited by GPU/NPU or more memory bandwidth at this point?
I'm curious what instruction sets may have been included with the M3 chip that the other two lack for AI.
So far the candidates seem to be NVIDIA digits, Framework Desktop, M1 64gb M2/M3 128gb studio/ultra.
The GPU market isn't competitive enough for the amount of VRAM needed. I was hoping for an Battlemage GPU Model with 24GB that would be reasonably priced and available.
The framework desktop and devices I think a second generation will be significantly better than what's currently on offer today. Rationale below...
For a max spec processor with ram at $2,000, this seems like a decent deal given today's market. However, this might age very fast for three reasons.
Reason 1: LPDDR6 may debut in the next year or two this could bring massive improvements to memory bandwidth and capacity for soldered on memory.
LPDDR6 vs LPDDR5 - Data bus width - 24 bits, 16 bits Burst length - 24 bits, 15 bits Memory bandwidth - Up to 38.4 GB/s, Up to 6.7 GB/s
- Camm ram may or may not be maintain signal integrity as memory bandwidth increases. Until I see it implemented for a AI use-case in a cost-effective manner, I am skeptical.
Reason 2: - It's a laptop chip with limited PCI lanes and reduced power envelope. Theoretically, a desktop chip could have better performance, more lanes, socketable (Although, I don't think I've seen a socketed CPU with soldered RAM)
Reason 3: In addition, what does hardware look like being repurposed in the future compared to alternatives?
- Unlike desktop or server counterparts which can have a higher cpu core count, PCEe/IO Expansion, this processor with its motherboard is limited on re-purposing later down the line as a server to self-host other software besides AI. I suppose could be turned into a overkill, NAS with ZFS and HBA Single Controller Card in new case.
- Buying into the framework desktop is pretty limited based on the form factor. Next generation might be able to include a 16x slot fully populated, a 10G nic. That seems about it if they're going to maintain the backward compatibility philosophy given the case form factor.
I love seeing tools like this. I could see a light LLM for classifying elements augmented by Voice Recognition for Accessibility. Natural language will never be a great interface for High Domain, Low latency use case such as accessibility.
Quick clarification. Low domain knowledge is okay for those who don't have experience and don't know what to say, like Alexa. High domain is somebody who has expertise with a specialized workflow.
So, they will rely on voice commands for recognition, not natural language. Often one to two words to set a chain of tasks in motion. Think of having to control your entire computer, including navigating by voice. That would be very exhausting and inefficient through natural language. There needs to be a hybrid solution that can leverage low domain natural language, but also high domain command-based recognition. I cannot overstate how important of low latency between the beginning of a command and an action produced. High latency means a big cognitive load and not to mention just inefficiency.
There's a lot of overlap between UI automation and accessibility control tools. However, UIA automation has always been a slow process simply because the stack doesn't have the demand from devs for low latency.
It's a difference between having an independent agent do something on your behalf, not caring how long it takes, versus you waiting for a aynchronous task to be completed.
appreciate clarification. The low domain vs high domain distinction is spot-on:Latency kills expert workflows. keeping this in mind when integrating/designing voice recognition and more accessibility control options.
I'm pretty torn to self-host AI 70 B models on Ryzen AI Max with 128gb of ram. The market seems to be evolving fast. Outside of Apple, this is the first product to really compete in this category Self-host AI. So... I think a second generation will be significantly better than what's currently on offer today. Rationale below...
For a max spec processor with ram at $2,000, this seems like a decent deal given today's market. However, this might age very fast for three reasons.
Reason 1: LPDDR6 may debut in the next year or two this could bring massive improvements to memory bandwidth and capacity for soldered on memory.
LPDDR6 vs LPDDR5 - Data bus width - 24 bits, 16 bits Burst length - 24 bits, 15 bits Memory bandwidth - Up to 38.4 GB/s, Up to 6.7 GB/s
- Camm ram may or may not be maintain signal integrity as memory bandwidth increases. Until I see it implemented for a AI use-case in a cost-effective manner, I am skeptical.
Reason 2: - It's a laptop chip with limited PCI lanes and reduced power envelope. Theoretically, a desktop chip could have better performance, more lanes, socketable (Although, I don't think I've seen a socketed CPU with soldered RAM)
Reason 3: In addition, what does hardware look like being repurposed in the future compared to alternatives?
- Unlike desktop or server counterparts which can have a higher cpu core count, PCEe/IO Expansion, this processor with its motherboard is limited on re-purposing later down the line as a server to self-host other software besides AI. I suppose could be turned into a overkill, NAS with ZFS and HBA Single Controller Card in new case.
- Buying into the framework desktop is pretty limited based on the form factor. Next generation might be able to include a 16x slot fully populated, a 10G nic. That seems about it if they're going to maintain the backward compatibility philosophy given the case form factor.
I wonder if a wire connection (ethernet/fiber) will have limited use in residential. Speed isn't so much concern although a wired connection still better in terms of reliability and backwards compatibility.
I say this as somebody that's about to do Ethernet runs and fiber in the house.
Behind every good wireless network, there is a fantastic wired backbone. I have access points in every room that comes out of a wired ethernet network. My daughter stopped complaining about latency with her games the day she learned that it is better to stay wired with her computer.
Wireless is a convenience, but I’d always prefer wired connections wherever I can.
My house is wired for ethernet, and I don't use all the jacks. Best practice is probably run two lines to two opposite walls to most rooms. But you'd expect to only really use one of those for most rooms. It's just nice to have options for placing wired devices (which is why you want wired to opposite walls) and to have a spare in case it fails.
1gbps at low latency and low jitter is table stakes for wired networking, but pretty difficult for wifi (regardless of the sticker bandwidths). Wifi 7 will likely do a lot better than wifi 6 on well equiped clients and access points because of aggregation across frequencies, but it's still going to be not entirely consistent, and it still won't hold a candle to 10g-baseT which isn't too expensive if you only need a few ports and you shop well on ebay.
If you do fiber runs, there's lots of used stuff much faster than 10g for not much more. 40gbe seems to have had a short lifetime in enterprise so there's a lot of decommisioned hardware out there, if you've got fiber to run it on.
Gaming? Maybe, since some gamers are apparently willing to spend extra money on 5000hz mouse to shave <1ms of latency over a "normal" 1000hz gaming mouse. WFH? Probably not. Wifi adds maybe 5ms of avg latency with occasional 100ms spikes. Most video conferencing software introduce enough of a lag (through buffering, encoding, echo cancellation, noise suppression) that you won't notice the difference. Same goes for bandwidth. Video calls with screen sharing barely goes over 5Mb/s for me. You don't need fancy 10Gbe hardwired connection for that.
It’s 5ms of average latency - under optimal conditions! Wifi latency can range anywhere from 5-50ms. Not everyone’s in a single family home, where there’s little interference. :)
Wifi 7 is nice in that regard because the 6ghz band is barely used these days.
I think ethernet is a nice to have, certainly not required.
I joke that I prefer Ethernet because I know how WiFi works.
Generally, WiFi is fine if you have line of sight or close to it. And you don't have strong demands on total bandwidth (not doing large local file transfers).
Reliability and speed, especially if the airwaves are congested around you, plus simplicity in that there's less configuration than Wifi, and for some IoT-y devices like cameras you can use PoE and not have to worry about powering them either
Stabile, high bandwidth connections for wireless APs, Home A/V, IP cameras (wireless is typically junk in this category) & other PoE IoT devices. Wired connections are still incredibly useful.
It’s hard to see the value of a wired network, but there are about a million. As other commenters have pointed out, most of the scenarios are more than one access point and thicker walls / building materials.
You don't even necessarily need to run Ethernet. For example, my parents house was built without Ethernet cable runs to every floor, but they ran coax cables to basically every room for TV.
Thanks to MoCA (Media over Coax Alliance) you can route your Ethernet over Coax, and thus we have a really stable and performant (> 1GbE) Ethernet connection to every room.
Wireless repeaters/mesh doesn't really work well when you have reinforced steel-concrete floors and walls.
I'm pretty torn when to self-host AI 70 B models. The market seems to be evolving fast. Outside of Apple, this is the first product to really compete in this category Self-host AI. So... I think a second generation will be significantly better than what's currently on offer today. Rationale below...
For a max spec processor with ram at $2,000, this seems like a decent deal given today's market. However, this might age very fast for three reasons.
Reason 1: LPDDR6 may debut in the next year or two this could bring massive improvements to memory bandwidth and capacity for soldered on memory.
LPDDR6 vs LPDDR5 - Data bus width - 24 bits, 16 bits Burst length - 24 bits, 15 bits Memory bandwidth - Up to 38.4 GB/s, Up to 6.7 GB/s
- Camm ram may or may not be maintain signal integrity as memory bandwidth increases. Until I see it implemented for a AI use-case in a cost-effective manner, I am skeptical.
Reason 2: - It's a laptop chip with limited PCI lanes and reduced power envelope. Theoretically, a desktop chip could have better performance, more lanes, socketable (Although, I don't think I've seen a socketed CPU with soldered RAM)
Reason 3: In addition, what does hardware look like being repurposed in the future compared to alternatives?
- Unlike desktop or server counterparts which can have a higher cpu core count, PCEe/IO Expansion, this processor with its motherboard is limited on re-purposing later down the line as a server to self-host other software besides AI. I suppose could be turned into a overkill, NAS with ZFS and HBA Single Controller Card in new case.
- Buying into the framework desktop is pretty limited based on the form factor. Next generation might be able to include a 16x slot fully populated, a 10G nic. That seems about it if they're going to maintain the backward compatibility philosophy given the case form factor.
> Linux distributions such as Ubuntu [4] and Debian [5] provide more than one Python version at the same time to their users. For example, Ubuntu 9.10 Karmic Koala users can install Python 2.5, 2.6, and 3.1, with Python 2.6 being the default. [...]
> Because these distributions cannot share pyc files, elaborate mechanisms have been developed to put the resulting pyc files in non-shared locations while the source code is still shared. Examples include the symlink-based Debian regimes python-support [8] and python-central [9]. These approaches make for much more complicated, fragile, inscrutable, and fragmented policies for delivering Python applications to a wide range of users. Arguably more users get Python from their operating system vendor than from upstream tarballs. Thus, solving this pyc sharing problem for CPython is a high priority for such vendors.
> This PEP proposes a solution to this problem.
> Proposal: Python’s import machinery is extended to write and search for byte code cache files in a single directory inside every Python package directory. This directory will be called __pycache__.
Should the package management tool also install multiple versions of the interpreter? conda, mamba, pixi, and uv do. Neither tox nor nox nor pytest care where the python install came from.
And then of course cibuildwheel builds binary wheels for Win/Mac/Lin and manylinux wheels for libc and/or musl libc. repairwheel, auditwheel, delocate, and delvewheel bundle shared library dependencies (.so and DLL) into the wheel, which is a .zip file with a .whl extension and a declarative manifest that doesn't require python code to run as the package installer.
> It includes pure-python replacements for external tools like patchelf, otool, install_name_tool, and codesign, so no non-python dependencies are required.
The launcher first appeared with Python 3.3. There was nothing like the python standalone build project at that point, and `venv` had also just been added to the standard library in the same version.
But it could and should have been redesigned at some point to include those kinds of things, sure. And the second best time to plant a tree is now.
Installations could be managed better on Windows, too. I can envision a cross-platform (rather, made separately available for each major platform) `py` tool which, on Windows, would set up Program Files with the `py` executable in an overall folder (added to PATH), and then subfolders where the actual Python version installations go. NTFS does support hard links and symbolic links (https://en.wikipedia.org/wiki/NTFS_links), so it could conceivably make the executables available directly, too. Then perhaps there could be a `py get` command to grab and run the installer for another version.
On Linux, of course, much the same, but using `/usr/{bin,lib}` in normal ways. Or perhaps `/usr/local/{bin,lib}`. (And the system would come with `py` as well as a specific Python version, and system scripts would have a `py -3.x` shebang.)
In general, it is impossible to tell what a computer program may do even if you can inspect the source code. That’s a generalization of the halting problem.
That’s not correct. There is not a general solution to tell what any arbitrary program can do, but most code is boring stuff that is easy to reason about.
But a malicious actor can hide stuff that would be missed on a general casual inspection.
Most of the methods in https://www.ioccc.org/ would be missed via a casual code inspection, esp. if there weren't any initial suspicion that something is wrong about it.
Yes, but again, that doesn’t apply to the vast majority of code. My point is that saying you can’t know what code does “in general” is not true. We wouldn’t have code reviews if that were the case.
My one annoyance with RustDesk they haven't implemented auto update. Non-techie users don't want to go download a binary. Those that deploy software don't want to deal with at scale unless without the proper tooling. Good SAS candidate.
Even when you manually download and update, you still have to start the service.
I'm curious what instruction sets may have been included with the M3 chip that the other two lack for AI.
So far the candidates seem to be NVIDIA digits, Framework Desktop, M1 64gb M2/M3 128gb studio/ultra.
The GPU market isn't competitive enough for the amount of VRAM needed. I was hoping for an Battlemage GPU Model with 24GB that would be reasonably priced and available.
The framework desktop and devices I think a second generation will be significantly better than what's currently on offer today. Rationale below...
For a max spec processor with ram at $2,000, this seems like a decent deal given today's market. However, this might age very fast for three reasons.
Reason 1: LPDDR6 may debut in the next year or two this could bring massive improvements to memory bandwidth and capacity for soldered on memory.
LPDDR6 vs LPDDR5 - Data bus width - 24 bits, 16 bits Burst length - 24 bits, 15 bits Memory bandwidth - Up to 38.4 GB/s, Up to 6.7 GB/s
- Camm ram may or may not be maintain signal integrity as memory bandwidth increases. Until I see it implemented for a AI use-case in a cost-effective manner, I am skeptical.
Reason 2: - It's a laptop chip with limited PCI lanes and reduced power envelope. Theoretically, a desktop chip could have better performance, more lanes, socketable (Although, I don't think I've seen a socketed CPU with soldered RAM)
Reason 3: In addition, what does hardware look like being repurposed in the future compared to alternatives?
- Unlike desktop or server counterparts which can have a higher cpu core count, PCEe/IO Expansion, this processor with its motherboard is limited on re-purposing later down the line as a server to self-host other software besides AI. I suppose could be turned into a overkill, NAS with ZFS and HBA Single Controller Card in new case.
- Buying into the framework desktop is pretty limited based on the form factor. Next generation might be able to include a 16x slot fully populated, a 10G nic. That seems about it if they're going to maintain the backward compatibility philosophy given the case form factor.