You can run models the size of this one locally, even on a laptop, it's just not a great experience compared with an optimised cloud service. But it is local.
The size in bytes of this 120B model is about 65 GB according to the screenshot, and elsewhere it's said to be trained in FP4, which matches.
That makes this model small enough to run locally on some laptops without reading from SSD.
The Apple M2 Max 96GB from January 2023, which is two generations old now, has enough GPU-capable RAM to handle it, albeit slowly. Any PC with 96 GB of RAM can run it on the CPU, probably more slowly. Even a PC with less than 64 GB of RAM can run it but it will be much slower due to having to read from the SSD constantly.
If it's a 20B MoE, it will read about one fifth of the data per token, making it about 5x faster than a 120B FP4 non-MoE would be, but it still needs all the data readily available for multiple tokens.
Alternatively, someone can distill and/or quantize the model themselves to make a smaller model. These things can be done locally, even on a CPU if necessary if you don't mind how long it takes to produce the smaller model. Or on a cloud machine rented long enough to make the smaller model, which you can then run locally.
> is it possible that at some point the computational photography makes a mistake and changes text?
Yes it is. I've seen that happen in real-time with the built-in camera viewfinder (not even taking a photo) on my mid-range Samsung phone, when I zoomed in on a sign.
It only changed one letter, and it was more like a strange optical warping from one letter to a different one when I pointed the camera at a particular sign in a shop, but it was very surprising to see.
You can recreate the instance every 60 minutes, I've tried such approach once. But such setup is useless anyway, most services block datacenter traffic by default.
Unfortunately, I think sync_file_range() provides much weaker guarantees than byte-range fsync() and even byte-range fdatasync().
As I understand it from historical behaviour and documentation, sync_file_range() doesn't push durability barriers down the underlying storage devices, nor does it ensure that all metadata needed to access the written pages is itself written and made durable, for example when writing to a hole in a sparse file, to the end-hole created by enlarging a file with ftruncate(), or to fallocate'd pages.
As a result, that means sync_file_range() can only be used as a performance tweak, and not for any durability guarantees that fdatasync() / fsync() are used for.
I'd be delighted to find this has improved since I last looked, but that's what I recall about sync_file_range().
That CLA is a curious take, as the old, avowedly non-commercial, GNU foundational tools such as the ones we're talking about like "ls" in coreutils, have always required a kind of strong CLA to be signed from the very beginning, even when they wee new, nimble and fun to contribute to.
Their kind of CLA was designed to uphold community and openness values more strongly than GPL alone, by helping GNU to pursue GPL violators through the law, to fource vendors to release source code when GPL code was shipped in products..
So I've never understood the blanket "don't like regardless of what it says" attitude to CLAs and such.
Surely it should depend on what the CLA says?
Some people object to CLAs that grant upstream less rights than BSD/MIT/Apache licenses grant upstream by defaut. ("No way, the CLA lets them make. a private, commercial fork of my code!"). Yet the same people contribute enthusiastically to BSD/MIT/Apache projects with exactly the same upstream property ("I don't mind that the license let's them make a private, commercial fork of my code!").
With 1000s of bots per month and 10,000 hits on an ecommerce site, with product images, that's a lot of data transfer, and a lot of compute if your site has badly designed or no caching, rendering all the same page components millions of times over. But...
Part of the problem is all those companies who use AWS "standard practice" services, who assume the cost of bandwidth is just what AWS charges, and compute-per-page is just what it is, and don't even optimise those (e.g. S3/EC2/Lambda instead of CloudFront).
I've just compared AWS egress charge against the best I can trivially get at Hetzner (small cloud VMs for bulk serving https cache).
You get an astonishing 392x(!) more HTTPS egress from Hetzner for the same price, or equivalently 392x cheaper for the same amount.
You can comfortably serve 100+ TB/month that way. With 10,000 pages times 1000 bots per month, that gives you 10MB per page, which is more than almost any eCommerce site uses, when you factor that bots (other than very badly coded bots) won't fetch the common resources (JS etc.) repeatedly for each page, only the unique elements (e.g. HTML and per-product images).
I've had the version of that where I called my bank's listed number to confirm the incoming "call us on this number" voicemail was legit, and they said NO, the call is not a legit number of theirs, the account looks fine, I was right to check, and they agreed it seemed like a scam call.
A few days later I found out the call really was from the bank, and the bank had blocked my account, in a way that took a long time to unblock (don't get me started...). As ever, I found out the hard way, when I needed to use the account for something in real-time and it wasn't available.
But the call was from a different department than general customer support, the department's number wasn't known to customer service, and the account status change wasn't visible to customer service either.
So the bank's own customer service thought it was a scam call!
Name and shame whoever did that. The last time that a bank tried to pull such shit at me I wrote about it all over the Internet and to this day it comes up when you search for that bank (either my post or others complaining/warning others of the same problem).
The size in bytes of this 120B model is about 65 GB according to the screenshot, and elsewhere it's said to be trained in FP4, which matches.
That makes this model small enough to run locally on some laptops without reading from SSD.
The Apple M2 Max 96GB from January 2023, which is two generations old now, has enough GPU-capable RAM to handle it, albeit slowly. Any PC with 96 GB of RAM can run it on the CPU, probably more slowly. Even a PC with less than 64 GB of RAM can run it but it will be much slower due to having to read from the SSD constantly.
If it's a 20B MoE, it will read about one fifth of the data per token, making it about 5x faster than a 120B FP4 non-MoE would be, but it still needs all the data readily available for multiple tokens.
Alternatively, someone can distill and/or quantize the model themselves to make a smaller model. These things can be done locally, even on a CPU if necessary if you don't mind how long it takes to produce the smaller model. Or on a cloud machine rented long enough to make the smaller model, which you can then run locally.
reply