Growing up in Moore, Oklahoma in the 1960s and 1970s, my mother frequented the C.R. Anthony's store in the City of Moore Shopping Center. They used pneumatic tubes; the person at the check-out counter would put a receipt and money tendered into a tube and off it would go to somewhere, I figured some office in back, where they made change and sent the tube back. You could hear it rattle around a bit as it went on its way, and the way it would come back, zip out onto a open curved "landing strip" and slam into the stop was impressive.
Depends on how many experts are active in any given pass. If it's a 10 expert mix of 33B experts (grok-0 is 33B, grok-1 is ~314B which is ~10x) and only runs two of them (like Mixtral's 2/8) then it'd have about the same inference requirements as a 70B model (2*33=66B).
So if this was quantized using ~4 bits per parameter you'd need ~40GB of vram. So you could spread it across 2x 3090 24GB using llama.cpp.
MoE has the same “loading” RAM requirements as any other model with the same total parameters (not just for the fixed portion plus whatever experts are activated at any one time) because it has to load all the parameters. The additional needed because of context may be lower (not sure), but the big difference is that it has much better inference speed (and, as a result, can be tolerable with layers split between VRAM and system RAM where a similarly-sized non-MoE model would not.)
> So if this was quantized using ~4 bits per parameter you’d need ~40GB of vram.
No, Mixtral 8x7B (which is a total of 45 billion parameters, because there is a shared portion of the 7B, so its not 56 billion) at 4-bit quantization takes ~29GB [0]. A 314B model is ~7 times as large; with a similar architecture its not going to take only another 1/3 as much RAM.
Just to load the model without actually running it requires 1GB of whatever RAM it is loading and running in (could be VRAM, system RAM, or a combination, with different performance characteristics for each option) per billion parameters at 8-bit quantization. Though models often are usefully run at 4-5 bit quantization, which saves half (or nearly so) of that.
You also need additional RAM that increases as some function of context size (not sure what function, and ISTR there are big-O differences between architectures in how it varies) to actually do inference.
I can only speak for myself, but Python struck me as much simpler to learn and understand than Perl. Also, my hatred for sigils burns with the heat of a million suns, going back to the $ suffix for string variables and functions in BASIC.
I'm curious. What is "accolade-free code"? It's referenced and highlighted as if it had a link but it doesn't, and Google appears to think it's a hapax legomenon (or whatever the equivalent is for phrases instead of words).
I think I confused the translation here. I'm Romanian where the {curly brackets} are called "acolade". I read the word "accolade" in English a few times and wrongly assumed it means the same thing as in Romanian.
