Hacker News new | past | comments | ask | show | jobs | submit login

I can run a certain 120b on my M3 max with 128GB memory. However I found that while it “fits” Q5 was extremely slow. The story was different with Q4 though which ran just fine around ~3.5-4 t/s.

Now this model is ~134B right? It could be bog slow but on the other hand its a MoE so there might be a chance it could have satisfactory results.




From the article, should have the speed of a ~36b.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: