Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Running 32b on what hardware?


I'm running it on a pure cpu 2020 model ryzen server with 2x128 GB RAM with llama.cpp, it seems as intelligent as gpt4. I optimized a little bit by forcing it to run on a single RAM stick and tuning llama.cpp build parameters, going from 3-5 tok/s to a more acceptable 5-8 tok/s.

It can call tools and reason adequately enough to use them when appropriate.


Does keeping it on one stick make it more performant? I have a epyc server with 1tb of 64 gig sticks and a 3060, looking to maximize what models I can run on it


In my case it definitely helps, but keeping 64k context window would probably require more than 64 GB.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: