how's the generation speed on CPU? | Hacker News

Hacker News new | past | comments | ask | show | jobs | submit

login

FrozenSynapse on Aug 25, 2023 | parent | context | favorite | on: Code Llama, a state-of-the-art large language mode...

how's the generation speed on CPU?

selfhoster11 on Aug 25, 2023 [–]

On Ryzen 5600X, 7B and 13B run quite fast. Off the top of my head, pure CPU performance is about 25% slower than with an NVIDIA GPU of some kind. I don't remember the numbers off the top of my head, but the generation speed only starts to get annoying for 33B+ models.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact