I was so excited until I saw that it's CPU based only. Would you happen to know ... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		syntaxing on May 10, 2023 \| parent \| context \| favorite \| on: Hugging Face Releases Agents I was so excited until I saw that it's CPU based only. Would you happen to know of any alternative for GPU support, particularly GPTQ models? Edit: I think textgen itself can support this nowadays

anton5mith2 on May 11, 2023 [–]

For now it is CPU only yes, uses AVX instructions. But it's pretty fast anyway, try it out. I have it running on my mbp M1 and it's pretty decent. I think GPU support will come eventually. I wrote an app that uses the openai API and it was nice and simple to just point it at my own local service instead.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact