For anyone confused about the project, it is using whisper.cpp, a C-based runner and translation of the open whisper model from OpenAI. It is built by the team behind GGML and llama.cpp. https://github.com/ggerganov
You can fork this code, run it on your own server, and hit the API. The server itself will use FFmpeg to convert the audio file into the required format and run the C translation of the whisper model against the file.
By doing this you can separate yourself from the requirement of paying the fee that OpenAI charges for their Whisper service and fully own your translations. The models that the author has supplied here are rather small but should run decent on a CPU. If you want to go to larger model sizes you would likely need to change the compilation options and use a server with a GPU.
Similar to this project, my product https://superwhisper.com is using these whisper.cpp models to provide really good Dictation on macOS.
Its runs really fast on the M series chips. Most of this message was dictated using superwhisper.
Congrats to the author of this project. Seems like a useful implementation of the whisper.cpp project.
I wonder if they would accept it upstream in the examples.
I haven’t used faster-whisper so I can’t compare performance, but whisper.cpp does support cuda via CUBLAS, and it’s noticeably faster than the cpu version. I used it earlier this year to generate subtitles for 6 seasons of an old tv show I backed up from dvd that didn’t include subtitles on the disc.
Fwiw decent acceleration works on any avx2 compatible chipset. I get realtime speed for everything but the large models with a recent Ryzen system. The apple silicon is good but not as special as folks think!
Many of you are asking if the project is completely self-hosted and does not rely on any third-party services. Yes, it is completely self-hosted and does not rely on any third-party services. The user is for authentication, so no one can use the service without authentication.
Getting an authentication token does rely on a third-party service, if the README instructions are correct. It requires sending an email address to that third party.
This open source project provides a self-hostable API for speech to text transcription using a finetuned Whisper ASR model. The API allows you to easily convert audio files to text through HTTP requests. Ideal for adding speech recognition capabilities to your applications.
Key features:
- Uses a finetuned Whisper model for accurate speech recognition
- Simple HTTP API for audio file transcription
- User level access with API keys for managing usage
- Self-hostable code for your own speech transcription service
- Quantized model optimization for fast and efficient inference
- Open source implementation for customization and transparency
Are you able to provide more information on the fine tuning? Any improvement in WER and what language it was fine tuned in and the size of the dataset used?
It is surprisingly good!
Not clear if you can go without a manual step where you need to change the likes of SPEAKER_00 to Bob and SPEAKER_01 to Sarah, but I've not had it mess up on me at all transcribing 2 hour long conversations between 6 people.
It is completely self-hosted, but it currently supports only the tiny and base models. You can soon expect support for large models. For any requests, you can create an issue.
Nice! This will be very useful for me. Think I can run this locally can spin a basic telegram bot around it for personal use.
One issue I faced with all the whisper based transcript generators is that there seems to be no good way to make editing/correcting the generated text with word level timestamp. I created a small web based tool[0] for that.
By any chance if anyone is looking to edit transcripts generated using whisper, you'd probably find it useful.
So is "real time" translation a thing yet? I've long wanted to be able watch non-english television and have the audio translated into English subtitles. It's doable for pre-recorded things, but not for live.
An iPhone app that could do this from the microphone would also be amazing. Google Translate and it's various competitors from Microsoft/Apple are nearly there, but they all stop listening inbetween sentences. Something that just listened constantly, printing translated text onto the screen, would be amazing.
Just wait for a couple of weeks. I am working on speech-to-speech translation. Instead of subtitles, you can listen to it directly. I am also working on subtitles.
No, it is not a third-party. It is a just PostgreSQL database for logging everything. You can simply visit the /docs endpoint. It is just for authentication so that you can work with different users.
One Again its completely self hosted
Many live streamers, and platforms, would love to have custom real-time transcription elements. I actually looked into this exact project of yours when I thought about creating such a thing.
Even if it meant delaying the broadcast for a second while transcribing the accessibility value could be immense.
getToken is just an authentication layer for authenticating your request. If you want to self-host it, just clone the repo and please check the .env.example file.
I think people are confused by the README showing localhost in some places and your instance url in others, causing them to assume you were centrally issuing tokens for using this project.
I guess I should have phrased it better: If you were building an app, does Android provide the ability to listen to arbitrary speech and convert it into text for your app?
For anyone confused about the project, it is using whisper.cpp, a C-based runner and translation of the open whisper model from OpenAI. It is built by the team behind GGML and llama.cpp. https://github.com/ggerganov
You can fork this code, run it on your own server, and hit the API. The server itself will use FFmpeg to convert the audio file into the required format and run the C translation of the whisper model against the file.
By doing this you can separate yourself from the requirement of paying the fee that OpenAI charges for their Whisper service and fully own your translations. The models that the author has supplied here are rather small but should run decent on a CPU. If you want to go to larger model sizes you would likely need to change the compilation options and use a server with a GPU.
Similar to this project, my product https://superwhisper.com is using these whisper.cpp models to provide really good Dictation on macOS.
Its runs really fast on the M series chips. Most of this message was dictated using superwhisper.
Congrats to the author of this project. Seems like a useful implementation of the whisper.cpp project.
I wonder if they would accept it upstream in the examples.