Any example of this being used in the wild? I feel like it could save with backend costs (especially for hobby projects) but the bandwidth/loading time might make it infeasible?
Also if you have any novel model it will be trivial to reverse engineer it. You gotta send the weights over to the client and they can just run tensorflow.js themselves right?
I used it in the wild for some week-end project [1][2]. To get around the bandwidth issue, I used it as a browser extension. This means that weights are downloaded and installed only once during the setup. But I also used it in the past in a website for real-time audio processing when model are small enough [3].
Novel models will probably need to have some custom layers, which are quite painful to write. The weights for the web version will probably be a low quality of what you achieve on a more powerful machine. And you don't need to provide the code for the training, nor the dataset used, so you will still have an edge over people copying you.
The main problem I have with tensorflow.js is that it's not production ready yet. It's probably 10 years ahead of it's time. Most people don't have recent GPU or recent video drivers. As it's doing some tricks with the GPU to have some acceleration, a fraction of your clients will encounter random bugs and crashes. I even got some bad review for causing reboots on Firefox :). For some people it will be so slow it will feel unresponsive, and others will find that it's unimpressive because you would have to use the minimal model. In a day and age where you need 99% satisfaction to exist, picking tensorflow.js is a mistake.
I was playing with tf.js last year and there were some core components still being converted to WASM, which gave me some big performance boosts once I switched. I had some problems with WebGL-related bugs too. It seems that they're currently in the process of writing WebGPU-based back-ends. I'm hoping that when WebGPU is on-by-default in most browsers, and there's been a year or two to sort out all the major teething problems, we'll see some wide-spread uptake.
Internet speeds are still increasing exponentially, so hopefully the model size problems will become less of an issue - perhaps aided by some CDN-served (and thus cached across domains) "base" models that are fine-tuned with some parameters downloaded from the server. I think I could start playing with it seriously if I could get the model sizes under 30mb or so. In a few years (with increasing internet speeds) that might be 50mb. I think huggingface's distilled GPT-2 model is a couple of hundred megabytes, for reference, so we're certainly not going to be doing anything revolutionary in the browser, but I have a bunch of neat little ideas that I think would be useful.
For bringing the "big" stuff to the web we're probably stuck with APIs, like OpenAI's new GPT-3 offering. I think access to SOTA models on hard problems is going to be almost exclusively via APIs for the foreseeable future.
Yes, the idea is that the model is local to the user and runs in their browser (pushing the computation to the edge). Here's a project demonstrating pose estimation from Twilio's engineering blog: https://www.twilio.com/blog/pose-detection-video-tensorflow-....
Re: bandwidth. Do you mean the weights? If so then yes, you'd have to think about distilling large models to smaller ones. There are techniques for doing this. Here is an explanation of distillation from Floyd Hub's engineering blog: https://blog.floydhub.com/knowledge-distillation/.
Also if you have any novel model it will be trivial to reverse engineer it. You gotta send the weights over to the client and they can just run tensorflow.js themselves right?