I don't think (some) people understand; a slick data annotation tool like this is vastly more useful than the 20th variant of GAN that DeepMind produces :)
Totally, I think people have this weird sense of entitlement when it comes to high-quality datasets without the commensurate respect for how they're created or the level of effort that goes into them.
Fei-Fei Li gives a good sense for this in her history of ImageNet [1][2].
Looks promising and definitely a needed tool. I signed up for the beta and I used the demo version and have a couple of thoughts.
1. This seems closer to a reinforcement learning system than a pure annotation system. That seems to be by design, however based on the demo, I am not able to change or add to the annotations as I go, which is a big limitation. It's just yes, no (no feedback), ignore and undo. This is in contrast to something like the VGG annotations system: http://www.robots.ox.ac.uk/~vgg/software/via/via.html
2. I don't see an actual annotations capability for images in the demo. Not sure if that is just a pretotype page, but IMO image classification/segmentation is the place where this tool would really benefit the community.
3. It's unclear to me how or if I retrieve my trained model or even just the annotated structure (.csv?, .json?) from this system. Do I get a .pb somehow that I can import into TF or am I locked into an API with my new model served from Prodigy? My guess would be the latter.
I think what this wants to be is a human validation system for training, which also improves the Prodigy nets through crowd sourcing. Definitely a win-win in the short term, but it has the limitations of the initial model and the ability for the user/client to tweak the system and output the results.
Matroid is doing something similar here, but I have been unimpressed with their offering so far.
Thanks for the engaging questions! Reading between the lines, I think there's an important point that hasn't come across. Prodigy isn't SaaS --- it's a library you download and run. You can extend and customise every aspect of it, and there's definitely no lock-in. The model (and annotations) never have to leave your servers.
For the specific questions:
1. The built-in web views all have binary annotation interfaces. This is more of a design choice than a fundamental limitation, and the front-end is extensible --- you can add your own web views if you need to.
The binary interface is sort of a position statement. We think this is The Way, so we want you to try it. We'll have more input components in future, but at the start we want to guide people towards the intended workflow.
3. You can usually get some accuracy improvement by retraining once all the annotations are available. I've not found a streaming SGD algorithm that works as well as the simple iterate-and-shuffle batch process. Batch training also lets you tune the hyper-paramers. You can read more about this here: https://prodi.gy/docs/workflow-named-entity-recognition#trai...
I would suggest writing a Prodigy recipe to do the batch training. That way you can pass in the dataset ID, instead of exporting the annotations. There's no problem with exporting the annotations and running a script, though. Again --- it's all on your computer. You can run it however you like.
I love how the SpaCy related websites are always so well designed. Their dependency graph visualizer is just amazing. I know that Ines is behind that one, but don't know about the other stuff.
Now coming back to the topic, I have so far just used Jupyter Notebooks and spreadsheets to do annotations and by golly, it is an extremely boring and tedious process. This looks like a fun tool to try out for my next NLP related project. Might spice things up!
But I hope that like all SpaCy related ideas, it doesn't assume too much about the problem at hand. I usually use NLTK instead of SpaCy because it allows me to be very flexible, except for the sentence tokenizer, where SpaCy's accuracy is hard to beat.
Explosion is just me and Ines -- so yes, the pages are all made by Ines (with the great illustrations by Frederique Matti). Ines also wrote the bulk of the code for Prodigy itself.
Anecdotally, I got better NER results with spaCY than OpenNLP and CoreNLP with their respective default models, and spaCY was easier to install (though I'm biased, being more familiar with Python tooling and documentation style). I was eventually implementing in Java so I did use OpenNLP for sentence splitting, but I retrained the NER with data bootstrapped from spaCY, in a way similar to what the Prodigy tool is aiming to facilitate, by first classifying using the default/vanilla model and then manually correcting labels where they were incorrect.
They have some hard comparisons in some of their earlier blog posts on how spacy compares to the other popular open source NLP libraries. In my experience it has been much easier to use and faster than things like Stanford's library or NLTK. In general it's aimed at production or commercial use, whereas the other libraries I typically hear mentioned are aimed at a more academic audience.
OpenNLP is.... well I've never heard of anyone using it (except once in a ensemble). I think NLTK is more widely used.
Stanford CoreNLP give good accuracy and is pretty much the benchmark in English for accuracy. BUT it isn't great software. It falls over if you pass large amounts of text to it, the code is dreadful, it's hard to integrate (even in Java because of its own wacky config system), various parts aren't integrated (eg, SUTime), it doesn't have an embedding representation and it is pretty slow.
Having said all that I still use it sometimes. But Spacy is much nicer to use, and 99% (probably more) of the time the slightly lower accuracy is offset by things like the easy availability of word embedding right with the word tags.
I think it's pretty fair to say Spacy is the leading open-source NLP tool.
I have to agree. Explosion/Spacy has a lot of Marketing BS. That said I think SpaCy is actually pretty solid and when you are in the NLP field give it a try.
Self-respecting language geeks keep up with the times. What's your case for "leading open-source"? Here's a look at Spacy blowing Stanford Core NLP out of the water (via github stars, you can take a look at commits and more from the same tool): https://www.datascience.com/trends?trends=4812,7214,7165&tre...
When a new tool is announced, there is a lot of casual interest. Hence the 'explosion' in the beginning. After some time, things reach steady-state, and you can see that Spacy's interest is starting to fall below Core NLP's in the last few weeks.
It's interesting to see how this comes across to people who are outside our ML/NLP bubble. I can definitely relate to the feeling of being confused at why something that looks sort of basic is supposedly significant. It was actually very difficult to build, especially the active learning component for the named entity recognition system.
I'm a data scientist and getting annotations for our data is one of our most onerous issues. I upvoted this. If it works well, I could see myself using it all the time. Making a model that gets you most of the way there is the easy part; getting clean, annotated data. Uggh.
Since syllogism is participating in this thread, what kind of active learning are you using? I'm always hesitant to use anything except for IWAL since most of the more common ones aren't actually consistent. Even then, then payoff tends to be kinda disappointing.
Yes, it uses importance weighted active learning. You can set the priorities yourself, but the default built-in sorter is just uses distance from 0.5. There's a random component to help make sure the model doesn't get stuck asking the wrong questions.
This looks interesting because it add the ability to put the user in the loop of fixing/annotating the problematic observations relatively easily. I like the example of Tinder for data.
Are the examples picked those that have the highest objective function error rate, or something similar?
Does this apply only to text classification problems? Are there examples where this could be applied to tabular data?
Looks very nice, although it always takes me a bit to figure out what they're talking about with these sorts of things because I have to remind myself that most ML/DL stuff is supervised. What I research is unsupervised.
They kind of have this weird dissing of unsupervised scenarios, though. It's not like supervised or unsupervised is better or worse, they're just surrounding different problems. They can talk up their product without needing to criticize a problem domain.
It's like if you were making motors for boats, and then started talking about "these crazy people who think it's better to fly." ???
I see how that came across as obnoxious, so thanks for the perspective.
I do think there's a pretty common failure mode for teams who don't have much experience with ML, though. Teams who don't have much experience with ML often take "We don't have much data" as a parameter of their problem, and don't see that this is something they can decide to change. This can lead to a lot of time spent experimenting with different unsupervised approaches that are a poor fit for what they're trying to do.
How many languages are supported? I see many more languages in Google's Syntaxnet library. What's keeping you from having the same list of 40 languages for POS tagging?
The UD treebanks have made it very easy to offer lots POS and dependency parsing models under a CC-by-NC license. We'll be putting up more of these for download as spaCy 2 stabilises.
We're mostly worried about saying we "support" a language when we've just trained a tagger on a UD treebank, though. We like at least having the stop words and tokenizer exceptions filled in by a native speaker, so the usual flow has been that someone needs the functionality, and they make a pull request.
If you just need the UD model for say, Bulgarian, you can do:
python -m spacy train xx /path/to/output_model /path/to/bulgarian-train.conllu /path/to/bulgarian-dev.conllu --no-entities
We don't have a spacy.bg.Bulgarian language class yet, so you can either add one, or use the multi-language class, which usually works OK.
I've been pulling my hair out and losing sleep over a specific problem I need to solve for a client. This tool, along with the linked spaCy lib have not only reduced the complexity of the task to be manageable, have also drastically reduced the projected completion time. In other words, Holy shit thank you OP.
Nope, not just fluff. New ML model architectures get too much hype, while it's relatively simple tools like this that actually make the difference in whether or not ML can applied to industry problems. The low hanging fruit in the ML industry are in workflow tools rather than novel model architectures. I have a huge amount of respect for the folks at explosion.ai, largely because their solutions are consistently good in practice rather than good in theory.
Exactly. I'm working on something related: building a UI on top of declarative ETL pipelines to drive ML models. I think a lot of time (and big data resources) can be saved.
No, I don't think so. People tend to over-emphasize the latest ML techniques when the greatest improvements right now come from better and cleaner data sets. I am interested in this sort of thing because we were just about to try and build something like it ourselves.
Examples for text data: topic tags, marking mentions of companies, finding descriptions of protein interactions. For images people mostly do segmentation (finding object boundaries) and classification.
If the model is generating images or text, you usually need to do annotation to evaluate the model. For instance, you need to know whether the translations are grammatical, whether an image is coloured correctly, etc.
I have an NLP bot as a hobby, but use old-fashioned statistics instead of ML. It looks like annotations here mean manually training the bot by seeding the learning data with hints.
Yes, with a bit of a twist. As I understand it, it'll keep retraining the model and asking you to label the examples it's least sure about. This is a lot faster and better than randomly labelling your data or trying to do it all.
More intensive than adding numbers together, sure, but computers are pretty fast. If you're doing 100k connections a second you might have to give some thought to that. Meanwhile, if you have KeepAlive on, 2~5 clients per second will kill you.
