Nice work! But people should be aware that this type of thing is patented left right and sideways. I wrote a patent for automated actor replacement in filmed media in '05, filed it globally, and it began being awarded in '08 in the various regions it was filed. Here's one of the US versions: http://www.google.com/patents/WO2007035558A2?cl=en I no longer own it; now it is part of a broad family of patents collectively forming a "Personalized Media" infrastructure. It is owned by a major media company I'm not allowed to disclose.
Essentially, it is possible to replace people, objects, environments, & audio elements automatically through a series of techniques. Those techniques form the basis of a powerful advertising and message delivery infrastructure. And the collective IP is already owned by a major media conglomerate.
This is literally what's wrong with the patent system.
Discouraging people from exploring ideas and building things 'because its patented' is fundamentally wrong.
I bet there are a million patents out there for 'automated face replacement'. I just hope very very much, that somewhere there is some prior art that makes them all completely worthless and bankrupts whomever is making any money from them.
What you said was a little strong, but I do fundamentally agree with you that patent clerks do not perhaps have the benefit of 5-year technology trajectory to make informed decisions.
When you look at the patents that Newegg fights (and wins), clearly something has gone wrong at USPTO.
Patents are for particular methods of doing this. You can't reduce all patents to "moving atoms and molecules around in different configurations" claim prior art, and invalidate all past, present and future patents.
To illustrate this, a patent (as originally intended at least) wouldn't be for something broad like "walking down the street" but maybe for something more like "walking down the street with a particular gait."
While I don't necessarily agree with the patent system, it detracts from the conversation with it's not an informed discussion.
True that the face is just a pattern, but matching and replacing in a manner that a human viewer is not aware of the manipulation is an entirely different matter. Integrated lighting, media grain, obscuring elements, including any reflections of your subject in the replacement, matching motion blur aliasing and so forth form the realistic problem to solve.
Those are all problems, not solutions. Problems aren't patentable. A method for solving problems is - and even then, only if it's a novel, nonobvious method.
The automated face replacement patents begin in the late 70's. By '05 the process had to be entire in-motion actor replacement to be patentable. To some degree, I'd say patents force people to work in uncharted areas.
>> Discouraging people from exploring ideas and building things 'because its patented' is fundamentally wrong.
Who's discouraging? Being aware that it's highly patented doesn't mean you shouldn't play around with it, it just means that you probably shouldn't be trying to make money off of someone else's work.
The thing with software patents though is that most of the time they've preventing you from making money off of your own work, just because someone at one point did that same work first. Patents are needed in industries where r&d is expensive enough to warrant protecting inventors for a sizeable time so they can recoup their investment. Software r&d is too cheap for patents on software to do anything but harm.
As a quick rule of thumb, whenever people are being advised never to look at patents, as they are in software, that's when those patents do more harm than good.
I agree with the first part, and disagree with the second part of your statement.
Just because there are patents doesn't mean a person can't play around and even try to make money off of their work. It just means they need to do a little bit of investigation before trying to monetize their work.
Just because something looks like it could be patent encumbered doesn't mean you shouldn't pursue it. Patents need to be fought over in court. It costs millions of dollars to bring a patent lawsuit. In the event that your little side project or startup becomes a target for a patent lawsuit, frivolous or otherwise, you've likely already won.
Even with a semi-broken patent system, the world simply favors active, high-quality execution over passive "intellectual property" generation. Focus 364 days a year on a building a great product that people love and then spend a day to talk to your attorney about patents, IP, and how to protect the valuable company that you've built by focusing on your customers.
Be aware that a single technique can infringe on multiple patents, and a patent can extend other patent(s), triggering a situation where a single technique requirements licensing from multiple entities.
It is exactly the issue of fighting patents costing millions of dollars that makes technology development financially risky. When accused of infringing, one has to respond, and that response costs both significant time and significant money.
I agree a side project need not worry. A startup does. I also agree that the world favors active execution over passive intellectual property generation. Our world also forces the active executing organization to protect itself if they need any financing to realize their goals.
Aren't patents supposed to be public information ? One of the (rather hollow in practice) justification for patents is to (also) encourage eventual disclosure of knowledge.
> The patents are, but their ownership does not have to be.
I disagree.
It's important that people be able to determine whether the patent is in use or held by an NPE that's obligated to license it to people who want to implement products.
Were I to want to work in motion capture, I have no way to ascertain who owns your patents, if they're an NPE, and what possible licensing options exist.
I think patent owners should be forced to register with the patent office in a look-up table if they want to hold their patents.
I think the GP was just stating the fact, not expressing his opinion about it. But I agree that secret patent ownership sounds a little dodgy. Say I want to apply for a license to use some patented technology? Who am I supposed to turn to?
... and people should also be aware those patents might be invalidated by re-examination or an action in federal court. I don't know why you aren't allowed to disclose who now 'owns' the patent you refer to. It is trivial to locate this information with the link you provide and some googling.
Also your statement "the collective IP is already owned by a major media conglomerate" if taken to refer to totality of technology which makes it "possible to replace people, objects, environments, & audio elements automatically through a series of techniques" is false.
I beg to differ, there most definitely is a series of patented technologies that in total enable replacement of people, objects, environments and audio elements in traditional media. That key is traditional, linear media. New forms of media, interactive forms, VR/AR and hybrids are wide open fields. That's a big reason new fields of technology are so exiting - it's not just interesting to us geeks, but valuable to the businesspeople.
What's your view on the patent? Pretty broad? Or a sign of very quick progress that in 10 yrs something innovative (hence, a patent) turns into a few open source libraries and few lines of code to replicate. Most responses (including my thought) call 'patent troll' etc, but perhaps there was substantial thought and effort in the techniques below the patent.
The majority of these "Personalized Media" patents rely on existing integrated media - meaning imagery and audio composed of multiple elements - and the ability to isolate one element, remove or modify it, and/or replace it with something else in a manner that the original media's intent and message is retained, only now it is enhanced for a specific audience.
Think product placements that change for every single viewer.
Real time rendering, VR/AR and that entire enchilada of media techniques completely sidestep these patents because there is no original media, it's all created and composited on the fly, dynamically for the single specific consumer of the media.
Just to echo this a bit, I spent 3 years working on image similarity tech and a lot of our research was challenging, not because of the intrinsic complexity of the problem, but because we had to implement it in such a way as to step around existing patents.
I guess you found the math combined with the search for alternative routes interesting? But did you also try to find prior art? My first thought would be that for unreasonably broad patents prior art is the easiest way to be able to use the tech?
I wouldn't say the patents we hit were broad. In some cases we thought they were fairly obvious, but that's much more subjective. We didn't chase prior art, mainly because none of the barriers were unavoidable and we wanted to keep momentum on the build.
I only had one foot in the algo camp to be honest. My main task was making the query engine scale.
Which is the exact situation I found myself in: I began developing in this area, and learned of patents I needed to sidestep. Realizing a hole in the patent landscape, I filed to protect what I was doing.
It's only an issue because the original "sidestepper" sold-off, or was perhaps bullied into selling (theory), the patent to an entity. An entity that will unfortunately just hold it in a "bag of tricks" until they need it, or decide to play with it, or decide to litigate with, or decide to "license" out usage of it to another big player willing to fork over money for it.
Had an innovative tinkerer kept it, then who knows what might have happened. But I'd take a wild guess and say that the odds are more likely that a such an innovator would allow others the use/play/profit off of their novel ideas than would a large entity that sees it as a potential avenue of competition.
Try doing this 10 years ago, putting your life savings into it, and then VCs and media companies "not getting it", your money runs out, and then you have to sell your IP simply to pay the rent.
I'm not meaning to criticize you personally (at all). I'm just saying that the idea of being able to "declare ownership" of what is essentially a series of mathematical operations just doesn't feel quite right.
I realise that people have to make money some how, but developing a useful, competitive, monetisable product that people are able to choose to buy is, in my opinion, the way forwards. Patenting the technology so that you can operate without competition rather than developing your product to stay ahead of the competition "organically" seems like the wrong way of doing it to me.
I agree with you. But the patent system exists... A startup has to protect its work. Plus, when pitching for financing the first set of questions include "Is this patented?" If you answer no and don't have a valid reason why not, they tell you to get it patented and come back when that is done.
So, there are an infinite number of things that are just series of mathematical operations. It's the ones that are useful that matter. Why would we not compensate people for discovering them?
I needed financing for my startup based on this technology. After an initial financing search, it became clear it had to be patented to get financing.
To some degree, patents are a mechanism forcing early technology companies into expensive legal engagements, which drain resources from the company, triggering hefty financing requirements, and a dilution of the inventor's equity.
It is a carefully crafted system that benefits the 1% entirely, and the remaining 99% are merely the crop they farm.
Don't worry about the naysayers. I am in the same position as you right now as we are doing work with AR and computer vision that people/investors etc... don't get yet. Maybe we are too early in the market, but probably not. If we are too early then at least we have something that we can fall back on without losing everything we have put into it.
At the end of the day I have to be able to say that I did everything I could to make our technology "win" in the marketplace.
If we are too early then at least we have something that we can fall back on without losing everything we have put into it.
No, see, what'll happen is your patent will be bought up during your exit event/firesale, and then used to harass other people in the space.
Especially in AR/VR/whatever, there's a huge incentive to patent things which are fairly obvious with just a little applied engineering time, and which will be used to harass a lot of people later.
If you wanted your technology to "win", you should make it open and free (in the software sense) to use--as it is, you've just got some tepid AR entry in the backwards swamp that is AEC.
Ha thanks for all the kind words. Actually our patents don't have to do with AEC and are actually pretty novel in the computer vision realm.
I actually ran the idea of putting our work for "free" by our current investors because I am a fan of FOSS but it's a non-starter when it comes to raising money and it's just a fact of life that we need investment to keep going.
I'm curious, did you put together an algorithm and\or program which actually did this automated actor replacement or did you just patent the idea? No judgement either way, "don't hate the player..." and all that :)
edit: looking at your bio it seems pretty likely that you had properly implemented it!
Fully implemented, but expensive to use. It's modern visual effects technology, automated. Also, not a single algorithm or program, but a media pipeline of over a dozen custom programs, plus integration with the major VFX software packages.
Come on guys. Like all patents, the owner must be disclosed to the USPTO. A quick Google search turns up Flixor as the original owner, and "Big Tent Entertainment, LLC" as the current owner of some of the recent patents in this family. A further search of their website indicates it licenses various media technology to companies including "Concept One, Nintendo, Fifth Sun, Parragon Publishing, Harry Abrams, Kids Station, Merch Source and more".
Likely, this is a standard term in their inventor assignment contract. When bsenftner signed it, the patent applications were probably not public yet (there is an 18 month period between filing and publication in which the application remains "secret"), and during that period it makes sense to have a non-disclosure term.
As for the current owner being a front... if it is, it's not a very good one (real patent troll companies are much better at it, using shell corporations in various jurisdictions, each tied to a single or a small number of patent families).
Not true. The applicant must be disclosed during the application process, but the owner can be legally change without notifying the PTO or updating ownership records. Assignments can be registered, but you don't have to register assignments. So Big Tent could theoretically sell the patents to another party and never register the sale with the PTO.
Noob question: What if the developer who "might" have infringed on the patent, open sources his work, and it's made available on github with multiple forks, and now there's a whole community behind it, and so on?
OpenCV is an awesome Swiss-army steam-shovel (thank you Willow Garage!)
In the 90's, there was Kai's Power Goo (now owned by Corel) and probably something similar used by Conan O'Brien's art dept. to produce the "If They Mated" segments. It's such a shame tools owned by big companies are effectively locked away from ever being marketed again by asserting IP rights.
I really think intellectual property needs a principle similar to adverse possession [0]; that is, if it's not being actively used in a meaningful, sustained venture, another shop can reboot it. (Giving capitalism the right to fork.) Btw, I'm not the first, nor the last, to conceive of this. [1]
The amount of mathematics that goes into automating faceswaps is awesome. The fact that anyone can do this at home with the sole purpose of generating a funny image is great!
Isn't this like totally giving away the Enquirer's[1] secrets? I enjoyed the discussion too. Next up, deriving a set of makeup (cosmetics) steps that need to happen to have your face recognized as some other persons face.
[1] a gossip newspaper which often (always?) has dubious photographic evidence backing up their claims.
