The idea that private companies were responsible for the majority of the innovation that led to these video compression standards is a complete fabrication.
The majority of the work was done by the grad students of the professors who are on the standards committee. Private companies then race to patent or buy up anything that's tangentially related during the draft period, so that they can sign on and receive royalties after standardization. Many of these companies are owned by or have board members who are standards committee members. The whole thing is a giant racket.
Licensees were getting tired of this, so they started to do the same thing, and now you have multiple patent pools, no clear guidance on which ones are legit, and even fewer people who are willing to play along.
And so what do you think is the solution? Do you think that "Only by becoming a Technical Committee can MPEG [...] stay competitive in the market." is true, or what could be a valuable alternative?
It seems like there's an answer: the market actually wants free standards because they're realizing it's more beneficial to them than any other outcome. So money and effort is being pumped into developing free video encoding standards. It seems there is an incentive for the foreseeable future to continue to improve video codecs simply because it saves on bandwidth and disk space costs, and improves user experience.
Though obviously, everything's too good to be true. Just like MPEG seems to be hitting the end, some day the AOM will hit the end, too. The world changes. Something else will emerge and impact the business model. It may be worse, or maybe it'll be a return to norms. Could just be another one of those things that cycles every so often.
With audio codecs, it doesn't look like we ever really needed much in the way of commercial entities, since Vorbis and Opus have been outperforming the competition for a long time now.
> …Vorbis and Opus have been outperforming the competition for a long time now.
Sorry to be pedantic about one bit of an otherwise-fine post, but AAC is comparable to Opus at any given bitrate[1]. Plus, even the most esoteric flavor of AAC has the enormous advantage of 7+ years of ubiquity[2].
[2] All versions of Android support HE-AACv2 playback, iOS introduced support for HE-AACv2 playback in iOS 4 (released 2010), macOS introduced support for HE-AACv2 playback with iTunes 9.2 (released 2010), Adobe Flash Player introduced support for HE-AACv2 in 2007, All versions of Windows Phone support HE-AACv2 playback, all open source players support HE-AACv2 playback via FAAD2
That comparison appears to says that the difference in quality between opus and he-aac is the same as the difference between he-aac and vorbis.
Opus is definitely better, but we have basically hit the wall in audio compression (we'll need a serious breakthrough to go much farther). There also isn't as much interest in audio codec research because even very large raw audio files pale in comparison to fully compressed video.
Incremental improvements are happening in opusenc, current status (as of 1.2 and 1.3-beta) is that musical content is generally transparent at ~96kbps VBR, with a few needing ~128kbps and a tiny minority needing ~160kbps.
The developers are slowly reducing the required bitrate for transparency, it wouldn't surprise me if they reach transparency at ~64kbps VBR in a year or two.
Generally Transparent = MP3 LAME @ ~ 128kbps. That is the good old standard.
There has been PR and marketing materials, much like Video Codec making claiming they are 50% better then H.264 or AVC or whatever. AAC or MP3Pro even claim to be MP3 128kbps quality at 64Kbps. Of course which never happened even after years of fine tuning.
Opus is the first and only codec in years, or decades that had better music quality then MP3 @ 96Kbps. Even AAC cant do that, at least not in majority of cases.
I have come to the conclusion that Audio compression has come to end of the S curve, with diminishing returns. It literally took us all the years till all MP3 patents has expired to get an Audio codec that is better at it with 20% less bit rate.
I very much doubt we could have 64Kbps VBR to sound better without many more breakthroughs.
>Generally Transparent = MP3 LAME @ ~ 128kbps. That is the good old standard.
Says who? And which version of LAME? That encoder has seen huge improvements over the years. We should not define audio transparency by an outdated and flawed format. We should define audio transparency by whether it is distinguishable from a lossless source or not.
But yes, a modern version of lame (3.98 or newer) will be transparent around V5 VBR, which is ~130kbps. BUT -- and this is a huge twerking 'but' -- the MP3 format suffers from pre-echo. There is nothing you can do about it. It is lessened at higher bitrates, but you can never completely get rid of it. There is also the problem of the SFB21 defect, which can severely bloat the allocated bitrate on content that is heavy in high frequencies: http://wiki.hydrogenaud.io/index.php?title=LAME_Y_switch
Newer formats such as AAC, Vorbis and Opus do not share these limitations. I don't know where you read that AAC is worse than MP3, because that is simply objectively false, as shown by every listening test you can care to dig up. Did you mistakenly use FAAC (probably the worst AAC encoder out there) or something?
I would highly recommend that you give it a go yourself. Take a FLAC source, encode Opus files yourself. The encoder is freely available from https://opus-codec.org/downloads/ and integrates nicely into Foobar2000's converter function (Opus is already there as a profile, it will ask for the location of opusenc.exe the first time you use it).
Use the ABX Comparator plugin for Foobar2000 (both available from http://foobar2000.org) and run a few trials.
After all, public listening tests are one thing, but to be absolutely sure a format/bitrate is transparent to you, individual testing is needed.
I have run a number of tests myself (of Opus and other formats) and found general transparency at ~96kbps Opus, lower for some tracks. For those tracks that weren't transparent at that bitrate, the artifacts were generally benign and not that annoying (slightly increased noise levels, mostly).
OK, fair enough: HE-AAC had been out-performing the leading open source audio codec for a while. Easy to forget primarily because even after many platforms supported HE-AAC, it wasn't necessarily common across all media. I really saw it explode on the internet around the time DASH streaming became more common, since adding new audio formats was much less expensive.
That being said, I still think Opus is better now, since it is more versatile. I've yet to see anyone use HE-AAC in a real-time context, unless they're using it under a different name.
Further nitpick: the page is very misleading. Audio compression has different targets, a crucial one being music.
The Opus official page has comparatively little test references for music, both of which are at 64/96kbps for AAC [1-4], which is significantly below the general audience listening standards (160+ kpbs).
This implies that the red line on the top right of the first diagram is literally made up.
Until further blind listening tests are performed, generic conclusions such as "Opus outpeforms the competition" are unfounded.
> It attempts to summarize results [...] and (when no data exists) show anecdotal evidence
This is borderline offensive for people who are serious about researching music quality. Any anectodal evidence [in this field] is garbage.
There's a good reason all the listening tests have stopped at 96 kb/s. Above that, the quality of Opus, Vorbis and AAC is so close to transparency that it's pretty much impossible to get statistically significant results. Even the latest 96 kb/s test was really stretching things.
No it's not. Business is a non-moralistic action. People can be immoral doing business, but business itself cannot.
Also, I happen to know a few very moral and ethical business owners who would rather lose money than screw someone over or do a shady business deal. (and have done this)
A major part of the research — e.g.the entire audio part of the MPEG standards — was done at research institutes such as the Fraunhofer institutes, which finance themselves from patent licenses.
So your comment is also missing a major part ofthe picture.
I'll recommend reading the other pages on Fraunhofer's site explaining their goals and why they were created, and how they differ from the other German research societies .
But of course, government funding is today one of the largest sources still.
I don't need to, I'm fully aware of the differences between Fraunhofer and the others (basically PPP, applied sciences etc.).
I simply posit that (software) patents actually aren't needed for that.
Or to be more specific: economic exploitation of their patents - nowadays, I could e.g. imagine them to offer their considerable patent pools for defensive purposes, to "disarm" patent minefields in problematic fields.
Of course, that would be a tightrope walk in regards to cooperation with private entities...
The use of patents by Fraunhofer is to ensure that the research is actually real-world usable (and the government isn't just wasting their grants) and that it is worth it to the industry, while at the same time ensuring that the industry helps in funding the research that it will make profit fron
Sorry, read my edit. IMO it's an incredibly reductionist view that it's only worth something to the industry if someone can make (most of?) money via patents from it.
And another edit: One could argue that's a side effect from the modern anglo-american tradition of patenting (that has seemingly spilled over to continental europe) that almost completele ignores "Erfindungshöhe", stripping patents of their inherent technical merit leaving exploitability as the only measurement.
That's a valid view — but that's specifically what separates Fraunhofer from the others.
The Helmholtz-, Leibniz- and Max-Planck-Societies exist to do science for science's sake. They build stellerators and map the oceans, because it provides knowledge.
The Fraunhofer soxiety exists to turn theoretical research into something companies can directly implement. That is what separates it from all the others. It is supposed to provide finished templates and blueprints with its patents, so you can license something, and have it ready to use.
That is why Fraunhofer didn't just develop an algorithm for mp3 and aac, but developed production ready encoders, decoders, hardware en- and decoders, 5.1 decoders and stereo, even simple ones for mono. They created the source code, heavily documented, and the algorithms, and blueprints for building them in hardware.
So that companies can license it, just integrate it, and are done.
That is why the Fraunhofer society exists, because our economy of small to middle businesses can not compete in R&D with giants unless we do all the R&D in such a research institution licensing ready-made solutions to your small companies.
And yes, the US-view on patents with regards to not needing implementation detail and no requirement for Schöpfungshöhe (in fact, the US system even has the opposite of that) is severely damaging this concept.
I know - and that's where wanted to get at: I'd argue that, esp. in regard to their cooperation with SMBs and the benefits for the latter, nowadays it's the sheer concentration of R&D expertise (resulting in better risk mitigation, faster time-to-market etc.) that's the main driver for institutes like Fraunhofer, and monetization to third parties might sound that it is, but is actually incidental in practice.
(Ironically - in contrast to defense against patent violations/litigation, where having Fraunhofer's back is certainly a big benefit! :) )
I mean, we are in a time where e.g. even Sony (of all companies!) freely licenses their LDAC codec to the AOSP. To me it seems that after decades of countless multi-front patent wars many finally start to admit they are sick and tired of it. (Esp. when looking
to the rising China, which seems to do well at the innovation front while not giving much of a damn about patent protection...)
The professors and their friends on the committee are self-dealing when allocating the committee's research funds. Grad students at the Fraunhofer partner universities still do the bulk of the work. I don't see how I've missed anything.
Fraunhofer can't pay its researchers anymore if their patent income goes away? Research then is only done with government funds or in corporations, where the corporations definitely won't license their research (e.g. selfdriving tech) to competitors.
In places where the research is done like you said, all your points are true, but IMO institutions such as the Fraunhofer institutes are worth keeping.
I agree that they're worth keeping, but patent income isn't the only way to fund this research. It's just been one of the easier ways to fund it up to now.
Also note that I never claimed that the patent income wasn't being put to good use by funding research. I was arguing that many of the patents themselves were bullshit, and that many of the companies that own them had acquired them from universities on slightly-too-friendly terms because of the MPEG cartel.
There isn't really another way to fund Fraunhofer institutes — they're supposed to turn theoretical research into something that can be directly applied ny corporations, so they have to have an incentive to create something like that. With funding through patent fees, exactly this happens.
The alternatives are government funded research, or companies like Google doing it all internally, but then Google will keep some of it as trade secrets, which is also undesirable.
I don't think there's an alternative if your goal is go keep all research publicly available and in a way that any company can easily license it.
But for this model to work, the international community of licensees has to actually believe that your patents have merit.
If you act like a cartel, generate reams of questionably valid patents so that you can hand them out to your member corporations under sweetheart terms and at the expense of your licensees, you're eventually going to piss off enough people to make the music stop.
Evidently, this is what happened. The patent pool is now so large that it's really easy for other players to come in with their own, similar patents. They can use these to create new patent pools and try to take everybody hostage with higher fees (HEVC Advance), or they can build a protective moat and threaten to sue anyone who demands licensing fees from them.
Patent licensing can be made to work in a lot of cases, but these guys really dug their own graves here.
Also, I think there are a number of alternative funding routes that can be made to work. Government funding, as you mentioned, could be one of them. Terms of the funding could stipulate that the research be made publicly available and licensable under reasonable terms.
Companies can also choose to fund the research via Fraunhofer rather than doing it all internally. Not everybody has Google's resources, and there are major advantages to having improvements included in a standard that everybody can access.
Wait... someone at MPEG actually believed their own bullshit? That a massive campaign of extortion, intentionally complex legal maneuvering, and outright intimidation is actually beneficial to technological progress in some way? (Sure, it's beneficial to some of the companies involved in the licensing extortion scheme; I don't think anyone disagrees with that.)
Or is this guy just an incredibly skilled troll with 30 years of experience in pulling the wool over people's eyes? I don't think so, though. He really seems to think that imposing massive amounts of friction on a system somehow makes it work better.
Wow. Just wow.
I mean, I'm not unsympathetic to the argument that patents can promote advancements by funneling money to inventors, however weak it starts sounding when you look at actual events in certain spheres. But the claim seems to be that nobody is going to bother to research improvements in an interesting, impactful technical area because they'll no longer be able to (1) invent something, (2) hire an army of lawyers, then (3) profit. Er, okay. Let the people who will only work with that sort of path go do their ICOs instead.
When people first started discussing which codecs to use on the web a decade ago, the argument made for the MPEG-LA stuff was that a) we're better, b) we're reasonably priced, c) we're already everywhere in hardware, and d) you don't have to worry about the patents because we've got you covered. Ultimately, they won on that basis. However, when HEVC rolled around, that argument was destroyed by the multiple patent pools, and the consumer groups for the patents were pissed off enough to make competitive alternatives.
I suspect that he believes that the perpetual funding stream of ever-better patents is needed to fund the research for ever-better codecs (which, as you say, is disputable but not unreasonable). I also suspect that he would argue that some of the problems of the protection racket amounts to a necessary evil (a protection racket does give you a viable means to combat submarine patents). However, the AV1/HEVC situation undermines these arguments, and this is someone who realizes the gravy train is running out of rails and is desperate to keep it from derailing instead of bailing while he still can.
So how does this work out fee/patentwise in the OSS world? I know ffmpeg/libav can play HEVC/x265 streams (or at least mpv can). Can it not encode to HEVC? Or do the fees start to apply when you start distributing content? Or is it like mp3 where fees are only attached to hardware encoders/decoders (which gets all iffy because where do you draw the line between hardware sold for encoding/decoding that has software codecs on it vs general purpose OSes with the same codecs?)
And doesn't Firefox have some special deal with Cisco to use their x264 decoder on the web?
ffmpeg just ignores patent licensing - you're responsible for paying the appropriate licensing fees if you use it. The same applies to x264 and x265. Generally the pools won't go after ffmpeg itself but rather companies shipping it once they get big enough / have money. For example: https://github.com/mbebenita/Broadway/issues/124
HEVC is currently literally impossible to license. One of the pools hasn't decided on pricing yet. You basically have to set aside some cash and hope it's enough to back-pay once they announce the pricing. But even with the existing ones, you have to pay for encoders, decoders, and videos encoded in it. https://en.wikipedia.org/wiki/High_Efficiency_Video_Coding#P...
There's no special deal per se, Cisco released OpenH264 for everyone, and it can be used royalty-free as long as you only use the binaries (instead of the source): https://en.wikipedia.org/wiki/OpenH264
Resolution isn't the only part to consider, the amount of detail you can store in that resolution can differ widely. With the push towards enhanced colour spaces, higher framerates, etc. there's plenty of need for better compression technology.
Besides, for a company like Netflix or Youtube, shaving 5% off the file size without a drop in quality can literally mean millions of dollars saved per year.
If you consider compression from an information theory point of view, for any given measure of "quality" there is theoretically a limit to compression. It's not clear where the limit is, or how close we are to it, but because it does exist stagnation is inevitable eventually.
As it gets harder to make gains it will become less of a business proposition and more of a research/academic thing.
If I’ve said it once, I’ve said it a thousand times:
Fuck the Fraunhofer Institute.
There have been a lot of mistakes in the growth of the internet, but mpeg... Trying to own an entire space isn’t about making the world a better place. It’s about being in control. Me me me.
Given how big video is now, and the interplay between available bandwidth and video quality, one wonders if certain bumps in the road would have been smoothed out if Netflix or YouTube had ramped up a few years earlier.
I think it was interesting when they released the mp3 patents, every organization was broadcasting "mp3 is dead" when really the headlines should have been "mp3 encoding/decoding is now totally free."
I'm convinced many big players wanted people to think mp3 was going away to push newer, patent encumbered formats.
I still remember being appalled by just how wrong that news cycle presented the information. Even a bunch of tech news sites repeated the same "mp3 is dead" headline.
Usually I'd expect someone in this position to toe the party line pretty hard, but he actually sounds reasonably level-headed to me. He even cautions the reader not to get their hopes up that anything will improve.
> AOM will certainly give much needed stability to the video codec market but this will come at the cost of reduced if not entirely halted technical progress. There will simply be no incentive for a company to develop new video compression technologies
I don't think is correct. The driver for innovation in video compression technologies will continue to be the practical problems of encoding and distributing video. The quest for smaller video file sizes while maintaining quality is what has driven the development of VP8 to VP9 and now to AV1. Smaller video file sizes directly help reduce the costs of companies like Google, Amazon, Netflix, Hulu, Apple, etc. and I don't see that changing any time soon. There will be an AV2 eventually.
Yes. This is the lie of software patents. That somehow without thousands of lawyers and corrupt business contracts engineers will stop innovating. We don't need them.
It's the lie of intellectual property in general: Without a way to own ideas and inventions, and control how others use them, nobody will invent or create new ideas.
In many areas, engineers need to get paid to innovate. Dating back to at least the standardization of CD-ROM, that was mediated through the use of patents. Note that until recently, there was almost no open source development of media codecs.
Today, there is an advertising and entertainment industry generating huge surpluses that can also bankroll engineers to develop codecs. I’m not convinced that this is a better model.
Hi. I have personally been developing open source media codecs for 20 years. I've had a lot of help from other open source developers.
I have been paid by Mozilla to do this in some capacity for a decade. The budget for our entire team (which is the largest it has ever been), is less than the cost for just Mozilla to license H.264, by a non-trivial margin.
Here are some of the things that team has done:
- Made a Theora (VP3) encoder easily superior (in quality/bit) to what Youtube was shipping in Flash at the time (VP6), and (favorably) comparable with H.264 baseline.
- Developed and standardized the Opus audio codec.
- Provided infrastructure and maintainence for Xiph's older codecs (Vorbis, Speex, FLAC, etc.).
- Provided some minor assistance to VP8 and VP9 development.
- Made a from-scratch video codec (Daala) competitive with HEVC while avoiding the patents (i.e., with one hand tied behind our backs).
- Made significant contributions to a video codec superior to HEVC (AV1).
Now, maybe you want to call all of that almost nothing, but this code is in every Apple and Android handset in existence, and every current desktop browser, and used by some of the largest websites in the world (Youtube, Wikipedia).
The major concern with open source development is the free rider problem: why should Media Company X spend resources on an open source codec when Google will build one for them?
But the value current patent holders are attempting to extract from an individual licensee is larger than the cost for them to develop the technology from scratch by themselves. That value isn't created by the technology. It's created by having a common standard. And we don't need patents to incentivize that.
And it was under development all the back to 1993 or 1998 depending on your opinion. Including image codecs, PNG was 1996. For video codecs, Huffyuv was 2000. (Though to be fair lossless codecs are relatively easier to create, which is why there’s an unnecessarily large variety of them.)
And even lossy video codecs had some OSS experimental developments in the early 2000s that admittedly never went too far - Tarkin predates VP3 open sourcing if you want to count that, and Snow got somewhat decent results if you ignore that H.264 predates it.
Ogg Tarkin predates Theora and tried some completely new ideas in a time when digital video wasn't as ubiquitous (and so there wasn't as much incentive to keep going when problems arose).
Tarkin also suffered from the patent minefield everybody else put up (so they couldn't go for the obvious solutions because these were already locked down for another 15 years)
> In many areas, engineers need to get paid to innovate.
What do you think: How many engineers can be paid for reducing Netflix's traffic by 10% simply by these lowered traffic costs, and not counting secondary effects as a larger user base because more people can watch the videos with their available bandwidth? Also YouTube, Amazon Video, public television, ...
I agree it isn't correct, the implication is that the only "value" that generate new codecs is 'money for royalties.'
There are many reasons for developing better codecs, whether they perform better on hardware, they allow for more channels in a pipe with fixed bandwidth, or optimize the viewing experience for particular types of signals.
When I read the reference post I heard the echos of rationalizations that have been used for years and years to justify applying rents against someone else's labor. I can only hope that the author is correct in that this rent seeking behavior has become unsustainable. That would be a good thing indeed.
This means that codec development makes no sense for anyone who doesn't own either a large distribution platform or a large playback platform. (Much of the research has been done by middleware companies attempting to tax the two; their business goes from royalties to work-for-hire, at best, which is way less attractive for them).
This is fine – in a macroeconomic sense – but of course it sucks if you're one of the companies being disrupted.
By that logic Free Software wouldn't exist, not to mention codecs like Vorbis.
The logic of software patents systematically underestimates the existing incentive structures. That's because of the simplifications economists usually make. (See https://en.wikipedia.org/wiki/Assume_a_can_opener)
The assumptions underlying patents become less justifiable as
1) the pace of innovation in an industry increases, compounding first-mover advantage;
2) complexity increases, because integration of an invention into a viable product becomes much more difficult--often such that copyright and other IP protections alone are more than sufficient to provide a moat;
3) as financing becomes cheaper and potential markets larger the time horizon for and amount of ROI demanded by investors diminishes;
Those are especially applicable to many modern high-technology industries, software (Silicon Valley) and industrial (SpaceX).
Other industries have proven to be able to thrive without strong patent protections or even copyright protections for myriad other reasons. The fashion industry has thrived without strong patent or copyright protections, largely based on trademark and marketing. Likewise for the beverage industry and consumer foodstuffs more generally. Major automakers had poor copyright protections, and while they had stronger patent protections, as safety regulations increased they shifted to a market segmentation model such that the big auto manufacturers often freely license patents among each other long before the patent expires.
Economists, especially conservative economists, often argue against anti-trust, reasoning that as markets become larger (finance markets, consumer markets, etc), the typical lifetime of a monopoly become shorter--often shorter than the typical lifetime of anti-trust litigation.
I would argue that as markets grow larger and more fluid they're far more capable of discovering and leveraging sufficient incentives to support innovation in the absence of government-imposed monopolies. Especially in the realm of software, the circumstances where there's a legitimate market failure requiring patents are so uncommon that there's no reason to even entertain the notion of a software patent system.
In the absence of _manifest_, _systemic_ market failures, patent advocates make the same mistake Communists make--an inability to imagine how a market can support an endeavor is not evidence that a market is incapable of supporting that endeavor; it's merely evidence of one's own lack of information and imagination. And the surest way to introduce market failure is by dangling the prospect of government-imposed monopoly rents, artificially increasing the opportunity costs of discovering and leveraging all the non-obvious incentives that naturally exist or could evolve.
Free software addresses a fairly limited universe of software, and often is bankrolled by some other monopoly or quasi monopoly. Where would web standards be without the trio of Google (advertising monopoly), Microsoft (operating system monopoly), and Apple (brand-marketing driven fashion company)?
The value of the patent system is that it permits decoupling. Compare the Wintel monopoly to the decoupled ARM ecosystem. It wouldn’t be better if mobile chip R&D had been bankrolled by Google’s say search business or Apple’s cell phone business instead.
> The value of the patent system is that it permits
> decoupling. It wouldn’t be better if mobile chip R&D had
> been bankrolled by Google’s say search business or Apple’s
> cell phone business instead.
I think your examples prove precisely the opposite. The development and emergence of smartphones were effectively bankrolled by Google and Apple. The mobile phone sector was stagnant while every vendor was busily pursuing IP monopolies (Wintel, Sun Java, etc). And in any event, ARM's intellectual property moat largely derives from copyright-like protections on its IC masks, not from patents. MIPS was also a contender early on, but importantly ARM won because it had strong SoC designs ready-to-go.
Your argument is a variant of the argument that by creating a transferable property right, we reduce the transaction costs of negotiating access to inventions. My point is that those transaction costs are dramatically overestimated (don't confuse complex or opaque for costly), and in any event far exceeded by the unintended costs of a patent system.
Importantly, when it comes to complex technology a patent no longer embodies the requisite know-how. People and companies don't lose much by giving away their secret sauce because very often the only people capable of applying and productizing it in a timely enough fashion to capture the lion's share of profits are the inventors themselves. Moreover, the best way for a market to discover who is best at leveraging an invention--if not the inventors--is by making it more freely accessible.
Also, the shift to SaaS and cloud computing means that complex systems never leave the control of their creators, anyhow. This is happening despite the fact that the patent system is supposed to incentivize disclosure.[1] Worse, the patent system makes it easier to stop competitors without substantial disclosure of the end-to-end systems. You can patent all the small sub-inventions, permitting you to block competitors, without every having to disclose the larger, integrated whole that makes it functionally useful in the market.
Even if the patent regime ensures that we get an MPEG codec that is 10% more efficient (or arrives 10% sooner) than an equivalent codec that would arrive in the absence of patent protections, is that worth allowing MPEG rights holders to effectively extort money from smartphone makers?
Because that's what we're talking about here. In this day and age, patents don't make innovation commercially viable; at best patents making them viable a little sooner and a little better. But the _costs_ are enormous, and I would argue far outweigh any meager benefit.
[1] Similarly, film and TV producers pursue DRM despite copyright protections, and despite the fact that DRM seemingly provides little marginal extra profits. The transactional costs to Hollywood of negotiating DRM would seem huge. But either the costs are less than they seem on the outside, or they're already making such huge profits that they can afford to try to lock down the market even more. But copyright was created to reduce the transactional costs of negotiating a locked-down distribution system. If companies are pursuing DRM anyhow, we really should question our presumptions about the necessity fo government intervention. For copyright we probably need some minimal protections (weaker than we have), but we can probably forgo patents altogether, especially in the realm of software.
Again, just because it seems that a patent regime would make a market more efficient doesn't mean that it would. Our measurements of efficiency might bear no relation to what a free market needs to maximize benefits. Communists thought they could devise a more efficient market in grain production, but obviously they missed or misjudged a few necessary incentive structures that allowed for a self-sustaining market (including secondary markets, etc) that maximized output.
Sure a company might want to fund development of a proprietary way to reduce their costs but why would they want to make that available to their competitors in a global standard for free?
Because they need their video to play on other company's devices. Netflix and Google want to use AV1. To make the best use of it they want it supported on as many phones, TVs, computers, tablets, etc. as possible. This "coopetition" is good for everyone.
Because a lot of companies pay costs on products they don't control. A company like Netflix generally has to store their movie files in multiple different formats to support all the various different devices that are out there. If they want to improve the format, they need everyone else on board.
Aren't the browser vendors the gatekeepers here? As long as there are browsers that are given away for free, the decoders would be have to be made also. And if the browsers refuse to add a decoder unless encoding is also patent-free, then we have free codecs.
> As long as there are browsers that are given away for free, the decoders would be have to be made also.
Or have major platforms (realistically, Windows, macOS, iOS, and Android) ship decoders at the platform level, and have them ubiquitous enough. This is, ultimately, how we got H.264 decoders commonplace in browsers.
That said, Firefox held out for a long time with H.264, and only once it was clear it was costing them users did they give in and add H.264 to the whitelisted platform codecs exposed.
> That said, Firefox held out for a long time with H.264, and only once it was clear it was costing them users did they give in and add H.264 to the whitelisted platform codecs exposed.
But even then MPEG (LA) didn't really make any money out of that because IIRC they pulled a hack where Cisco is providing a decoder binary, which they distribute. But the license fee has a cap which cisco was already almost at anyways, so that didn't cost cisco much.
Then they have (had planned?) reproducible compilation. So the decoder is open source, but you're not allowed to ship a binary that hasn't been compiled by cisco, but you can always verify the binary cisco provides matches the source code, as the resulting binary will be exactly the same if your compile it yourself.
Cisco essentially sponsored an open-source royalty-free decoder here.
So yeah, they gave in, but MPEG got nothing from them.
> IIRC they pulled a hack where Cisco is providing a decoder binary
OpenH264 is only used for WebRTC and, importantly, there's no AAC support (which doesn't matter for WebRTC because you can just use Opus). Firefox uses the OS platform's H.264 decoder for video element playback.
> which cisco was already almost at anyways, so that didn't cost cisco much
It apparently cost Cisco quite a lot more in licensing fees but they considered it worth it for the sake of integration with their video conferencing hardware. See the comments to this blog post from Monty Montgomery for some discussion on it:
> OpenH264 is only used for WebRTC and, importantly, there's no AAC support (which doesn't matter for WebRTC because you can just use Opus). Firefox uses the OS platform's H.264 decoder for video element playback.
And Firefox supporting H.264 through the platform decoder came a while before OpenH264 happened.
Right - it's patently false. If there was no incentive to develop new video compression, then why would anyone pay someone else to develop new video compression (buying licenses) either?
> AOM will certainly give much needed stability to the video codec market but this will come at the cost of reduced if not entirely halted technical progress. There will simply be no incentive for a company to develop new video compression technologies.
If you read between the lines, this is an admission that patent licensing is now simply too complicated to make proprietary codecs viable. Especially when enabling an extra compression "profile" requires extra licenses.
Also, that the indirect value of having an open-licensed codec to the big players in AOM is far more than the possible license revenue.
> Companies will slash their investments in video compression technologies. Thousands of jobs will go and millions of USD of funding to universities will be cut.
that's quite a lot of FUD. hosting providers have very aligned incentives to invest in better - and not necessarily proprietary - compression technologies, as do ISPs who have to carry the packets.
The crisis is simple, MPEG rarely cared to make royalty free standards, except may be in the case of DASH. So this is now haunting MPEG tech in a big way, since free codecs are replacing patent encumbered ones for various technical and financial reasons. DASH counter example itself demonstrates it, since it's adopted by big users just fine.
> The situation can be described as tragic.
I'd call the situation liberating. Tragic for parasitic patent aggressors, but great for actual technology progress. Choose what you prefer.
> AOM will certainly give much needed stability to the video codec market but this will come at the cost of reduced if not entirely halted technical progress. There will simply be no incentive for a company to develop new video compression technologies knowing that it assets will be thankfully – and nothing more – accepted by AOM for use in its video codec.
That's false, because instead of patent trolling, proper company would be adding resources to the common pool, benefiting in result from it. I guess the author finds the concept of collaboration to be foreign, but it actually pushes progress much faster than patent driven pocket lining.
> Can something be done?
MPEG can fix it going forward, and push for royalty free / patent unencumbered technology only.
It is, the above is just a list that trolls are assembling to intimidate others. They have it on many thigns, including established royalty free codecs. If they sue someone with it and prove in court they actually own patents on DASH - then it will be another story. Until then - intimidation doesn't equal ownership.
In this industry, such intimidating tactics (very protection racket and mafia like) are commonplace. Check the story how Qualcomm and some others tried to intimidate Opus project.
You can't just dismiss it and hope that it is. The MPEG LA has been successful with their other patent pools. They've got a good chance of being successful with this one as well.
That's what happened to VC-1. Microsoft tried to release VC-1 on royalty-free terms but the MPEG LA formed a patent pool and then it was no longer royalty-free.
Probably not. The better way is to do a review and have ready rebuttals of fake patent claims. I didn't exactly look into how it was done for DASH, but its big users like YouTube and Netflix should have done it.
> The MPEG LA has been successful with their other patent pools.
And completely unsuccessful with many others. There was the same type of list put out by MPEG-LA trolls for VPx, and I remember there was research published that debunked it all. So there is zero trust in anything that MPEG-LA say, but surely having concrete defense is a good thing.
> There was the same type of list put out by MPEG-LA trolls for VPx
No. They didn't succeed in forming a patent pool for VP8 or VP9. It was one of their "pools in formation" but was never established. The DASH patent pool is beyond that stage.
Which doesn't make it any more valid. But I agree that big players like Google and others should spend resources on preventive defense and debunk all that in advance.
And if they'll fail, there should be some concerted effort to make a patent unencumbered alternative.
MPEG LA has been succesful similarly to why Netflix and the iTunes Store were succesful: make it easy and affordable to comply. On the other hand suing costumers who torrent movies was not succesful at all.
The article completely glosses over the existence of the Alliance for Open Media[1]. They mention it and go straight into "Can something be done?". To me, the real question: Why should we care about MPEG anymore?
It is indeed true that the MPEG approach is dead. Once AV1[2] is out, the open, royalty-free codecs will be the better performing codecs for both audio (Opus[3]) and video.
So yes, if MPEG is dead, its components are free to join AOMedia, as long as they leave their silly RAND ideas behind.
The article mentions AOM (and vaguely what it is) about 5 times, and even dedicates a paragraph to AOM:
> Alliance for Open Media (AOM) has occupied the space left free by MPEG: outdated video compression standard (AVC), no competitive Options 1 standard (IVC) and no usable modern standard (HEVC). AOM’s AV1 codec, due to be released soon, is claimed to perform better than HEVC and will be offered royalty free.
This isn't just about being able to decompress video on your smartphone so you can watch youtube without draining your battery, by the way. All compression algorithms, whatever domain they operate in, are about understanding the entropy stored in a stream of data and using that understanding to extract just the useful information. (Lossy compression incorporates understanding of our perceptual systems to only transmit what is necessary to transmit relevant entropy from the source to our brains.)
The primary use is to display media streams with reduced network, CPU, and battery usage. But it's a much more general field, replete with all kinds of fascinating insight about the nature of visual/aural manifestations and human perception. Anything you can remove yet still recover the relevant parts of the original signal is an opportunity to understand how our brains, and the world in general with its physical laws, work. There's a ton of interesting and useful research that could be done -- but would you really want to look into how texture is encoded, processed, and reacted to in our neurons when you have to worry about various mafia-like patent licensing consortiums? The whole licensing setup has a chilling effect on many different area of inquiry, even apart from its impact on classic compression algorithms. (Though that is harmed too -- if you come up with an incremental improvement that works really well for some type of input, and you'd like other people to build on your work, what do you do?)
> Here are a few, out of close to 180 standards: MP3 for digital music (1992), MPEG-2 for digital television (1994), MPEG-4 Visual for video on internet (1998), MP4 file format for mobile handsets (2001),
well mp3 was mostly frauenhofer, not mpeg (yes they created the standard, etc. but the most patentable work was done by the frauenhofer) also the date... on 1992 it was not (yet) called mp3. it's silly to actually name it in the mpeg sense...
Edit:
not a fan of mpeg, but if I would be them I would try to form a non profit and actually try to collect money like that instead of being a super duper not obvious big licensing company.
I mean H.265 was released in 2013 and it's still not as widely used as it could be and better fully open source alternatives emerge quicklier than ever.
It was called MPEG2 Layer III, right? And the licensing around mp3 is why many Linux distributions didn't offer mp3 in the default install, without adding "non-free" repositories.
We also saw open standards emerge, like Ogg Vorbis and FLAC (although neither were widely adopted by hardware vendors compared to Apple's AAC/ALAC)
It is not "Apple's AAC". Apple did not have any part in the development of AAC. It was actually developed by Fraunhofer as a successor to their MP3 encoding.
IIRC, MP3 was based on the audio component of the first "MPEG" standard (the one used for Video CDs) rather than MPEG2 (the one used for DVDs, amongst other things), which would make it "MPEG Layer III".
Standard MPEG-1 MPEG-2 MPEG-4 Visual MPEG-4 AVC MPEG-H HEVC
Bitrate vs previous 25% less 25% less 30% less 60% less
The table at the end is hiding one factor : the computation cost of each new codecs. Have we really found new better ways to compress videos, or is it just Moore's law in action ?
More advanced video codecs have higher visual quality at lower bitrates, but are more computationally expensive. All of these video codecs use the same underlying ideas of 2d transforms, intra- and inter-frame compression, motion compensation, entropy coding etc., but using more complex techniques to preserve video quality.
I wouldn't say it's a simple as "Moore's law in action".
Yes, the computation per pixel has increased ~2x per codec generation so codecs are taking advantage of (a fraction of) the gains provided by Moore's Law.
I remember that time when I tried to play a 4k video (x264 or x265, I am not certain) on my Mac Mini. I was pleasantly surprised that it actually ran smoothly, but the poor thing nearly melted.
Depends on what you mean by "just Moore's law." Certainly each standard was developed with a higher target decode complexity; H.264 and H.265 both targeted 2x greater than the previous standard, JVET is currently targeting 16x, and I think AV1 is aiming for less than 2x HW area.
So yeah, there were obvious gains left on the table for H.264 for complexity reasons that H.265 picked up; likewise for H.265/VP9 that AV1 picked up.
That said, there has been a lot of refinement that didn't necessarily increase complexity, and there have been various tools that saw reinvigorated research once complexity targets were raised.
I think it's because the major cost is bandwidth, so everyone is incentivized to optimized on size primarily, maybe at the cost of being more cpu/memory intensive on enc/dec.
Of course ! I was just wondering about the real technical improvement... Talking about bitrate/(video quality) is forgetting about (computation complexity) that's all
>> Companies will slash their investments in video compression technologies. Thousands of jobs will go and millions of USD of funding to universities will be cut.
I believe this is total BS. There are great open technologies(i.e http/http2) and they didn't need a extortion cartel to police anyone who dares to use them without to buy a license(if they could buy it in the first place). Codecs should be open and they can be developed/sponsored by people who benefit most(i.e. Media companies, internet companies, hardware companies etc)
I think it's worth noting that Google's initiative is an open source project, with its roots back in the days of the Theora (ogg, xiph, vorbis) open source codecs, with which av1 shares a common ancestry.
To argue there will be no innovation in this field without patents is similar to saying there could be no innovation in Linux or Apache without restrictive, object code only proprietary licensing.
tl;dr We were able to exploit the development of tech with patents for decades and now we won't any more.
I'm only happy about this. And I seriously doubt this conclusion: "There will simply be no incentive for a company to develop new video compression technologies".
It seems the author of this text thinks the only way to have an incentive to develop new technology is patent fees. This is outright ridiculous.
Companies like Google or Netflix obviously have an incentive to further improve media codecs, because transmitted gigabytes translates into costs for them.
You can buy a $50 SBC today with 4k HEVC decode HW blocks, enabled out of the box. For me, that puts the technology deep into the established box, and last I checked, whoever is making the IP core for it neither the many manufacturers shipping it are the least bit concerned. This post sounds like scaremongering.
Ehh, this is counting 1.7 billion mobile users for VP9 but not HEVC because of mobile Chrome installs. It's hard to take this serious when it's counting software decoding on mobile (if Chrome will even decode it on a lot of those phones, they know its pointless in SW.)
Meanwhile, nothing iOS supports VP9 HW decode. Until they come around, people could give a fuck about installed base.
Then there is the stuff this article doesn't very well measure. Like DVB-T2 in Germany using 1080p50 10 bit HEVC. That is expected to be around for a decade or more. That is a whole lot of TV settop boxes.
VP9 support is mandatory in Android. If you want to ship Play Store, that is. So it is not due to mobile Chrome installs.
The amount of Android devices dwarfs iOS devices and Apple knows that it is not viable position in long term. That's why they joined AV1. People in near future would not give a fuck about the Apple, if they couldn't play their Youtube or Netflix videos in 4K on them.
Unfortunately that's not where the licensing ends - HEVC pools also charge for content distribution. Hence why you see companies like Netflix in AOM, despite them not having to pay fees for the hardware itself. https://en.wikipedia.org/wiki/High_Efficiency_Video_Coding#P...
This is writing as someone who flavours HEVC, but still support the AOM initiative, hopefully AV2.
1. AOM was formed in 2015, and the idea of collaborating between Mozilla and Google and others were very well known within the industry before the formation of AOM. But let's just call it 2015, so three years later in 2018, the head of MPEG, just a year ago ( 2017 ) has seen this a threat coming.
May I ask What was he doing between 2015 and 2017?
2. >all the investments (collectively hundreds of millions USD) made by the industry for the new video codec will go up in smoke and AOM’s royalty free model will spread to other business segments as well.
Collectively Hundreds of millions? If there were only 10 companies paying the max cap of 25M per year, that is $250M per year from HEVC and to MPEG LA alone, excluding the other two groups and NPEs. And that is devices maker, excluding content producers. Assuming a video codec has a life time of 10 years, that is a total of $2.5 Billion minimum. Do you not think wasting hundreds of millions to save a few billion for the industry is a very good deal?
I do not known how the US and Canadian University works, but at least what the article said is true in terms of Universities funding in EU, Cooperate Sponsor etc for the work. Well at least it was true in my era of studying.
And I do not have a solution to this problem.
And I guess I cant really blame MPEG, what was originally an outrageous amount $25M now seems fair and reasonable compared to what others were asking. It is likely the greed that every company has for their IP portfolio wanted more.
I think it is still independently developed, but parts of it are included in AV1. Thea idea of lapped transforms in video is really interesting and AV1 isn't using it as far as I know. So hopefully Daala will continue on its own.
I agree with you on the whole lapped transforms concept. It looked quite promising, but they ran into issue. IIRC it was problems with the overlap between differing block sizes. My instinct tells me there is a way to overcome those issues but it may be a while before anyone finds it.
I feel complete opposite to the author. I believe that open soruce patent free codecs will drive innovation and produce better solutions in no time, more then MPEG does over those years.
Anecdote : some video gif on Reddit seems to display incorrectly... It's twisted with some green cream all over.
Video decoding is weird because while the software might be sound, I have no idea how the hardware supports it. I don't know if the hardware has to be specifically built to handle a family of codecs, or if it is a kernel matter or else...
> Companies will slash their investments in video compression technologies. Thousands of jobs will go and millions of USD of funding to universities will be cut.
This is true in theory. Unfortunately, it doesn't prevent people from patenting algorithms in practice.
There's an interesting loophole: just also include in your invention the machine (memory + CPU) running the algorithm. Then you're patenting a device, instead of simply math.
Stallman described this method more than 10 years ago. I was hoping this wasn't true, until the (french) company I work in successfully used it to patent an underlying algorithm from the software I had written.
Maybe such a patent would be debunked in an european court? I'd be glad if this were the case.
This must be an old topic for hn'ers, new to me, but this is incredible, we keep complaining about how stifling software patents are, why hasn't there been an explosion of innovation in Europe simply by having this benefit (has there been?)? Are there studies and comparisons of what has come of this policy?
Well, one question we could ask is why basically every single popular open source multimedia project (FFmpeg, x264, MPC-HC/madVR, mpv, avisynth/VapourSynth, MVtools/SVP, heck even LAME, .....) is made mostly by european authors? Some projects are almost 100% european, some are partially european with some russian and other non-american countries thrown in.
The only projects I can actually think of that are based in the US and made by american authors are those by xiph.org, and they only get away with it because their entire business model is developing royalty free alternatives to MPEG codecs.
Even if multimedia patents might not affect big corporations much, they definitely seem to strongly affect the open source community. I imagine if we had similar dystopian laws here in the EU, our best and most beloved multimedia software would plain not exist.
I wonder why compression seems to be extra sensitive to the existence patents. What about all other fields, aren't there ripe opportunities for European companies to be able to build software that Americans simply can't compete against? Shouldn't there be a cottage industry that does exactly that? What are some of the most stifling non-media patents?
While software patents impact every company's long-term survivability, short-term survivability for startups strongly hinges on investors. European investors are more risk averse and investments in Europe tend to be smaller than in the US. American investors prefer to invest in US companies for obvious reasons.
So a good portion of the startups you're imagining die early due to a lack of investment or incorporate in the US at some point in order to have a better chance of finding investors. And then some of the rest simply aren't as growth-focused because they need to focus on short-term profitability to survive, which means they'll likely end up silently dominating a particular industry niche rather than making a big entrance on the global stage.
Software patents are only a problem if you can survive long enough to be sued. If you're not exceptionally unlucky, you're more likely to go bankrupt before that happens.
“At long last everybody realises that the old MPEG business model is broke, all the investments (collectively hundreds of millions USD) made by the industry for the new video codec will go up in smoke and AOM’s royalty free model will spread to other business segments as well.”
One big issue is that research is getting privatized.
In the past it was common that researchers would go to research institutes such as the Fraunhofer institutes, and do research there. The research would be published, patented, and licensed. From the fees, the Fraunhofer institutes fund themselves.
Now this is being replaced by companies doing the research in-house, where it's often kept as trade secret, or, if it's patented, never licensed to competitors. This limits innovation.
I'm not sure how we can protect the German research societies, but it'd be a shame to see this continue. Research done by companies is inherently unfair, and gives them a competitive advantage.
In this case, of course, Google opened their codecs, but for example with Waymo's self-driving technology, that won't happen. Ideally, every carmaker could license the best selfdriving technology as soon as it exists.
This is relevant to this discussion because the Fraunhofer group has previously created mp3 and AAC, and those patents have funded hundreds of further openly available research projects
The majority of the work was done by the grad students of the professors who are on the standards committee. Private companies then race to patent or buy up anything that's tangentially related during the draft period, so that they can sign on and receive royalties after standardization. Many of these companies are owned by or have board members who are standards committee members. The whole thing is a giant racket.
Licensees were getting tired of this, so they started to do the same thing, and now you have multiple patent pools, no clear guidance on which ones are legit, and even fewer people who are willing to play along.