Very cool. I hope we can start seeing WebAssembly-pluggable codecs in all of the video/audio paths in the browser. Imagine being able to turn on a brand-new codec (or select a customized codec for your particular use) in any browser that supports WebAssembly.
Two ends of a WebRTC connection could even negotiate codecs based on power/processing needs, with one sending a WebAssembly codec to the other as a fallback if needed ("hey you don't support this thing that I support in hardware, but you are plugged in, so burn some more electrons on your end").
WebAssembly SIMD [1] will definitely help with performance here.
Although there is much to be desired still on the software end, the molasses of the situation is hardware support for the new formats. You mentioned it briefly, but there's only so much you can do with a software decoders. H.264 may not be ideal or technically have the best performance, but most video cards and cpu's have dedicated hardware decoders.
Completely unrealistic, but it would be interesting to have on the fly reprogramable FPGAs on die that can be assigned specific tasks on demand. This of course has tons of risks, but the pace that hardware moves is painful.
H.265 support is a patchwork in hardware though, look at all the recent Allwinner chips, only 8 bit H.265 playback is supported, despite quite a few H.265 files using 10 bit colors.
Worse yet, most Chromecasts and TVs don't support anything newer than H.264. It will be years before H.265 or any new competitor sees hardware support in most homes.
H264 was the same way early on, with several chips doing baseline or main profile, but not high. It's why I said (at the time) that H264 HW support wasn't important... being able to do Bluray profile (a specific variant of high profile) was what was actually important.
Now with H265, Bluray 4k profile is important, ergo 10 bit support is mandatory, as well as display output itself being able to handle signalling for Rec2020 HDR10 over HDMI 2.x (or equivalent DP).
Probably due to H.265 having much more expensive licensing costs than H.264, given enough volume. That alone might make lots of players in the industry to just prefer ignoring the new codec (and joining a joint effort for developing a new codec with more favorable licensing terms...)
Yeah, VP9 will probably monopolize the market as the libre, royalty free codec. With Google and others boosting it, I'd be surprised if it doesn't win like Opus has in the voice codec arena for any new application.
> Completely unrealistic, but it would be interesting to have on the fly reprogramable FPGAs on die that can be assigned specific tasks on demand.
I've always wanted this! "Hardware accelerated" would become a thing of the past, because everything could hardware accelerate itself. It would probably be more like ram than a graphics card, though, and could probably be designed so each application could specify how many gates it wants.
> I hope we can start seeing WebAssembly-pluggable codecs in all of the video/audio paths in the browser.
Oh great, one more thing to help ad networks to crank CPU usage of my browser to the extremes. Hope I'll be able to also disable that through about:config. :/
> I hope we can start seeing WebAssembly-pluggable codecs in all of the video/audio paths in the browser.
I have a n00b question though. How would I display the video in the browser using webassembly ? I mean using webassembly, I can do the decoding of the bits in the stream. But how to display those bits in the browser ? Do I use a canvas and directly dump the pixel buffer there ?
We already have an enscripten'd decoder, built into a nice analyzer tool that lets you inspect the AV1 bitstream! But you're right, it'll need SIMD to decode at high resolutions in realtime.
They're kinda unrelated. I don't think WebAssembly has explicit SIMD operations yet? Then again, maybe WebAssembly could be delivered in a form which makes autovectorization straightforward for some subset of use cases.
This codec is developed by the Alliance for Open Media, whose goal is to provide a patent-free video codec (more acurately, a video codec under a patent-umbrella protection). Let's not reproduce the HEVC fiasco (see "HEVC advance licensing terms" for more info).
AV1 is still in development at this time (the bitstream format changes slightly almost every day).
However, the compression performance (VQ) is already very good ; and lots of huge companies are part of the Alliance ( Amazon, Google, Microsoft, Nvidia, Mozilla, ... see http://aomedia.org/about-us/ )
Of course Apple is nowhere to be seen. Bad enough their refusal to support VP9 is locking Safari and Apple TV users out of 4K Youtube, it's only going to keep getting worse going forward.
Microsoft[1] and Sony[2] devices support VP9, and there is no reason to believe they won't support AV1 as well. GP was obviously talking about consuming video, so GoPro isn't relevant.
PS4 Pro supports VP9[1] as does XBox One X[2]. The point is that all of these companies now support VP9 in their consumer video consumption devices and will support free codecs going forward. Apple is the only holdout.
Again we don't know if AV1 is royalty free because it hasn't been tested yet. VP8 infringed on a number of MPEG-LA patents and would've required a license. VP9 could likely be the same.
And H.265 has been the industry standard for a number of years now shipping in hardware from Intel, Nvidia, GoPro, Sony etc.
> Again we don't know if AV1 is royalty free because it hasn't been tested yet. VP8 infringed on a number of MPEG-LA patents and would've required a license. VP9 could likely be the same.
On the contrary, we know that nobody has tried to set up a patent pool for VP9 or otherwise ask for royalties. On the other hand, new patent pools for HEVC keep popping up. You could pay one and then be asked to pay another. https://blog.streamingmedia.com/2017/05/velos-media-hevc-pat...
There's no reason to use a Mac anymore unless you're targeting iOS. The performance is better, the tools are better, the battery life is longer (for laptops), the input devices are better, and the connectivity is better for Linux development machines. Switch (TM).
I use Mac OS, Linux and Windows to roughly the same degree. It's all fine.
I spend most of my time developing (server) software in Mac OS, and I'd rather do it in Linux, since it seems obviously more suited to it. I'm glad I don't have to do it in Windows, since it seems obviously less suited to it.
The idea that a Linux laptop even comes close to a Mac one is a pretty laughable statement to make. The driver situation is extremely poor, the trackpads on almost all PC laptops are inferior, the applications are far better on a Mac (i.e. you can't run Photoshop/Office on Linux) and how is connectivity better ?
I've been quite impressed with the 7260AC chipset in the T440 and newer. Places where I used to get 30Mbps down, it'll hold well above 100Mbps, with the same exact 1st gen AC Access Point.
That being said, its worth it to swap a T460 trackpad in to get a decent trackpad. IIRC they're sub-$15.
As someone working in the web live-streaming space I will hold my breath until I see Apple on that list. There's always a party-breaker among the big players when it comes to supporting new tech. And then there we go again transcoding and adding software layers to make it work for everyone.
If it is implemented in Firefox, that means it works on OS X.
Apple is free to continue to devalue their products by refusing to support features, just as its customers are free to spend their money elsewhere.
> And then there we go again transcoding and adding software layers to make it work for everyone.
If that is what you want to do, that will work. If users want to support a platform that limits them to lesser quality, let them. Be explicit about why they are getting lesser quality.
Your software is not part of the Apple ecosystem, Apple is part of your software's ecosystem.
The claim is that it's more efficient than h264, but I assume that's comparing software decoders? With h264 hardware decoding pretty ubiquitous on current CPUs, it seems like it will continue to have a substantial advantage for the foreseeable future. Is the AV1 team pursuing hardware implementations down the line?
Its more efficient than h265, actually. It doesn't have anything to do with decoders, its the encoders that are more efficient (in bitrate/quality metrics, not frames encoded per second).
AV1 has heavy interest from hardware companies: AMD. ARM, Broadcom, Intel, NVIDIA, and Realtek, among others are already members [0].
Ah, thanks for clarifying. Maybe just my bias but when I see 'efficiency' w/r/t media codecs, I presume you're talking about decoder efficiency, because that's much more important for a popular codec IMO.
(Also, down-votes for asking a relevant question. Classy crowd here...)
H.265 hardware decoding has been shipping for at least a year on major platforms. All of those companies you listed ship H.265 hardware today. And then you have companies like Apple and Sony who are H.265 only.
There are TWO patent pools for HEVC, MPEG LA and HEVC Advance, plus patent holders that are part of neither and you have to negotiate on your own.
Even if you are patent holder, you still have to pay and then you will receive back your share. It doesn't mean it is free for you.
On the content producer side, while MPEG LA waives fees there, HEVC Advance does not. If you are not fta broadcast/cable/sat tv provider or free-to-user streaming provider, you are going to cash up per title or subscriber/month.
So in competition, I meant yours. If you are providing a non-free video content, and you insist on HEVC, it is going to cost you. Your competition with VP9 or AV1 won't have this cost.
I can pretty much imagine, that those using devices without VP9/AV1 support will be relegated either to software decoding, or will have to pay premium for their hw accelerated stream.
Whatever the successor to h.265 it will come a few years too late. This has also been VP8 and VP9's problem - they came years after h.264 and h.265, respectively, were on the market.
The difference is AV1 will be here before MPEG-LA's next codec, and now virtually all chipmakers will support it within 8-12 months of the final spec being released, instead of having Google beg chip companies for the next 2-3 years to implement its codec with small/moderate success.
Additionally, I don't think anyone will want to pay MPEG-LA for its next codec, just because it shares a name with h.265.
AV1 can be implemented with free software everywhere. No more leaning on H.264 where you are arbitrarily disallowed from using H.265, not to mention when you are also disallowed from using H.264!
AV1 will outperform H.265, even at low resolutions and framerates. That coupled with free software compatibility, and support from Intel, AMD, and NVIDIA means that it will have great hardware and software support everywhere.
VP8 was patent encumbered.
Nobody knows if VP9 is but it is highly suspected to be.
AV1 won't outperform H.265 when there is no hardware support for it. That's the whole issue here. AV1 is dead when you have major platforms not supporting it e.g. iOS, OSX, PS4, XBox and almost every photo camera.
> Nobody knows if VP9 is but it is highly suspected to be.
{{By whom}}
VP9 is out and widely deployed now, yet no patent litigation has ensued. Meanwhile, there are now three patent pools demanding license fees for HEVC, with the latest one appearing just months ago. If you fear litigation, one option sure seems a lot safer.
No it will never supplant H.265 and it's open but not necessarily free.
The number of companies supporting H.265 far exceeds AV1/VP9 and the biggest wildcard is: Apple. You simply can't have a codec that doesn't work on iOS or OSX. End of story. And you already know who has won by looking at torrent sites which have almost zero AV1/VP9 content but lots of H.265.
And we already know that VP8 was riddled with patent infringements and ended up getting a license from MPEG-LA. VP9 has not yet been tested in this respect and could easily require royalties from implementers in the future.
H.265 does not have a clear path to victory over VP9. Even with only YouTube serving VP9 I suspect it's already much more widely used than H.265. Given the current state of the world it seems more likely that Apple will be forced to support VP9 in Safari than the other browsers being forced to support H.265.
Where have you been ? H.265 is shipping on far more hardware today. It's been shipping on Intel, AMD, Nvidia components for years and is the standard codec for the PS4 and Xbox One. It's also the standard for broadcast terrestrial TV which means set top boxes, TVs etc.
And Apple is a patent contributor to H.265 so zero chance of adopting VP9.
HEVC (H.265) is only used in germany for DVB-T2 (all other countries do use H.264). And DVB-T2 is not in widely use (even DVB-T was a failed product). I guess the only users of DVB-T2 are campers.
(edit: Currently DVB-T2 is only free for open senders (ARD, ZDF, senders which are paid by the "Rundfunkbeitrag", the rest costs money. it's not in wide use and probably never will Astra Sat / Eutelsat or cable (DVB-S2/DVB-C which all only use H.264) is probably the way which most people go, after the DVB-T debacle. And heck even than H.264/H.265 is implemented by DVB-T2 vendors but DVB-T2 does not need H.264/H.265 it can be used with AV1 aswell.)
> While Opus was adopted as a mandatory format for the WebRTC wire protocol, we don’t have a similar mandate for a video codec.
Now if we could play audio in all browsers using Opus. Even Apple started supporting Opus, but being Apple they messed things up and you can't use Ogg container there :(
Very cool to finally see this in a browser. The video continually stops for me though, although audio is still playing. Looks like it's making use of the GPU to decode as well. Both CPU and GPU usage were pretty high for me.
Edit: I think the GPU usage was high due to using WebRender
That can sometimes happen if the CPU gets too far behind. There is a bunch of missing SIMD code at the moment - I'd expect it to be at least 4 times faster by the time it ships.
Two ends of a WebRTC connection could even negotiate codecs based on power/processing needs, with one sending a WebAssembly codec to the other as a fallback if needed ("hey you don't support this thing that I support in hardware, but you are plugged in, so burn some more electrons on your end").
WebAssembly SIMD [1] will definitely help with performance here.
[1] https://www.chromestatus.com/feature/6533147810332672