I haven't really done a whole lot with Vulkan yet but this really makes me curious if Vulkan could ever find itself as an alternative to VAAPI on Linux in some cases. Vulkan for compute purposes already seems like a pretty compelling idea, as it is, but in both cases I wonder, but was unable to ascertain an answer with a cursory glance, if it is possible today or maybe will ever be possible, to expose devices to Vulkan that are not GPUs (like compute accelerators.) Perhaps it's not even really a good idea :) The case is probably stronger for compute devices a la OpenCL, since at least for video, if you're targeting Linux anyways, it seems like you could use dma-bufs to pass buffers between Vulkan and VA-API and other stuff. Still, it does inspire curiosity.
A cursory search did seem to suggest encoding was supported as well as decoding in Vulkan, which is very cool.
On Linux there is already an alternative to VAAPI: The Video4Linux Memory-2-memory (V4L2-M2M) API, which provides a standardised API for hardware encode and decode accelerators (and supports using DMA-bufs for copy-free decoding or using decoded frames as textures). I don't know how much support there is for AMD/nvidia/intel but it works well on the Raspberry Pi at least.
It would feel weird for Vulkan to become the standard API for encode/decode given that (as you say) video accelerators aren't necessarily tied to the GPU (although not necessarily weirder than doing it through an extension of the webcam API).
Truth told, Raspberry Pi is the only device I've ever used v4l2 to do video encoding with. It does seem to work well enough, but since I was using it via GStreamer, I have no idea what the API looks like. The advantage of Vulkan is that they've already got a functional and productive committee with all of the GPU vendor stakeholders to work out the design, whereas VA-API and VDPAU seem to have somewhat different designs and seem to be mostly championed by a single GPU vendor. I have a bit more confidence that the Vulkan designs for hardware accelerated video will be good for both hardware vendors and application developers (even if it is still low-level.)
In a neighboring comment thread, hrydgard states that it is very much possible to have a Vulkan driver that has only video encoding/decoding queues, which if true resolves the potential awkwardness in my opinion. That's especially relieving if true as well since Vulkan looks to be a prime contender to take up the mantle for OpenCL as well; especially believable now seeing what has been done with rusticl.
Perhaps it is still a little awkward, but Vulkan as a general hardware acceleration abstraction layer seems like it could be a very good thing. As it is, as an application developer, you're probably better off on Linux just using GStreamer or Pipewire rather than v4l2 directly, since some devices (e.g. Blackmagic Decklink) provide GStreamer integration but not v4l2 drivers. In that case, it hardly matters what API abstracts the hardware as long as it does a good job of it.
Hmmm, interesting: maybe the last bit is the key to working around the issue that Vulkan drivers can really only exist for GPUs (to my knowledge, anyways). You could still have VA-API drivers for devices other than APIs, alongside a Vulkan implementation.
> if Vulkan could ever find itself as an alternative to VAAPI
The bar to beat VAAPI is really low. For all practical purposes, VAAPI doesn't exist. So any alternate plan, even if it is essentially the same thing underneath, is worth a shot.
except all the practical usage that a lot of linux users have. using your logic, linux on the desktop doesnt exist, and yet it does. Yes, much fewer use it than windows, but those who use it do still use it
My single desktop experience is that no matter what I do with VAAPI on my netbook (the only device left with GNU/Linux), I only get software rendering on Chrome/FF.
That OS-supplied high-level video player object can demux containers, decode, play audio, and copy uncompressed frames to D3D11 textures. Then, it’s relatively easy to use DXGI surface sharing API to share the resulting RGBA8 frames from D3D11 into D3D12.
Yeah. I was trying to do something like render a scene and encode it on-GPU before sending it to disk. Unfortunately, there’s no way to give FFMPEG video textures afaict so even if it uses hw decode, you’re copying the full pixel buffer data twice instead of the drastically smaller encoded frame once (or at least there’s no mechanism exposed that the JS ecosystem can make use of which relies on command-line ffmpeg)
If you use ffmpeg’s libavcodec interface then you can get it to give you the decoded framebuffers as exported DRM-prime descriptors which you can turn into textures. This is how Firefox does video decode with VAAPI, using libavcodec as a wrapper.
Edit: missed the part about JS ecosystem. You can move DRM prime descriptors between processes, but I assume you can’t do this from the ffmpeg CLI and would need to write your own little C wrapper around libavcodec
> Take a look at FFMPEG, the most popular and feature complete video library out there: it’s huge, consisting of several DLL files and lots of dependencies.
What? You can compile it with just the stuff you need and only activate stuff without any dependencies. Static linking is also possible if too many 'DLL' files is a issue.
> What? You can compile it with just the stuff you need and only activate stuff without any dependencies.
Sure, but running config alone takes like 10-15 minutes on my PC with an Intel I7-9700K processor at 4.9GHz. This is while using all my cores at 100% CPU usage too, it's ridiculous. And not only that, you'll likely have to run this massive script several times to tweak the config until you're happy with what you're using. Then actually compiling the library statically takes another 15-20 minutes. So I think the remark that it's huge and consisting of several DLL files is spot on.
> Static linking is also possible if too many 'DLL' files is a issue.
FFmpeg doesn't recommend static linking.
> The following is a checklist for LGPL compliance when linking against the FFmpeg libraries. It is not the only way to comply with the license, but we think it is the easiest...Use dynamic linking (on windows, this means linking to dlls) for linking with FFmpeg libraries.[0]
Correct me if I'm wrong: when we decode a video we either use software decoding via the cpu or we let a specialized hardware module (usually inside a gpu) handle the decoding in an efficient way. Using Vulkan to decode video basically means writing a decoder in a special way as to accelerate it using the gpu instead of the cpu, which makes it more efficient and faster than cpu decoding and allows older gpus to decode newer formats. Of course it will always be slower and less efficient than the hardware accelerated module but it is more flexible and can actually be updated
Vulkan is primarily a GPU API, but it relatively recently gained a hardware-accelerated video API called Vulkan Video, which is the topic of this post. That it's part of Vulkan is perhaps because video decoders are often bundled with GPUs, though that's not always the case and I believe Vulkan Video can be implemented also where it's separate from the GPU (common on mobile).
"built-in fixed function video unit", "hardware decoder" -- that just means the decoder of each supported codec is physically present as an ASIC, right?
I wonder if this will be a path towards crashless video decoding in web browsers*. Probably not though :(
* modern web browsers have incredibly complex behavior to try and detect whether your system's video codecs have crashed, since they can and will, and then if they crash too many times the system decoder is disabled and either replaced with a software decoder or nothing at all
A cursory search did seem to suggest encoding was supported as well as decoding in Vulkan, which is very cool.