Instead of sandboxing the entire app, you should use privilege separation inside VLC. Low-risk code would run in processes with higher privileges, while the high-risk code (e.g. the decoders) would run in highly sandboxed processes and communicate with the privileged processes over a simple interface. This is the approach used by security-conscious server apps (e.g. openssh, postfix, dovecot) as well as Google Chrome, and I think it has more promise than trying to sandbox entire apps. Many apps have complex access requirements (much more than the game used as an example in this blog post) and trying to sandbox them will require granting so much privilege that the sandbox becomes almost meaningless.
I'm currently working on converting a PDF viewer to use privilege separation. I'd love to see a media player use privilege separation too. Does VLC currently have pretty clear separation between its various components? Do you think it be much work to spin security-conscious parts like the decoders into separate processes?
> Does VLC currently have pretty clear separation between its various components?
Very clear separation. One of the best, tbh.
> Do you think it be much work to spin security-conscious parts like the decoders into separate processes?
Extremely difficult. We've thought about it.
For a video player the 3 parts that are sensitive, are protocols (file, http), demuxers (mkv, avi) and decoders.
The crashes mostly happen in protocols and demuxers, but not in decoders (a contrario from what people think).
The main issue is that the video decoder MUST be in the same process than the video output, for performance reasons (buffer sharing: memcpy is murder) and for hardware decoders. And video output are usually with very high access in the kernels. Moreover video outputs are almost necessarily in the process with the UI thread.
For audio output, it's not good either, although some platforms are better (pulseaudio+Kdbus might work).
The main issue is that the video decoder MUST be in the same process than the video output, for performance reasons (buffer sharing: memcpy is murder)
Why not used shared memory? I believe that's what Chrome uses for its rendering buffers. The renderer is sandboxed and shares buffers with the X11 process.
Still, it appears it used this method for a long time, so it seems feasible.
Also, Chrome has multiple video playing paths, including Flash, which is sandboxed, so I don't see the difference. The Flash video has to be decoded in a sandbox. I imagine HTML5 video is sandboxed somehow too.
It is very platform-specific and difficult, and performance is a prime concern, but not impossible.
Forgive me for a short rant, but this sort of thing is one reason I think other kernels should look at OS X's virtual memory interface for inspiration. There, you can always share any existing mapped page with another process, regardless of how it was allocated. Usually useful because it reduces the need to redo the way you allocate some (regular) memory just because you want to use it a certain way (sharing), but it works with driver-mapped pages too.
Sharing a normal memory mapping is not necessarily what is required here though. What you want is to share the same GPU-side memory buffer between two processes. It may not even be mappable to cpu side memory.
If it's mapped into the CPU address space, then it can be shared. If not, then there is a copy going on and you may as well just do it in the non-sandboxed process, no?
There is no copy. You have a reference to a GPU side buffer, and you want to pass a reference to that buffer to another process, where it will continue to use the GPU to access it.
> The main issue is that the video decoder MUST be in the same process than the video output, for performance reasons (buffer sharing: memcpy is murder) and for hardware decoders.
I was afraid that this was the case. Might shared memory be a viable solution to buffer sharing?
In any case, it makes me really happy to know that you've thought about this!
> The main issue is that the video decoder MUST be in the same process than the video output, for performance reasons (buffer sharing: memcpy is murder) and for hardware decoders. And video output are usually with very high access in the kernels. Moreover video outputs are almost necessarily in the process with the UI thread.
Did you look into the new memfd support in the kernels? Seems like this is a pretty cool way to do process splitting while sharing memory in a "safe" way. (i.e. the recieving side can guarantee that the sending side is not playing silly-buggers with the buffers while you read them).
For hardware decoders, maybe using dmabuff buffers shared between multiple processes would work?
Keep in mind that privilege separation does not cover all possible use cases of sandboxing.
Privilege separation is an answer to the question: "I have a program whose authors I trust even though they are fallible, and the program needs to process untrusted data"
In desktop scenarios, I really want an answer to the question: "I have a program whose authors I don't trust, but I want to run it anyway"
For that, you really need full sandboxing; privilege separation is not enough.
I'm currently working on converting a PDF viewer to use privilege separation. I'd love to see a media player use privilege separation too. Does VLC currently have pretty clear separation between its various components? Do you think it be much work to spin security-conscious parts like the decoders into separate processes?