One thing that’s not clear to me (and maybe this has to do with how vp8 works?). In h264/h265 you have I frames (which are standalone) and P frames.
If I recall correctly, P frames require having decoded the immediately preceding frame or video corruption occurs until a new I frame is received.
With SFU selectively choosing which resolution stream to send a frame from next, how does this work? Is the client encoding each frame as an I frame? Is there feedback from the SFU to tell the client which stream its using so that the client knows to emit I frames (and thus generating a few corrupted frames until the stream is resynced)? Does vp8 work differently?
If I recall correctly, P frames require having decoded the immediately preceding frame or video corruption occurs until a new I frame is received.
With SFU selectively choosing which resolution stream to send a frame from next, how does this work? Is the client encoding each frame as an I frame? Is there feedback from the SFU to tell the client which stream its using so that the client knows to emit I frames (and thus generating a few corrupted frames until the stream is resynced)? Does vp8 work differently?