honestly when starting this project I knew that the rendering is most expensive, thats what I started digging into first and I found webcodecs API, it was like dream come true for my project, but that was just a start of my painful journey :D theres little tutorials how to use it, fortunately most of the issues I faced I could find solution on just github issues. Webcodecs is just decoding and encoding, but part of rendering is also muxing and demuxing, I recommend using ffmpeg wasm for that, or build your own webassembly version which should be leaner if you really need that. Generally the rendering process consists of a canvas that you need to draw things on in right order and place and time. First you demux your video file and decode its frames using webcodecs decoder, decoder is doing its work inside worker, you push those frames as they are decoded to some array and in the same time you draw those frames and other stuff on canvas and dispose/close the frames that are already drawn and are not needed, at the end of each canvas draw you send that canvas to another worker with encoder, encoder is doing its work encoding frames, and you push those encoded frames to some binary array, at the end of the work you just mux those frames and save the file :) I dont know if it makes sense, might not because i know those stuff may sound confusing .. but If you take it bit by bit i assure you will make it

I did watched this video to get some broad idea how all that works, https://portal.gitnation.org/contents/pushing-the-limits-of-...