The rolling shutter compensation is pretty cool and isn't something I would have thought of. Did you know that would be an issue from the start or notice it only after you built the rest of the system?
I knew it would have an effect (most of the literature for similar projects just uses global shutter cameras for this reason), but wasn't sure how significant it would be. It turned out to be small enough that it usually wasn't super noticeable, but in certain cases it really showed up (e.g., rotating the pen while keeping the tip in one place).
The thing I was most surprised by was how effective my solution was, given that it's a pretty gross approximation of reality. There are lots of much more sophisticated techniques for dealing with it, which I didn't end up needing.
One thing I would've liked to try out is using rolling-shutter-aware PnP [1], which can theoretically estimate pose and velocity simultaneously from one image, by exploiting the rolling shutter distortions.
