How common of a practice is it in the Ruby community to run the trunk version? Is it mainly just developers, or is it pretty widely used in production/development environments?
I've never heard of anyone running the trunk except those who are hacking the interpreter. There are fairly regular patch releases so there's no real drive to do so.
This is nowhere remotely nearly as bad as Debian's bug, which broke the underlying library underpinning most of the crypto on Linux, and broke the most universally sensitive component of that library.
Both cases cause key generation to produce weak, easily broken keys. I agree that far less software does key generation with Ruby than with OpenSSL, but that aside the severity of the bug seems quite similar.
While the effects caused by each bug might be on a similar level, I would argue that the fact that this particular bug was never released in a patch (I don't think I've ever seen anyone using Ruby trunk outside of those hacking the Ruby interpreter) makes it less severe.
It'd be pretty bad if it were in any of the stable builds of ruby that people actually use in production. As it stands it's a month-old bug in a bleeding-edge repo, not a huge deal. Nice that someone caught it early!
Could someone please explain to me what this code is doing in the Ruby tree in the first place?
My uses of OpenSSL in straight C don't involve performing this kind of bit twiddling, or really any bit twiddling, calculation, or manipulation at all. That's kind of the point of using OpenSSL.
Since the OpenSSL::PKey::RSA.generate method lets you generate RSA keys with a different exponent there is some manipulation needed between the ruby side and OpenSSL. From my understanding the code was part of a refactor to release the global interpreter lock, if possible, while generating RSA keys.