For point 2. Do you compile to C or is it running as an interpreter? I too have written something similar, but I compile to C++ and yes the speed is amazing, and memory consumption minimal, because, you only keep 1 record in memory (if using more cores then this will not be true).

If you are interested in a solution for entering data, let me know. My solution is open source.