Using sun.misc.Unsafe to access array elements would make it much faster.
sun.misc.Unsafe should not make any difference to performance here. OK, you skip bounds checks, but modern VMs are very good at optimising those away anyway.
Also ArrayList with preset size is faster for inserting elements.
Exactly what I was thinking. The only case this is really going to help is where you don't have an idea of what the size is going to be.