I need to handle (for work) a graph with 40 million nodes and more than 130 million edges.
As expected, networkx couldn't handle more than a million nodes so I had to search for python libs which might handle that much data.
Having a graph with several million nodes isn't just some edge case, social graph for instance grow way faster than anyone could expect.
So I do totally agree that there is a gap between popular graph libs which handle very small graphs and real life libs which need to handle way bigger graphs.
Btw, thanks for the work you've done with 'nodevectors'.
PS: I'm not criticizing either networkx which is very handy and quite good when prototyping a solution.
nodevectors implements many more algorithms than fastnode2vec that focuses on node2vec.
I'm pretty sure fastnode2vec is the fastest node2vec implementation because it uses CSR format, JIT compilation but also supports multiprocessing. The sampling algorithm has been improved compared to the paper (and to all other implementations) and allows truly linear memory consumption.
On the other hand, nodevectors has a lot of very cool methods like ProNE so you should definitely try it on your data. However, the original ProNE code is probably faster as it is written in C++ and uses multiple cores while nodevectors just uses the installed BLAS (you can probably get a massive speedup by installing OpenBLAS).
This is why I've been using your lib (https://github.com/VHRanger/nodevectors) for at least 2 weeks now as well as these 2 other libs: https://github.com/louisabraham/fastnode2vec and https://github.com/sknetwork-team/scikit-network. What do they have in common? They handle sparse graphs (using CSR representations).
Having a graph with several million nodes isn't just some edge case, social graph for instance grow way faster than anyone could expect. So I do totally agree that there is a gap between popular graph libs which handle very small graphs and real life libs which need to handle way bigger graphs.
Btw, thanks for the work you've done with 'nodevectors'.
PS: I'm not criticizing either networkx which is very handy and quite good when prototyping a solution.