You asked about why Python but it has a lot to do with JS too. But first lets compare against Python:
Ruby is a delightful language, but it's dominated by one niche (web development) and one framework (Rails).
Over the last 10-20 years Python has become the main teaching language in Academia. This has greatly contributed to it's ecosystem for math and science, which in turn has contributed to its rise in Data Science and Machine Learning. It also gets a lot of support from Google.
All of these things combined mean that the broad rise of Python is lifting up Python for web development, but if anything I'd say web dev in Python is lagging behind those other uses.
There's one more thing that I think is the root cause of why Rails - the framework - has faded somewhat in the minds of web devs. Rails has always been an opinionated framework, and early on they came up with "The Rails Way" to do front-end assets. The combination of Sass + CoffeeScript + Asset Pipeline with built-in concatenation, minification, and asset hashing was pretty neat compared to the old world of a bunch of es5 files loaded via script tags.
But this hit before the rise of NPM and being able to `require('foo')`.
The JS packaging wars went on for a while and none of those asset solutions are as simple and "just work" as the Rails Asset Pipeline -- but the asset pipeline doesn't hook you into NPM. Rails dragged its feet for a long time on that before adding webpack support.
So I think during that time you had a TON of the trendy hipster open source people moving into Node first on the front end, and then eventually losing interest in Rails because the framework wasn't making it easy for them play in the very latest js ecosystem.
And I think there quickly became some saltyness among Ruby devs about Javascript, that people were overdoing it and that really you only needed "sprinkles" of Javascript to make a great app.
So (1) Python has risen a ton overall, lifting Django, and (2) Rails was slow to embrace the JS craze and that held Rails back.
Those two trends pretty much tell the story as far as I can tell.
Yes, one of the biggest drawbacks in the Ruby world is the lack of strong libraries outside of web dev. Python has scrapy, numpy, pandas, nltk, beautifulsoup, matplotlib, etc. Basically if you want to do almost anything (except high performance real time stuff) you can do it in python. it is an excellent general purpose language.
In ruby, the world isn't so great. When I was in the ruby world, I would always hit roadblocks on my side projects. There were so many times where I wanted to do something not web dev related and I couldn't find a popular and well supported ruby package. For example, I wanted to do text analysis of some sites I scraped and python's nltk was perfect (but there wasn't a perfect ruby library).
The ruby world needs to expand beyond web dev if it wants to grow.
There is only a shortage of scientific libraries, you can do others tasks than web development with Ruby. Ruby is great for sysadmin tasks too for exemple. Many configuration tools was developped in Ruby.
>The ruby world needs to expand beyond web dev if it wants to grow.
I mentioned this in my other post. Ruby is the only language on the top list that doesn't have a major cooperate backing. Basically Ruby community are really lack of resources.
This is a great answer, and to expand on one particular point regarding "lagging behind", Django is not the most popular web framework in Python, Flask is, and by a lot. 24.8m downloads for Django, vs. 58.0m for Flask in the past year.
Additionally, neither Flask nor Django support ASGI, which is the future of how web frameworks will be written in Python. Quart is "Flask for ASGI", and Django has roadmap plans to start moving vaguely in that direction, though they're still a long way off.
Python has also struggled mightily with package management, and how to do it "one right way", which is one of the mantras of Python. Pipenv is a step in that right direction, but it's taking time to get people to adopt, and it's got warts of its own still (though it's getting better every day). I think it's still technically in "beta" (for whatever value that has these days, I am sure plenty of people use it in production).
Let's also not forget the python2 vs. python3 migration problem that I think we're finally now starting to get over. With python2 set to depreciate in 2020, the clouds are starting to lift on the major problems in Python, so naturally you're seeing a rise in converts.
Ruby is a delightful language, but it's dominated by one niche (web development) and one framework (Rails).
Over the last 10-20 years Python has become the main teaching language in Academia. This has greatly contributed to it's ecosystem for math and science, which in turn has contributed to its rise in Data Science and Machine Learning. It also gets a lot of support from Google.
All of these things combined mean that the broad rise of Python is lifting up Python for web development, but if anything I'd say web dev in Python is lagging behind those other uses.
There's one more thing that I think is the root cause of why Rails - the framework - has faded somewhat in the minds of web devs. Rails has always been an opinionated framework, and early on they came up with "The Rails Way" to do front-end assets. The combination of Sass + CoffeeScript + Asset Pipeline with built-in concatenation, minification, and asset hashing was pretty neat compared to the old world of a bunch of es5 files loaded via script tags.
But this hit before the rise of NPM and being able to `require('foo')`.
The JS packaging wars went on for a while and none of those asset solutions are as simple and "just work" as the Rails Asset Pipeline -- but the asset pipeline doesn't hook you into NPM. Rails dragged its feet for a long time on that before adding webpack support.
So I think during that time you had a TON of the trendy hipster open source people moving into Node first on the front end, and then eventually losing interest in Rails because the framework wasn't making it easy for them play in the very latest js ecosystem.
And I think there quickly became some saltyness among Ruby devs about Javascript, that people were overdoing it and that really you only needed "sprinkles" of Javascript to make a great app.
So (1) Python has risen a ton overall, lifting Django, and (2) Rails was slow to embrace the JS craze and that held Rails back.
Those two trends pretty much tell the story as far as I can tell.