I recently worked at a company with a working batch legacy system and an ongoing update to Python and SQL with newer tools. It processed large data but it fit in a big server. 3x the amount of people required.
They replaced it with Spark and Scala. Rewrote the whole thing and bumped into the same previously fixed problems for 8 months. It was not faster and it was a pain to extract the result data for the online website. Oh, and they had only 1 server running Spark. On top of the existing one. They also dumped IRC with very useful plugins for proprietary Slack channels.
Yeah, their resumes are very competitive now with the Big Data hype. Worst part is the higher ups liked it because they had more keywords in the bag for investors.
They replaced it with Spark and Scala. Rewrote the whole thing and bumped into the same previously fixed problems for 8 months. It was not faster and it was a pain to extract the result data for the online website. Oh, and they had only 1 server running Spark. On top of the existing one. They also dumped IRC with very useful plugins for proprietary Slack channels.
Yeah, their resumes are very competitive now with the Big Data hype. Worst part is the higher ups liked it because they had more keywords in the bag for investors.
We are living in very strange times.