Argo is pretty amazing if you want to take advantage of the work Kubernetes has done to scale resource efficiently across a cluster of compute nodes.

If you’re looking for something that’s a bit more high level and friendly to expose directly to your data team (data scientists/data engineers/data analysts) you can check out https://github.com/orchest/orchest

You can think of it as a browser UI/workbench for Argo scheduled pipelines. Disclaimer: author of the project