I find there's a bit of an "if you give a mouse a cookie" effect with this stuff.
I recently wanted to organize a data processing pipeline at our company. In the past, the system was: there's a bunch of scripts, some of them call each other, and sometimes a person logs in and kicks off one or more scripts without recording their inputs and sending the output to wherever they feel like. I thought implementing this on Argo Workflows would allow us to keep track of previous runs and generally clarify things.
Well, Argo runs on Kubernetes. Also, it kinda wants your data in a bucket. And argo kubernetes wants kustomize and skaffold and or / helm and soon I'm installing a package manager that manages package managers and I spend half my time managing systems we don't really need.
I'm learning I need to to draw the line and deny Argo some of the things it wants. No, we don't need to run minio and call our files "artefacts." The filesystem works just fine thanks, we don't need another layer of it. This means we can't follow some of the Argo examples and lose some of its functionality.
I recently wanted to organize a data processing pipeline at our company. In the past, the system was: there's a bunch of scripts, some of them call each other, and sometimes a person logs in and kicks off one or more scripts without recording their inputs and sending the output to wherever they feel like. I thought implementing this on Argo Workflows would allow us to keep track of previous runs and generally clarify things.
Well, Argo runs on Kubernetes. Also, it kinda wants your data in a bucket. And argo kubernetes wants kustomize and skaffold and or / helm and soon I'm installing a package manager that manages package managers and I spend half my time managing systems we don't really need.
I'm learning I need to to draw the line and deny Argo some of the things it wants. No, we don't need to run minio and call our files "artefacts." The filesystem works just fine thanks, we don't need another layer of it. This means we can't follow some of the Argo examples and lose some of its functionality.