Estuary Flow [1] may be interesting to those in this space.
We're still building, but it's a GitOps workflow tool that tightly integrates schema definition (JSON Schema), captures and materializations from/to your systems & SaaS, rich transformations, catalog and provenance metadata tracking, built-in testing, and a managed runtime. All with sub-second latency.
Flow's runtime uses nascent but really promising open protocols for building connectors to the myriad systems and APIs out there. We're seeing Airbyte's work (itself built off of Singer) as the best steps in this direction and are leaning into that effort ourselves.
Feel free to throw in the data mesh community Slack[1]. There was an interesting approach that sounds kinda similar re schema contract management from FindHotel that they posted a few weeks ago re data mesh[2].
Yep. You can, for example, have it watch file(s) in S3, and every time a file changes it will flow its records through into a table it creates in your DB, keyed on your (arbitrary) primary key.
We're still building, but it's a GitOps workflow tool that tightly integrates schema definition (JSON Schema), captures and materializations from/to your systems & SaaS, rich transformations, catalog and provenance metadata tracking, built-in testing, and a managed runtime. All with sub-second latency.
Flow's runtime uses nascent but really promising open protocols for building connectors to the myriad systems and APIs out there. We're seeing Airbyte's work (itself built off of Singer) as the best steps in this direction and are leaning into that effort ourselves.
[1] github.com/estuary/flow