Too many times I've become very frustrated when an installation doesn't work the first time or it has some dependencies that you haven't installed (or worse, you have a different version). Then you end up in some deep rabbit hole that you can't dig out from. Now for each tool I make, it must have a quick start with a single command.
The world of mock data generation is now flooded with ML/AI solutions generating data but this is a solution that understands it is better to generate metadata to help guide the data generation. I found this was the case given the former solutions rely on production data, retraining, slow speed, huge resources, no guarantee about leaking sensitive data and its inability to retain referential integrity.
As mentioned in the article, I think there is a lot of potential in this area for improvement. I've been working on a tool called Data Caterer (https://github.com/data-catering/data-caterer) which is a metadata-driven data generator that also can validate based on the generated data. Then you have full end-to-end testing using a single tool. There are also other metadata sources that can help drive these kinds of tools outside of using LLMs (i.e. data catalogs, data quality).
I recently went down the rabbit hole of using PyScript for running a Python CLI app in the browser.
It felt hacky the whole time, especially when dependencies were involved. I had to create wrapper classes to work around Pydantic 2.x not being available to use. I tried to put all logic into the Python files but found some things missing that I had to put in JavaScript.
I think it could be good in use cases where you want some simple UI with custom UI logic on top of your Python code but maybe Streamlit or Gradio could be more suitable.
Thanks for sharing. Happy to see another solution that doesn't just slap on AI/ML to try to solve it.
I am also among the many people who have created a solution similar[0] to this :). The approach I took though is being metadata-driven (given most anonymisation solutions cannot guarantee sensitive data not leaking and also open up network access from prod to test envs, security teams did not accept it whilst I was working at a bank), offering the option to validate based on the generated data (i.e. check if your service or job has consumed the data correctly) and ability to clean up the generated or consumed data.
Being metadata-driven opened up the possibility of linking to existing metadata services like data catalogs (OpenMetadata, Amundsen), data quality (Great Expectations, Soda), specification files (OpenAPI/Swagger), etc., which are often underutilized.
The other part that I found whilst building and getting feedback from customers, was having referential integrity across data sources. For example, account create events coming through Kafka, consumed and stored in Postgres whilst, at the end of the day, a CSV file of the same accounts would also be consumed by a job.
I'm wondering if you have come across similar thoughts or feedback from your users?
Working on a data generation and validation tool called Data Caterer. The focus of it is being data source agnostic, fast and simple. Just last week, I released a UI for it.
I think they make the biggest difference when testing data pipelines (which have historically been difficult to test). You can now easily test out compatibility between different versions of databases, verify data types, embed as part of your build, etc.
I believe the next step, once using test containers, would be automating data generation and validation. Then you will have an automated pipeline of integration tests that are independent, fast and reliable.
You can automate data validation with snapshot tests. I do it this way with a data pipeline and have a function that queries the destination DBs and puts them unto json to be written validated with a snapshot
This is what I am trying to solve via building Data Catering (https://data.catering/). It gives you the ability to generate data into any database (along with maintaining any relationships between data) via metadata that can be retrieved via a source database or other types of metadata sources (i.e. Open metadata).
Too many times I've become very frustrated when an installation doesn't work the first time or it has some dependencies that you haven't installed (or worse, you have a different version). Then you end up in some deep rabbit hole that you can't dig out from. Now for each tool I make, it must have a quick start with a single command.
reply