If I remember correctly Google open-sourced an ML toolkit to do just that a few years back, I forget the name but someone spun up an online sandbox with it that lets you experiment with combining different sample sources

edit: found it here: https://experiments.withgoogle.com/sound-maker