In Ubuntu i can do apt remove to just uninstall the binary or i can apt purge to remove config files as well. Is there an native macOS equivalent for this?
The package manager will never attempt to clear stuff under user's home directories, just like macOS will not clear stuff under ~/Library. The package manager doesn't and cannot know what content an application has created while in use. Only the application itself knows that.
Flatpak applications under Linux get their little sandboxes under ~/.var/app/$application-id where they can create whatever they want and the user can remove it from there when needed. Except for that, any other path that the app can access is decleared in the permissions; or the user is very well aware, because he picked the file via filechooser (portal).
Soma macOS applications can use similar scheme, under ~/Library/Containers/$application-id, but for the user more difficult to know which one is supposed to and which isn't.