The smart speaker is always fixed in place, always listening, always ready to play and independent of anything else.
Your phone needs to be in range of the 'dumb' speaker, needs to be paired to it / connected to the same wireless network, can't be used for any other audio functions, and probably can't be in your pocket if you want it to be listening for voice commands. It now also can't be trained to your voice specifically if you have others in your home who want to use it, which makes it ripe for abuse whenever you leave the home.
Your phone needs to be in range of the 'dumb' speaker, needs to be paired to it / connected to the same wireless network, can't be used for any other audio functions, and probably can't be in your pocket if you want it to be listening for voice commands. It now also can't be trained to your voice specifically if you have others in your home who want to use it, which makes it ripe for abuse whenever you leave the home.