You don't need self driving cars for such a scenario to happen -- cars are increasingly drive by wire, and driver assistance features being added to cars (automatic lane keeping, automatic breaking, smart cruise control, etc) mean computers are already capable of taking over cars.
I've come to that realization when driving my Leaf.
You see, with an internal combustion engine, there are several ways that you can stall the engine, even if the computer is controlling it. As long as you can stop it from rotating, it will stall.
Now, take a Leaf. The engine can't physically stall. It is completely controlled by electronics – in contrast, even an ICE with an engine control unit will have some of it being driven mechanically (valves and driveshaft are all mechanical). This also causes cars to "creep" when you release the brakes, as the engine has to keep rotating. In the Leaf, the "creep" exists, but it is entirely simulated.
Similarly, the steering is also electric and controlled by algorithms (more assist in parking lot, less in the highway).
Braking is also software-controlled. The first ones had, as people called them, "grabby breaks" (it would use regenerative breaking with a light force, if you pressed more, the breaks would suddenly "grab" the wheel). This was fixed in a software update.
Turning on and off is also a button. Can't yank the keys either.
So yeah, presumably, a Leaf could turn on, engage "drive" and start driving around, all with on-board software. It lacks sensors to do anything interesting, but the basic driving controls are there.
