The solution is to learn a world model amenable to A*. Then you can always get from point a to point b. You don't need to follow breadcrumbs generated by random walks, you can just plan your trajectory and go. Similarly if you can decompose the problem into linear control, then you can always find a control law that takes you to your destination.