I actually want that kind of AI, as long as I'm in control of it and it runs locally. I want a great foreign language tutor. I want an assistant who will figure out what I should be doing today to work towards the things that I want. Why wouldn't I? And there's no way you get those things without creating some kind of dependence. The more transparent AI is, the more I can train it and tune it myself, the more it will conform to my life, and paradoxically the more dependent I'll be upon it.
The big fear of AI is that it will be used to make people conform, but the ability for it to conform to us would embed it even deeper into our lives.
We already give a fair amount of control of our future over to a variety of systems. As long as the AI system is under full control, operated safely/locally and seen not as a boss, but an assistant or advisor, I see no issue with that.
The big fear of AI is that it will be used to make people conform, but the ability for it to conform to us would embed it even deeper into our lives.