I think it’s obvious that the current crop of models are just not quite there yet. The ability to give them “tools” that interact with the things we want to interact with is just going to take time. Ideally it would be a model that just “gets it” similar to the ideas Adept AI are pursuing by interacting directly with the UI.
The bigger issue with this thing though… why does it need to be its own device? It’s not going to replace your phone so why not just have this all happen on your phone?
For AI to augment the lives of to the extend the iPhone did it's essential for it to be always on listening and able to act effortlessly.
Only Apple, Google and major android makers can deliver this experience.
There is however a window of opportunity for a team with the right talent to get there first if they're able to build their own device in time.
Apple are too privacy conscious to send all the data up to the server so we need to wait till they can build chips to do that locally or figure out a way to bend their own rules enough that makes it seem privacy focused, they also have a much weaker ML team so there is extra runway there while they choose who to acquire to fix that.
Google while extremely strong ML team its too academia brained to productize AI currently so need to wait for them to solve that, they also just suck at shipping products in general. They'll get there in the end but it's safe to say they'll only get there once someone else has shown how it should be done then they'll just clone it.
You have about 3 years before Apple solves this, so if you get yours to market and succeed in that time you capture a segment of the market before that happens.
Some YouTuber talked about this and I think they were pretty on point: Of course for consumers this could all happen in some app on the phone.
But a 3rd party app will always be less integrated, have less permissions than functionality included by the manufacturer.
And for all this AI integration wide access is pretty much required as you'd want it to access your photos, notes, all kind of apps, etc.
This way manufacturers would have too much leverage over companies developing that kind of AI, as they could always develop better features than them with their own AI agent.
I think Apple Watch is a pretty good example of that already. Third party watches will never be as good as Apple Watch just because Apple won't let them.
Apple and Google will do that, no one else will be allowed to. Even if you could get the necessary device permissions (you can't), you're going to get sherlocked and be dead next year when Apple and Google bake whatever interesting thing you did into the OS and all apps get it for free.
You're at such a disadvantage on iOS and Android that it's a fools errand to try and build that app.
The bigger issue with this thing though… why does it need to be its own device? It’s not going to replace your phone so why not just have this all happen on your phone?