Yessssssss!!!!! However, we need something that doesn’t rely on iPhone. We need webcam. You can use your iPhone as a webcam. You can also use more powerful video devices as a webcam. I would love a DIY mudface map that’s a b/w displacement map so you can capture the wrinkles of the face and map that with blender trackers. Seriously though, this is a huge leap towards that future.
Is it using structured light / lidar of the iPhone, or just the camera? I don’t know how the project works, but calling out iPhone specifically makes me think it’s using a hardware feature that isn’t in a generic webcam.
This plugin to blender is basically just receiving those values from the OS API and applying it. It’s a fairly common integration and almost all alternatives depend on ARKit on an iPhone as a result rather than implementing any algorithms themselves.
Variations of this plugins functionality have existed since the introduction of the iPhone X in 2017.
the face recognition trick (generating a 3d vertex mesh for the video) should also be doable with a homelab setup. i assume lidar would improve the signal a lot adding factually correct depth values though.
Would it be possible to integrate directly into Blender instead of an add on? If so I think Blender is GPLv2 and this is GPLv3. If merg8ng is something you see in the future you may want to change that.
This is something that is unlikely to be merged into Blender.
It’s not usable standalone as it requires a companion app and a companion device.
If Blender did want to integrate it, there’s nothing novel here that would prevent them writing their own. There’s plenty of similar plugins, and it’s just forwarding events from the companion device.
The place where it would make the most sense to add would be for Blender on the iPad where it would require no companion device at all.
There's a nice Blender extension called "FaceIt" that I used a few years ago for rigging and producing ARKit-compatible facial animations and characters. It worked quite well (for what it was designed), and I recommend it!
>Faceit is a Blender Add-on that assists you in creating complex facial expressions for arbitrary 3D characters.
>An intuitive, semi-automatic and non-destructive workflow guides you through the creation of facial shape keys that are perfectly adapted to your 3D model's topology and morphology, whether it’s a photorealistic human model or a cartoonish character. You maintain full artistic control while saving a ton of time and energy.
reply