I saw Vision Pro demos with extending a MacOS desktop into the field of view, and I was highly intrigued, even at that price, as I've been hoping for something like this since Google Glass (which yes, I have). Everything seemed perfect until a reviewer stated that it only worked with a single monitor display, and I was like "what!?" This seemed entirely a waste of the concept!
My normal desktop is 3x 50" TV's I work in front of, which is more or less what my first thought of doing in Vision Pro with their ability to pull in the Mac desktop, but I would want it as multiple displays as well, same as I use my displays today. Only a single desktop seemed unnecessarily limiting, and apparently I'm not alone as someone has even come up with a product (Splitscreen app) to expand on this at least to 2, but it's still hackish and far from ideal.
Maybe this is a Mac limitation, as looking at what sort of Mac I would need to support a Vision Pro indicated I would need M3 Pro to do 2x displays, or a M3 Max for 3x displays. This seemed absurd as I am typing this on a 7-8 year old dell xps driving 3 displays at 4k on an intel gpu, and a brand new M3 needs a top spec Pro or Max to do so?
If I could replicate my 3x desktop experience in Vision Pro, I had honestly considered maybe even trying a MacOS system again (I use Linux for the past 20 years full time), but without at least that, it's all a half-baked tech demo today. Ideally I could even pull in application windows from the desktop apart into Vision Pro space as well, but multiple desktops otherwise would be the barrier to entry.
Here's to hope Collabora/Valve hasn't given up on XR Desktop for Linux VR Desktop integration entirely yet.
Regardless of limitations on the Mac side, there's simply no way to stream data into the Vision Pro fast enough. WiFi can only come close with very heavy compression and a low bar for reliability. One 1080p60 uncompressed video stream is 3.2Gb/s, already well beyond what WiFi can be relied upon to deliver. The existing single 4k desktop streaming is pushing the limits of low-latency, high-quality compression.
You’re forgetting about foveated rendering. You don’t need the 4k60 stream if you’re not looking at it, and you can only look at one screen at once. I think it’s doable.
Applying foveated rendering to compressing video streams would require the Mac transmitting video to have realtime access to the Vision Pro's eye tracking data—which even apps running on the Vision Pro don't get access to for privacy reasons. And with the latency of WiFi plus encoding, the foveated rendering/compression would have to be fairly conservative about resolution changes to avoid distractions from detail popping in your peripheral vision. Dropping the frame rate of video streams the user isn't looking at would probably work very well for streams that are mostly static windows. Overall, it might be workable, but it definitely isn't trivial from a technical or UX perspective.
Maybe he didn’t forget but left this thought out for brevity: foveated rendering and other tricks is certainly possible. But that requires a lot more software development. Apple is launching an MVP. Probably primarily to get developers to start making apps for it. They haven’t had time to develop all the features that would make the AVP great.
I suspect next generation of this solution will stream each window individually, and let you place Mac windows freely around your environment. Then you don’t need to stream anything from static windows, you mainly just need to stream data from the window you’re interacting with at the moment.
> Strapping on a helmet with 4 HDMI cables running out of it
Like 4 HDMI streams can't go into 'one' cable?
My Samsung TV has a box where all the inputs connect, and then there's one cable the diameter of a spaghetti noodle that carries the video signal AND power to the TV.
> Multi-screen Mac use is already an extremely niche market
Because much as I am all-in on Mac, it's a niche market because multi-monitor support on macOS is largely garbage.
And yet my Quest will do 6 screens just fine. You don't need every the content of each screen to update at 60-120fps. You only need it's representation to move smoothly.
If I had a video or game in every screen then yes, I might need the bandwidth. But most screens are mostly static with only the cursor and my input making any changes.
You can have more ultrawide retina displays going than your neck will let you crane your head at.
Only one display needs be hardware, the others are virtual.
PS. Separately, for real displays, most likely your Intel is doing DisplayLink (and most likely not retina resolution), while the Mac is not compressing the video. The Mac can also support a slew of DisplayLink monitors if you get a DisplayLink driver. Elegato and others use the standard driver to run their monitor accessories like Elegato's teleprompter, but you can also use DisplayLink on lower end Macbooks that only support 2 or 3 uncompressed 4K+ retina screens at once as well as on certain docks that split to multiple DisplayLink screens.
My normal desktop is 3x 50" TV's I work in front of, which is more or less what my first thought of doing in Vision Pro with their ability to pull in the Mac desktop, but I would want it as multiple displays as well, same as I use my displays today. Only a single desktop seemed unnecessarily limiting, and apparently I'm not alone as someone has even come up with a product (Splitscreen app) to expand on this at least to 2, but it's still hackish and far from ideal.
Maybe this is a Mac limitation, as looking at what sort of Mac I would need to support a Vision Pro indicated I would need M3 Pro to do 2x displays, or a M3 Max for 3x displays. This seemed absurd as I am typing this on a 7-8 year old dell xps driving 3 displays at 4k on an intel gpu, and a brand new M3 needs a top spec Pro or Max to do so?
If I could replicate my 3x desktop experience in Vision Pro, I had honestly considered maybe even trying a MacOS system again (I use Linux for the past 20 years full time), but without at least that, it's all a half-baked tech demo today. Ideally I could even pull in application windows from the desktop apart into Vision Pro space as well, but multiple desktops otherwise would be the barrier to entry.
Here's to hope Collabora/Valve hasn't given up on XR Desktop for Linux VR Desktop integration entirely yet.