It’s been a couple of months since Apple announced their Vision Pro headset, and it’s at least six months before most of us will be able to try it for ourselves. According to media who have tried it, one of the standout features is how video is presented. That includes not just standard 2D video in a huge virtual environment, but true 3D video. The headset itself can record 3D video which is — reportedly — so realistic, it’s like watching a memory. And it’d be a shame if this new, immersive experience remained firmly in the land of home video, wouldn’t it?
From a video production point of view, there’s a chance that we could see an uptick in demand for more immersive 3D 180° videos, but… hasn’t 3D been tried before, more than once, across multiple delivery methods? Sure it has. But there are a few reasons why it might be different this time. Let’s prognosticate.
Delivery is key
For something to be popular, it has to be easy, and 3D is not easy. In the past, you were able to buy 3D TVs or 3D capable projectors, but almost all of these required all viewers to wear special glasses, and most sacrificed either resolution or brightness to deliver the 3D effect. After all of that, it was very difficult to find any 3D content to play. The PlayStation 3 had some, and you could purchase some 3D Blu-ray releases, but streaming 3D movies never really went mainstream. Since 3D TVs are no longer being made, the audience is now too small for studios to care, and content is scarce. The only recent release on 3D Blu-ray is Avatar: The Way of Water, which is capped at 1080p and spread over two discs.
VR headsets, while being only a personal experience, do offer something new for 3D content consumption. Although Meta’s Quest and the earlier Oculus headsets have been around for many years, there’s still only a small range of 3D movies available from smaller marketplaces, and the user experience (judging by the app ratings) is pretty poor. Although resolution has been an issue with most headsets in the past, distribution of content is probably a bigger issue, and Apple is uniquely placed to solve both problems.
The Apple Vision Pro avoids the technical issues of past equipment by putting bright, high resolution screens right on your face. While it’s obviously an even more expensive type of special glasses, and this isn’t likely to be a shared family viewing experience for a while yet, distribution will be far easier than any prior solutions. Apple has their own content creation (Apple TV+) and studio-ready distribution (iTunes Store), Disney is on board with Disney Plus promised on day 1 and plenty of 3D-ready content, and the App Store will take care of the rest.
3D content creation isn’t straightforward
Even though the vast majority of movies are filmed in 2D, 3D versions of many popular movies are still being produced. These are shown in cinemas in markets where 3D is still popular, even though very few are released on 3D Blu-ray discs. But Apple can certainly deliver 3D content over streaming if they want to, and if the studios are keen.
While it’s not likely that movies will shift to full stereoscopic capture any time soon, the technology used to separate the different parts of the image for 3D depth separation is becoming more mainstream. With AI-based techniques, computers can segment 2D images more easily, and iPhones have been able to use LiDAR for this purpose for years already. The same tech that helps Cinematic Mode blur out a background can be easily repurposed to bring depth to a flat image, and on-the-fly 3D conversion is possible today.
Still, native 3D creation remains a whole lot harder than 2D. I was surprised to discover that one of the reasons why PlayStation-based 3D gaming never became mainstream, despite games being made in 3D engines, is that 2D techniques are still used heavily. A puff of dust, for example, is far easier to implement as a 2D texture than a 3D point cloud, and it just looks wrong when viewed in 3D. Moving every asset into true 3D space wasn’t worth the effort for its limited audience, and the same is true for VFX in filmmaking. Compositing is still non-trivial in 2D, but a much bigger challenge in 3D.
If you’ve become used to painting out things you don’t want in your videos, you’ll probably have to wait for your tools to be 3D-capable before you can jump up. Creating 3D content is going to be much easier for amateurs (who don’t use VFX) and animators (who are fully in 3D space already) than for most professional filmmakers. The high end can throw money at the problem and create great 3D content from their great 2D content, but creators beneath the studio level will need more capable tools to be able to embrace 3D.
Software updates we will need
All our NLEs will need some updates to be more 3D-friendly, but this might not be so hard. Both FCP and Premiere Pro have 360° video support at a reasonably high level, and stereoscopic 360° footage largely works today. Note that while 360° cameras are popular for capture, 360° delivery is niche, and likely to remain so. When a viewer can look anywhere they want, it’s difficult to hide your crew, difficult to move a camera, and difficult to deliver a quality image. Worst of all, it’s hard to tell a compelling story in 360°.
That’s OK, because 360° footage isn’t the main focus of the 3D footage that Apple’s trying to promote. Both the the Avatar 3D feature films and the personal 3D home movies are 3D presented in a frame, with a field of view that’s less than 180°, and this is much more like traditional filmmaking.
While 180° 3D movies are technically possible to deal with in both programs, we need more. Though the 3D text support in Motion and After Effects is quite good, we’ll need to see the same support inside FCP and Premiere Pro if we want to be able to position text in 3D space, let alone control the 3D positioning of our characters, or intelligently move 2D content into 3D space. We’ll also need more capable spatial audio support.
One big plus is that Apple is taking 3D video delivery seriously, and have created a new stereo extension to HEVC that’s designed to handle 3D content.
This new format is smart enough to compress one eye’s image based on the other eye, and to deliver just one eye’s image to 2D devices, which allows you to deliver a single file to every device. To date, 3D delivery has been cumbersome and inefficient (sacrificing resolution or file size) but this is a big step in the right direction.
More 3D cameras would be nice
Because not every filmmaker will want to record their masterpiece directly on the Apple Vision Pro, we’ll need more 3D-capable cameras to be able to capture in native stereo. There have been a few attempts at this in the past, but they’ve all been pretty niche, and have been discontinued. With luck, some forward-thinking manufacturers will re-introduce some 3D lenses for regular cameras (Panasonic used to make one) or software tools will get a whole lot better at faking it.
The danger in faking it is that it’ll be too easy to make bad 3D content, and bad content has sunk many previous attempts at 3D. Few filmmakers want another experience like The Hobbit in 3D HFR, in which the film looked like a video game, but 3D doesn’t have to be a gimmick. With care, I think we can take a great film, add depth, and be better for it.
Conclusion
To be clear, there’s no guarantee that VR headsets will ever become truly mainstream, but if Apple Vision Pro does lead to a a cheaper range of future headsets, it’s going to be much easier for more people to enjoy 3D content. And if there’s more 3D content available, and people are enjoying it, maybe we’ll finally kick off a virtuous cycle of 3D-capable headset, displays, cameras, and everything else. The foundations are laid, and the tech is close.
We’ve been here before, but I very much look forward to finding out if this time is any different.