At the moment, the XR community is buzzing with discussions about passthrough camera access. With Meta, Apple, and Pico having already declared their stances, all eyes are now on Google’s plans for Android XR. After chatting directly with Google, I’m excited to reveal they’re planning a solution quite similar to what you’d experience on smartphones. Stick around to find out more!
The Camera Access Conundrum
Let’s rewind a bit for clarity’s sake. The latest standalone VR headsets are more accurately MR (mixed reality) headsets, using front cameras to provide users with an RGB passthrough view of their actual surroundings. This technology is what powers fantastic mixed reality apps like Cubism, Starship Home, and Pencil.
The system leverages the camera-captured frames to deliver the passthrough experience. As developers, gaining access to these frames would let us use AI and computer vision to analyze the user’s environment, potentially transforming their reality in significant ways. I’ve said it before: true mixed reality hinges on camera access because it lets applications become fully context-aware. For instance, thanks to a workaround on the Quest, I was able to build an AI+MR app to assist with interior design—a project that wouldn’t have been feasible otherwise.
Yet, alongside the excitement comes a significant concern: privacy. Imagine a rogue developer gaining camera access—they could discreetly capture images of the user’s environment, run AI detection, and harvest sensitive information, like ID docs or bank cards lying around. It’s even possible to misuse personal images of faces and bodies.
Balancing user privacy with the unleashing of mixed reality’s full potential is a tightrope walk. It’s essential to both safeguard users and empower innovation.
How XR Companies Are Navigating This
In the early days, full camera access was freely available. If you’ve been following me, you might recall the experiments with camera textures on the Vive Focus around 2019. We explored various aspects like diminished reality and sound reactivity.
As MR started gaining traction, companies became cautious and restricted camera access, fearing privacy implications. Companies like Meta, Pico, HTC, and Apple ended up locking down camera frames for developers.
This policy became standard until XR developers realized the importance of having such a feature and began advocating for camera access. People like Cix Liv, Michael Gschwandtner, and I have been vocally calling for transparent and user-consented access to camera frames, allowing us to employ AI for object recognition and other tasks. We questioned why XR devices were subjected to such scrutiny when phones, deeply integrated into our daily lives, allow camera access with permission requests.
This advocacy prompted change. Meta, for instance, is set to release a “Passthrough API” this year. But the big question remains: what’s Google’s stance on Android XR?
Android XR: Treating the Headset Like a Phone
Android powers a vast majority of the world’s smartphones. Typically, Android app developers can request camera access with user permission. Upon obtaining consent, they can use specific camera IDs (like ID=0 for the rear camera) to capture frames.
Google intends to bring this familiar approach to Android XR. Though rumors abounded, I had a direct email confirmation from Google. Here’s a snippet that clears things up:
Similar to any Android app, a developer can use existing camera frames with user permission for XR. Our developer article outlines other permissions apps can request. A developer can access the world-facing (rear) camera using camera_id=0 and the selfie (front) camera using camera_id=1 via standard Android Camera APIs like Camera2 and CameraX.
The ability to use classes like CameraX on Android XR headsets is fantastic news, as these tools let developers capture frames, save media, or perform ML analysis.
While access to the front camera is straightforward, the rear camera operates by translating data into a user avatar, akin to Apple’s Vision Pro approach. This method keeps Android XR’s operations consistent with smartphone behaviors, aligning the user experience across devices.
Ensuring Compatibility with Android Apps
Google’s strategy is to ensure seamless functionality of Android apps on Android XR. Thus, the adaptations mentioned should ensure apps maintain their functionalities, even when accessing cameras. It’s a clever move, keeping permission protocols consistent between phones and XR headsets.
Some questions linger, such as whether all raw camera streams will become accessible down the line. Unfortunately, the current stance isn’t promising for non-standard streams like forward-facing or reconstituted inward cameras—but there’s hope for enterprise users.
Unity developers might wonder how this impacts them. Although Android Camera2 and CameraX are native classes, Unity users might leverage the WebcamTexture class to access camera frames. Should challenges arise, creative solutions like JNI wrappers might bridge the gap for unity developers.
A Preview Stage Warning for Android XR
Remember, Android XR is still in preview, and no official headset release has happened. Consequently, what I’ve outlined may evolve before the final launch. While I doubt drastic changes, it’s wise to remain aware of this possibility.
A New Era for Camera Access?
With Google and Meta opening doors to camera access, other companies might follow suit. The year 2025 could usher in groundbreaking opportunities in mixed reality. I’m eager to witness the innovations the developer community will bring to life!
(Header image courtesy of Samsung)
Disclaimer: This blog contains advertisements and affiliate links to support itself. Clicking on an affiliate link means I might earn a small commission. You can read the detailed disclosure here.
