Beyond the Phone: Exploring Phone-XR Integration through Multi-View Transitions for Real-World Applications

Abstract

Despite the growing prevalence of Extended Reality (XR) headsets, their integration with mobile phones remains limited. Existing approaches primarily replicate the phone’s interface in XR or use the phone solely as a 6DOF controller. This paper introduces a novel framework for seamless transitions among mirrored, magnified, and augmented views, dynamically adapts the interface with the content and state of mobile applications. To achieve this, we establish a design space through literature reviews and expert workshops, outline user journeys with common real-world applications, and develop a prototype system that automatically analyzes UI layouts to provide enhanced controls and spatial augmentation. We validate our prototype system with a user study to assess its adaptability to a broad spectrum of applications at runtime, reported its strengths and weaknesses, and suggest directions to advance the future adaption in Phone-XR integration.