ARnnotate: An Augmented Reality Interface for Collecting Custom Dataset of 3D Hand-Object Interaction Pose Estimation (UIST2022)
Xun Qian*, Fengming He*, Xiyun Hu, Tianyi Wang, Ananya Ipsita, and Karthik Ramani. "ScalAR: Authoring Semantically Adaptive Augmented Reality Experiences in Virtual Reality." In Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology. (UIST 2022) DOI: https://doi.org/10.1145/3526113.3545663
Abstract: Vision-based 3D pose estimation has substantial potential in hand-object interaction applications and requires user-specified datasets to achieve robust performance. We propose ARnnotate, an Augmented Reality (AR) interface enabling end-users to create custom data using a hand-tracking-capable AR device. Unlike other dataset collection strategies, ARnnotate first guides a user to manipulate a virtual bounding box and records its poses and the user’s hand joint positions as the labels. By leveraging the spatial awareness of AR, the user manipulates the corresponding physical object while following the in-situ AR animation of the bounding box and hand model, while ARnnotate captures the user’s first-person view as the images of the dataset. A 12-participant user study was conducted, and the results proved the system’s usability in terms of the spatial accuracy of the labels, the satisfactory performance of the deep neural networks trained with the data collected by ARnnotate, and the users’ subjective feedback.