Compensating Visual Hand Tracking Artifacts with Wearable Inertial Sensors

Keyword: Multimodal Fusion, Hand Tracking, Augmented Reality
Visual hand tracking is a prevalent solution to support multi-gestural hand interaction in immersive environments. Unfortunately, current visual hand tracking systems cannot work well dealing with cases like finger obstruction or a higher demand of spatial-temporal resolution due to the inherent weakness of camera. In this paper, we delved deeper into this problem and sought to find out the opportunities of wearable inertial sensors in compensating visual hand tracking artifacts. We first conducted a systematic analysis on visual artifacts by observing visual-inertial channel differences on a basic hand pinching dataset with both normal and edge cases, finding out 1) insufficiency in both event and hand dynamic recognition and 2) visual artifacts in terms of delay, occlusion, frequency response, and motion consistency. We proposed a FCN-based pinch recognition model and a visual-inertial filtering algorithm in optimizing both event detection and hand dynamics, achieving an optimal accuracy of 96.1% for detecting 8 pinching events. User study found our optimized hand outperforms Oculus hand tracking in terms of accuracy, realism, detail-preserving, responsiveness, lower delay, less artifacts, and overall preference.

The paper content will soon be available.