SL-DML: Signal Level Deep Metric Learning for Multimodal One-Shot Action Recognition

• Main Authors: Memmesheimer, Raphael, Theisen, Nick, Paulus, Dietrich
• Format: Article in Journal/Newspaper
• Language: unknown
• Published: arXiv 2020
• Subjects: Computer Vision and Pattern Recognition cs.CV; FOS Computer and information sciences; DML
• Online Access: https://dx.doi.org/10.48550/arxiv.2004.11085; https://arxiv.org/abs/2004.11085

Recognizing an activity with a single reference sample using metric learning approaches is a promising research field. The majority of few-shot methods focus on object recognition or face-identification. We propose a metric learning approach to reduce the action recognition problem to a nearest neighbor search in embedding space. We encode signals into images and extract features using a deep residual CNN. Using triplet loss, we learn a feature embedding. The resulting encoder transforms features into an embedding space in which closer distances encode similar actions while higher distances encode different actions. Our approach is based on a signal level formulation and remains flexible across a variety of modalities. It further outperforms the baseline on the large scale NTU RGB+D 120 dataset for the One-Shot action recognition protocol by 5.6%. With just 60% of the training data, our approach still outperforms the baseline approach by 3.7%. With 40% of the training data, our approach performs comparably well to the second follow up. Further, we show that our approach generalizes well in experiments on the UTD-MHAD dataset for inertial, skeleton and fused data and the Simitate dataset for motion capturing data. Furthermore, our inter-joint and inter-sensor experiments suggest good capabilities on previously unseen setups. : 8 pages, 6 figures, 7 tables