Skeleton-DML: Deep Metric Learning for Skeleton-Based One-Shot Action Recognition

• Main Authors: Memmesheimer, Raphael, Häring, Simon, Theisen, Nick, Paulus, Dietrich
• Format: Text
• Language: unknown
• Published: 2020
• Subjects: Computer Science - Computer Vision and Pattern Recognition; Computer Science - Artificial Intelligence; Computer Science - Robotics; DML
• Online Access: http://arxiv.org/abs/2012.13823

One-shot action recognition allows the recognition of human-performed actions with only a single training example. This can influence human-robot-interaction positively by enabling the robot to react to previously unseen behaviour. We formulate the one-shot action recognition problem as a deep metric learning problem and propose a novel image-based skeleton representation that performs well in a metric learning setting. Therefore, we train a model that projects the image representations into an embedding space. In embedding space the similar actions have a low euclidean distance while dissimilar actions have a higher distance. The one-shot action recognition problem becomes a nearest-neighbor search in a set of activity reference samples. We evaluate the performance of our proposed representation against a variety of other skeleton-based image representations. In addition, we present an ablation study that shows the influence of different embedding vector sizes, losses and augmentation. Our approach lifts the state-of-the-art by 3.3% for the one-shot action recognition protocol on the NTU RGB+D 120 dataset under a comparable training setup. With additional augmentation our result improved over 7.7%. Comment: 8 pages, 8 figures, 4 tables