Skinned Motion Retargeting with Dense Geometric Interaction Perception

Zijie Ye1, Jia-Wei Liu2, Jia Jia1, Shikun Sun1, Mike Zheng Shou2
1Tsinghua University, 2National University of Singapore
NeurIPS 2024 Spotlight
Paper Code

By aligning the DMI field during retargeting, MeshRet not only preserves motion semantics but also prevents self-interpenetration and ensures contact preservation.

Abstract

Capturing and maintaining geometric interactions among different body parts is crucial for successful motion retargeting in skinned characters. Existing approaches often overlook body geometries or add a geometry correction stage after skeletal motion retargeting. This results in conflicts between skeleton interaction and geometry correction, leading to issues such as jittery, interpenetration, and contact mismatches. To address these challenges, we introduce a new retargeting framework, MeshRet, which directly models the dense geometric interactions in motion retargeting. Initially, we establish dense mesh correspondences between characters using semantically consistent sensors (SCS), effective across diverse mesh topologies. Subsequently, we develop a novel spatio-temporal representation called the dense mesh interaction (DMI) field. This field, a collection of interacting SCS feature vectors, skillfully captures both contact and non-contact interactions between body geometries. By aligning the DMI field during retargeting, MeshRet not only preserves motion semantics but also prevents self-interpenetration and ensures contact preservation. Extensive experiments on the public Mixamo dataset and our newly-collected ScanRet dataset demonstrate that MeshRet achieves state-of-the-art performance.

Method Overview

Left: Derive SCS from the semantic coordinates (b, l, φ) across different characters. The sensor features encompasses the sensor’s location and its tangent space matrix. Right: The DMI field effectively captures both contact and non-contact interactions. In the second example, the body sensors (yellow points) are located in the tangent plane of the hand sensors (blue points), signifying a contact interaction.

Overview of the proposed MeshRet. The pipeline begins with the extraction of the DMI field using sensor forward kinematics, and pairwise interaction feature selection. This DMI field, in conjunction with static geometric features, is fed into an encoder-decoder network. The network predicts the target motion sequence, which is aligned with the target character’s geometry and the original DMI field.

BibTeX


@article{ye2024skinned,
  title={Skinned Motion Retargeting with Dense Geometric Interaction Perception},
  author={Ye, Zijie and Liu, Jia-Wei and Jia, Jia and Sun, Shikun and Shou, Mike Zheng},
  journal={Advances in Neural Information Processing Systems},
  year={2024}
}