Skip to main content
placeholder image

FT-HID: a large-scale RGB-D dataset for first- and third-person human interaction analysis

Journal Article


Abstract


  • Analysis of human interaction is one important research topic of human motion analysis. It has been studied either using first-person vision (FPV) or third-person vision (TPV). However, the joint learning of both types of vision has so far attracted little attention. One of the reasons is the lack of suitable datasets that cover both FPV and TPV. In addition, existing benchmark datasets of either FPV or TPV have several limitations, including the limited number of samples, participant subjects, interaction categories, and modalities. In this work, we contribute a large-scale human interaction dataset, namely FT-HID dataset. FT-HID contains pair-aligned samples of first-person and third-person visions. The dataset was collected from 109 distinct subjects and has more than 90K samples for three modalities. The dataset has been validated by using several existing action recognition methods. In addition, we introduce a novel multi-view interaction mechanism for skeleton sequences, and a joint learning multi-stream framework for first-person and third-person visions. Both methods yield promising results on the FT-HID dataset. It is expected that the introduction of this vision-aligned large-scale dataset will promote the development of both FPV and TPV, and their joint learning techniques for human action analysis.

Publication Date


  • 2023

Citation


  • Guo, Z., Hou, Y., Wang, P., Gao, Z., Xu, M., & Li, W. (2023). FT-HID: a large-scale RGB-D dataset for first- and third-person human interaction analysis. Neural Computing and Applications, 35(2), 2007-2024. doi:10.1007/s00521-022-07826-w

Scopus Eid


  • 2-s2.0-85139513289

Web Of Science Accession Number


Start Page


  • 2007

End Page


  • 2024

Volume


  • 35

Issue


  • 2

Place Of Publication


Abstract


  • Analysis of human interaction is one important research topic of human motion analysis. It has been studied either using first-person vision (FPV) or third-person vision (TPV). However, the joint learning of both types of vision has so far attracted little attention. One of the reasons is the lack of suitable datasets that cover both FPV and TPV. In addition, existing benchmark datasets of either FPV or TPV have several limitations, including the limited number of samples, participant subjects, interaction categories, and modalities. In this work, we contribute a large-scale human interaction dataset, namely FT-HID dataset. FT-HID contains pair-aligned samples of first-person and third-person visions. The dataset was collected from 109 distinct subjects and has more than 90K samples for three modalities. The dataset has been validated by using several existing action recognition methods. In addition, we introduce a novel multi-view interaction mechanism for skeleton sequences, and a joint learning multi-stream framework for first-person and third-person visions. Both methods yield promising results on the FT-HID dataset. It is expected that the introduction of this vision-aligned large-scale dataset will promote the development of both FPV and TPV, and their joint learning techniques for human action analysis.

Publication Date


  • 2023

Citation


  • Guo, Z., Hou, Y., Wang, P., Gao, Z., Xu, M., & Li, W. (2023). FT-HID: a large-scale RGB-D dataset for first- and third-person human interaction analysis. Neural Computing and Applications, 35(2), 2007-2024. doi:10.1007/s00521-022-07826-w

Scopus Eid


  • 2-s2.0-85139513289

Web Of Science Accession Number


Start Page


  • 2007

End Page


  • 2024

Volume


  • 35

Issue


  • 2

Place Of Publication