Egocentric Behaviour Analysis Based on Object Relationship Extraction for Cognitive Rehabilitation Support

Human behaviour recognition plays a significant role in early intervention for the cognitive rehabilitation of older people. While existing methods focus on improving third-person vision, human visual attention has been largely ignored in recognition, especially when humans interact with objects. This paper proposes an egocentric behaviour analysis (EBA) based on object relationships using multi-scopic approach. First, we use egocentric vision to extract features from hand posture, object detection, and visual attention movement at the microscopic level. Second, we develop the perception ability to validate the hand–object interaction (HOI) with visual attention at the mesoscopic level. Third, we implement the cognitive ability to describe human activity with multiple object interactions at the macroscopic level. We analyzed Graph Attention Networks (GAT) to evaluate the proposed method for categorizing some activities. The experimental results demonstrate that the GAT can successfully classify related objects with activities. Further research and development are expected to support the robotics application in our living laboratory.