Multiscopic CPSS for Independent Block-Design Test Based on Hand-Object Interaction Recognition With Visual Attention

This paper introduces a multiscopic cyber-physical-social system (CPSS) to bridge the gap between independent rehabilitation in physical and cognitive aspects. Specifically, we focus on hand-object interaction (HOI) recognition with visual attention for the block-design test (BDT). The proposed framework utilizes three levels which consist of microscopic, mesoscopic, and macroscopic models. In the microscopic model, a hand-tracking vision captures hand-skeletal data and finger joint angle features, enabling the estimation of physical hand postures. In the mesoscopic model, an egocentric vision with an eye tracker records to hand and eye movements, allowing for the symbolic representation of hand-eye coordination through hand gestures and visual attention focus during the test. An evaluation vision system employs color feature classification in the macroscopic model to determine whether the design matches the given task. Through the first eight designs of WAIS-IV BDT with two scenarios, the system successfully measures human behavior from the physical to the cognitive domain. The experiment involving eight healthy participants investigates the relationship between physical measurement and cognitive evaluation. Regression and correlation analyses between the dominant and non-dominant hands reveal that evaluation indices (task completion time, skewness-kurtosis of hand posture, attention to pattern and blocks) can indicate improvement during BDT. The outcomes of this study have significant implications for clinicians and researchers, providing valuable information that is typically unavailable in clinical settings. The proposed multiscopic CPSS framework holds promise for advancing independent rehabilitation practices. Code and datasets are available online at https://github.com/anom-tμbdt-multiscopic.