Recognition of Arabic Sign Language and Two-Way Translation between Arabic Text and Arabic Sign language using Natural Language Processing

In this project, we will conduct research on automatic recognition of Arabic Sign Language (ArSL) words and machine translation of the recognized signs to Arabic text and vice versa. This is expected to ease the two-way communication between deaf and non-deaf people. Recognition performance of ArSL lags far behind American and other sign languages. Researchers in the field of Arabic sign language recognition face many challenges, like the high similarities of the different signs, difficulty in determining the start and end of signs, lack of comprehensive and benchmarking databasesetc. In this project, we will explore the use of computer vision techniques using recently released devices like Microsofts Kinect for automatic ArSL words recognition. A prototype that carries this functionality will be implemented. This prototype involves developing effective algorithms and techniques for the different phases of the automatic recognition of isolated-words of Arabic sign language. The recognized words will be translated to Arabic text by applying Natural Language Processing (NLP) techniques like morphological, syntactic, and semantic analysis to generate a meaningful Arabic sentences from the recognized ArSL words/signs and vice versa. Most of the surveyed techniques translate the recognized signs directly into its equivalent Arabic words. However, this is inadequate since the structure and grammar of both languages are different. In this project we will develop a bidirectional machine translation prototype that translates Arabic sentence into a meaningful ArSL signs and vice versa. Our research outcome is expected to compare favourably with the state-of-the-art in the automatic recognition and translation of Arabic sign language. In order to facilitate the above two tasks, we will build a database for Arabic sign language isolated words. This database will be used in our research of automatic recognition of Arabic sign language and translation. In addition, the database will be made freely available as a benchmarking database to other researchers in the field.