Hand Gesture Recognition for Sign Language

Project Abstract

Majority of deaf-and-mute people use sign language produced by body actions such as hand gestures, body motion, eyes and facial expressions to communicate amongst each other and with non-impaired people in their daily life. However, it has become a barrier for mute and deaf communities which intend to integrate into society. Therefore, it is significant to have a medium that can recognize and translate gesture into understandable words by common people, as the information carried by hand gestures is always principal in sign language. To bridge the communication gap, a vision-based multimodal hand gesture recognition system for Sign Language Recognition (SLR) is proposed, which can recognize both static and dynamic gestures. As the core of the proposed system, 3 classifiers are trained and tested, i.e. dynamic gesture classifier implemented by 2 HMMs, static gesture classifier implemented by voting-based KNN and pre-classifier implemented by LSTM. Within the system, a faster and more efficient image cropping algorithm is proposed to eliminate the unwanted noise after applying skin color detection, which reduces the running time from 4.15s per sample to 0.03s per sample compared to the previous work. The experiment result illustrates that the proposed system achieves a promising performance with accuracy of 99.68% in recognition, testing on the self-built dataset with 12 classes of gesture in ASL, i.e. ‘A’, ‘B’, ‘C’, ‘D’, ‘E’, ‘F’, ‘G’, ‘H’, ‘L’, ‘U’, ‘J’ and ‘Z’, where ‘J’ and ‘Z’ are dynamic gesture and the rest gestures are static. In addition, this thesis compares and analyzes the performance and tuned parameters using various machine-learning-based and ANN-based classification models as the classifiers.

Code at Github