Hand Gesture Recognition for Multi-Culture Sign Language Using Graph and General Deep Learning Network

Aim:

        The aim of this project is to design and develop an advanced real-time American Sign Language (ASL) detection system.

Abstract:

      This project introduces a robust framework for real-time American Sign Language (ASL) recognition, focusing on enhancing communication for individuals who use sign language. The system integrates a custom dataset featuring ASL gestures, numbers from 0 to 9, and functional signs like “Delete” and “Space.” Using MediaPipe, hand landmarks are extracted, providing a lightweight yet effective representation of gestures. These landmarks are then processed by an Artificial Neural Network (ANN) trained to classify gestures with high precision. The system is designed to be real-time, integrating seamlessly with a web-based platform for live detection. Through meticulous data preprocessing, landmark extraction, and ANN training, the proposed system achieves both scalability and accuracy, offering a practical solution for gesture recognition in various applications.

Introduction:

       Sign language recognition is a pivotal area of research in human-computer interaction, aimed at enabling communication for individuals with hearing impairments. American Sign Language (ASL) is one of the most widely used sign languages, with its own unique gestures and symbols. Existing systems often struggle to generalize across diverse datasets or fail to achieve real-time recognition, limiting their practical usability.

        This project tackles these challenges by proposing an advanced framework for ASL detection, augmented by numerical gestures and functional signs like “Delete” and “Space.” By employing MediaPipe for accurate hand landmark detection and an ANN for robust classification, the system addresses key limitations in existing models. It also integrates a real-time prediction capability, ensuring that users can interact with the system seamlessly. The project emphasizes creating a lightweight, efficient, and customizable recognition system that caters to practical needs in sign language applications.

Problem Definition:

Accurate and efficient sign language recognition remains a significant challenge due to limitations in existing systems. Many current models rely heavily on computationally expensive deep learning frameworks that process raw images, making them unsuitable for real-time applications. Additionally, these systems often depend on region-specific datasets and struggle to adapt to new gestures or functional signs, reducing their generalizability. The absence of a unified framework that incorporates custom gestures like “Delete” and “Space” further complicates the problem. These issues highlight the need for a versatile and efficient system capable of adapting to diverse datasets, handling functional gestures, and providing accurate predictions in real time.

Existing System:

       Existing sign language recognition systems predominantly rely on image-based models such as Convolutional Neural Networks (CNNs) or hybrid models that combine CNNs with attention mechanisms. While these systems achieve high accuracy on specific datasets, they often fail to generalize across multiple sign languages or incorporate functional gestures beyond the predefined datasets. Some systems utilize advanced technologies like Graph Convolutional Networks (GCNs) and multi-head attention, but these approaches are computationally intensive and require significant resources. Furthermore, many existing systems are designed for offline use and lack the capability for real-time prediction, limiting their usability in practical scenarios. As a result, while these systems perform well in controlled environments, they often struggle in dynamic, real-world settings.

Disadvantages:

         Despite advancements in gesture recognition, existing systems face several limitations. Firstly, their dependence on raw image processing results in high computational costs, making them unsuitable for devices with limited resources. Secondly, most models are tailored to specific datasets and fail to adapt to new gestures or functional signs, such as “Delete” and “Space.” Thirdly, real-time recognition remains a significant challenge due to the lack of integration with live data streams. Finally, these systems often lack flexibility, requiring substantial retraining to accommodate additional gestures or changes in the dataset, which hinders their scalability.

Proposed System:

        This project proposes a novel system for American Sign Language recognition that addresses the limitations of existing systems. The framework combines a custom dataset featuring ASL gestures, numerical signs from 0 to 9, and functional gestures like “Delete” and “Space.” MediaPipe is utilized to extract precise hand landmarks, reducing computational complexity while retaining essential features for gesture recognition. An Artificial Neural Network (ANN) is then trained on these landmarks, enabling accurate classification of gestures. The system is designed to operate in real time, with predictions integrated into a web-based platform. Additionally, the framework is modular and scalable, allowing for the easy addition of new gestures with minimal retraining. This combination of real-time functionality, adaptability, and efficiency makes the proposed system a practical solution for ASL recognition.