AI Model Highly Accurate in Recognizing Sign Language Gestures (2024)

Dec 16, 2024 | Research | 0 |

Summary: Researchers at Florida Atlantic University developed a groundbreaking AI system using MediaPipe and YOLOv8, achieving 98% accuracy in detecting American Sign Language (ASL) gestures.

Takeaways:

1. High Accuracy: The AI model achieved 98% accuracy, 98% recall, and a 99% F1 score in recognizing ASL gestures, showcasing its precision and reliability.
2. Innovative Approach: Combining MediaPipe for hand tracking with YOLOv8 for object detection enabled the system to identify subtle variations in hand poses with high efficiency.
3. Real-World Potential: Future development will focus on expanding gesture datasets and optimizing performance for edge devices, making the system practical for real-time, inclusive applications in education, healthcare, and daily interactions.

Sign language serves as a sophisticated means of communication vital to individuals who are deaf or hard-of-hearing, relying on hand movements, facial expressions, and body language to convey nuanced meaning. American Sign Language exemplifies this linguistic complexity with its distinct grammar and syntax.

Sign language is not universal; rather, there are many different sign languages used around the world, each with its own grammar, syntax, and vocabulary, highlighting the diversity and complexity of sign languages globally.

Adapting Technology to Understand Sign Language

Various methods are being explored to convert sign language hand gestures into text or spoken language in real time. To improve communication accessibility for people who are deaf or hard-of-hearing, there is a need for a dependable, real-time system that can accurately detect and track American Sign Language gestures. This system could play a key role in breaking down communication barriers and ensuring more inclusive interactions.

To address these communication barriers, researchers from the College of Engineering and Computer Science at Florida Atlantic University conducted a first-of-its-kind study focused on recognizing American Sign Language alphabet gestures using computer vision. They developed a custom dataset of 29,820 static images of American Sign Language hand gestures. Using MediaPipe, each image was annotated with 21 key landmarks on the hand, providing detailed spatial information about its structure and position.

Incorporating Deep Learning (AI) Solutions

These annotations played a critical role in enhancing the precision of YOLOv8, the deep learning model the researchers trained, by allowing it to better detect subtle differences in hand gestures.

Results of the study, published in the Elsevier journal Franklin Open, reveal that by leveraging this detailed hand pose information, the model achieved a more refined detection process, accurately capturing the complex structure of American Sign Language gestures. Combining MediaPipe for hand movement tracking with YOLOv8 for training, resulted in a powerful system for recognizing American Sign Language alphabet gestures with high accuracy.

“Combining MediaPipe and YOLOv8, along with fine-tuning hyperparameters for the best accuracy, represents a groundbreaking and innovative approach,” says Bader Alsharif, first author and a PhD candidate in the FAU Department of Electrical Engineering and Computer Science. “This method hasn’t been explored in previous research, making it a new and promising direction for future advancements.”

Findings show that the model performed with an accuracy of 98%, the ability to correctly identify gestures (recall) at 98%, and an overall performance score (F1 score) of 99%. It also achieved a mean Average Precision (mAP) of 98% and a more detailed mAP50-95 score of 93%, highlighting its strong reliability and precision in recognizing American Sign Language gestures.

“Results from our research demonstrate our model’s ability to accurately detect and classify American Sign Language gestures with very few errors,” says Alsharif. “Importantly, findings from this study emphasize not only the robustness of the system but also its potential to be used in practical, real-time applications to enable more intuitive human-computer interaction.”

Improving Hand Pose and Gesture Understanding

The successful integration of landmark annotations from MediaPipe into the YOLOv8 training process significantly improved both bounding box accuracy and gesture classification, allowing the model to capture subtle variations in hand poses. This two-step approach of landmark tracking and object detection proved essential in ensuring the system’s high accuracy and efficiency in real-world scenarios. The model’s ability to maintain high recognition rates even under varying hand positions and gestures highlights its strength and adaptability in diverse operational settings.

“Our research demonstrates the potential of combining advanced object detection algorithms with landmark tracking for real-time gesture recognition, offering a reliable solution for American Sign Language interpretation,” says Mohammad Ilyas, PhD, co-author and a professor in the FAU Department of Electrical Engineering and Computer Science. “The success of this model is largely due to the careful integration of transfer learning, meticulous dataset creation, and precise tuning of hyperparameters. This combination has led to the development of a highly accurate and reliable system for recognizing American Sign Language gestures, representing a major milestone in the field of assistive technology.”

Future efforts will focus on expanding the dataset to include a wider range of hand shapes and gestures to improve the model’s ability to differentiate between gestures that may appear visually similar, thus further enhancing recognition accuracy. Additionally, optimizing the model for deployment on edge devices will be a priority, ensuring that it retains its real-time performance in resource-constrained environments.

Further Reading

“By improving American Sign Language recognition, this work contributes to creating tools that can enhance communication for the deaf and hard-of-hearing community,” says Stella Batalama, PhD, dean, FAU College of Engineering and Computer Science. “The model’s ability to reliably interpret gestures opens the door to more inclusive solutions that support accessibility, making daily interactions – whether in education, health care, or social settings – more seamless and effective for individuals who rely on sign language. This progress holds great promise for fostering a more inclusive society where communication barriers are reduced.”

Study co-author is Easa Alalwany, PhD, a recent PhD graduate of the FAU College of Engineering and Computer Science and an assistant professor at Taibah University in Saudi Arabia.

Featured image: ​​Bader Alsharif, first author and a PhD candidate in the FAU Department of Electrical Engineering and Computer Science. Photo: Florida Atlantic University