AI brings sign
language to life

A new real-time American Sign Language (ASL) interpretation system has been developed that uses advanced deep learning and precise hand point tracking to translate ASL gestures into text, enabling accurate and interactive spelling.

The system, created by researchers from the College of Engineering and Computer Science (EECS) at Florida Atlantic University (FAU), combines the object detection capabilities of YOLOv11 – a real-time model that identifies and localises objects within images – with MediaPipe’s precise hand tracking, which detects 21 key hand landmarks to interpret gestures.

Traditional solutions, like sign language interpreters, can be scarce, expensive and dependent on human availability. Smart, assistive technologies that offer real-time, accurate and accessible communication solutions are aiming to bridge this critical gap.

ASL is one of the most widely used sign languages, consisting of distinct hand gestures that represent letters, words and phrases. Existing ASL recognition systems often struggle with real-time performance, accuracy and robustness across diverse environments.

A major challenge in ASL systems lies in distinguishing visually similar gestures such as “A” and “T” or “M” and “N,” which often leads to misclassifications. Additionally, the dataset quality presents significant obstacles, including poor image resolution, motion blur, inconsistent lighting, and variations in hand sizes, skin tones and backgrounds. These factors introduce bias and reduce the model’s ability to generalise across different users and environments.

“What makes this system especially notable is that the entire recognition pipeline – from capturing the gesture to classifying it – operates seamlessly in real time, regardless of varying lighting conditions or backgrounds,” says Bader Alsharif, the first author and an FAU EECS PhD candidate.

“And all of this is achieved using standard, off-the-shelf hardware. This underscores the system’s practical potential as a highly accessible and scalable assistive technology, making it a viable solution for real-world applications.” 

Results of the study, published in the journal Sensors, confirm the system’s effectiveness, which achieved a 98.2% accuracy (mean Average Precision, mAP@0.5) with minimal latency. This finding highlights the system’s ability to deliver high precision in real-time, making it an ideal solution for applications that require fast and reliable performance, such as live video processing and interactive technologies.

With 130,000 images, the ASL Alphabet Hand Gesture Dataset includes a wide variety of hand gestures captured under different conditions to help models generalise better. These conditions cover diverse lighting environments (bright, dim and shadowed), a range of backgrounds (both outdoor and indoor scenes), and various hand angles and orientations to ensure robustness.

Each image is carefully annotated with 21 keypoints, which highlight essential hand structures such as fingertips, knuckles and the wrist. These annotations provide a skeletal map of the hand, allowing models to distinguish between similar gestures with exceptional accuracy.

Imad Mahgoub, PhD, co-author and FAU EECS Tecore professor, says: “This project is a great example of how cutting-edge AI can be applied to serve humanity. By fusing deep learning with hand landmark detection, our team created a system that not only achieves high accuracy but also remains accessible and practical for everyday use. It’s a strong step toward inclusive communication technologies.”

Mohammad Ilyas, PhD, co-author and FAU EECS professor, says: “The significance of this research lies in its potential to transform communication for the deaf community by providing an AI-driven tool that translates American Sign Language gestures into text, enabling smoother interactions across education, workplaces, health care and social settings. By developing a robust and accessible ASL interpretation system, our study contributes to the advancement of assistive technologies to break down barriers for the deaf and hard of hearing population.”

Future work will focus on expanding the system’s capabilities from recognising individual ASL letters to interpreting full ASL sentences. This would enable more natural and fluid communication, allowing users to convey entire thoughts and phrases seamlessly.

Stella Batalama, PhD, dean of the College of Engineering and Computer Science, says: “This research highlights the transformative power of AI-driven assistive technologies in empowering the deaf community. By bridging the communication gap through real-time ASL recognition, this system plays a key role in fostering a more inclusive society.

“It allows individuals with hearing impairments to interact more seamlessly with the world around them, whether they are introducing themselves, navigating their environment, or simply engaging in everyday conversations. This technology not only enhances accessibility but also supports greater social integration, helping create a more connected and empathetic community for everyone.”

The study’s co-authors are Easa Alalwany, PhD, a recent PhD graduate of the FAU College of Engineering and Computer Science and an assistant professor at Taibah University in Saudi Arabia; Ali Ibrahim, a PhD graduate of the FAU College of Engineering and Computer Science.