Smart Multilingual Bidirectional Communication System Using Media Pipe for People with Hearing and Speech Impairments

Abstract

Communication barriers between speech-impaired, hearing-impaired, and normal individuals present significant challenges in everyday interactions. To address these challenges, the proposed system integrates speech recognition, gesture recognition, and deep learning-based classification methods to enable natural, seamless communication across diverse user groups. In the voice-to-gesture module, audio input is captured through a microphone and processed using an Automatic Speech Recognition (ASR) engine. The ASR system converts spoken language into corresponding text, facilitating the translation of verbal communication into a format that can subsequently be expressed through gestures, thereby enhancing accessibility for hearing-impaired users. Conversely, the gesture-to-voice module captures hand movements using a camera setup to monitor them in real time. The video input is processed using MediaPipe, an advanced framework that efficiently extracts detailed hand landmarks and skeletal keypoints. These extracted features serve as input to a deep learning-based classification model, which interprets the hand gestures and maps them to predefined semantic units. The recognized gestures are then converted into natural-language output and synthesized into speech, enabling speech generation from nonverbal hand movements and thus supporting communication for speech-impaired users. By combining these modules, the system establishes a bidirectional communication platform that enhances interaction between individuals with varying communication abilities. This integration of ASR, computer vision-based gesture analysis, and deep learning classification promotes an inclusive environment by facilitating real-time translation between speech and gestures, ultimately reducing communication barriers and improving user experience.