Due to the COVID-19 pandemic, wearing a mask has become part of our regular routine. Lip reading, a commonly used technique to understand speech by visually interpreting the movements of the lips and facial expressions of speakers, cannot be used when wearing masks. This has posed significant challenges in communication for people who have difficulty hearing speech, including persons with hearing impairments and the elderly.

To cope with this problem, Aisin Corporation launched an AI-based speech recognition application project to support stress-free communication for anyone.

 YY Probe, a real-time speech recognition application

YY Probe, a proprietary application developed by Aisin in 2019, recognizes speech in real time, converts it into text, and displays the text on the screens of smartphones and other devices.

This application was initially designed to utilize a highly accurate speech recognition system that collects and analyzes speech data in work settings to accumulate knowledge about operations, thereby improving productivity. Time lag and misrecognition are often considered problematic when using a transcription tool, so YY Probe is equipped with Aisin’s proprietary algorithm to quickly and accurately recognize speech even in noisy plant environments, allowing for high processing speed and recognition accuracy. Notably, the application can capture and represent laughter in conversations. The AI model, which incorporates samples of various types of laughter through machine learning, can recognize laughter in conversations and displays “(Lol).” This reflects the developers’ intent to convert the feelings of speakers and the atmosphere of conversations into text in addition to the language transcription function.

*1 When laughter is detected, “(Lol)” is displayed, and the color of the text bubble changes to pink.
*2 When a speaker laughs while talking, “(Lol)” is added at the end of the sentence.

Enhancing communication for employees with hearing impairments

 Aisin Wellsmile Co.,Ltd., a subsidiary of the Aisin Group which promotes the employment of persons with disabilities, came up with the idea of using the system to support communication in workplaces with persons with hearing impairments.

In March 2021, a project was initiated in which devices with YY Probe installed were distributed to about 300 Aisin employees with hearing impairments. The application accurately recognized speech despite the presence of various noise sources in the plant, including machinery. It also displayed in-house jargon and technical terms correctly. Thus, the application demonstrated high performance in practical use.

Here are some of the comments from users:

“The application has made it easier to participate in meetings.”

“The quality of operations has improved because the application has helped deepen understanding within conversations.”

The application has turned out to be a highly effective tool to help eliminate inconvenience and inaccuracy in communication.

As a “caption display” at the information desk of a local government office

Based on the in-house track record, we were convinced that the application would be highly effective as a communication support tool for persons with hearing impairments and the elderly. In November 2021, the application went into service outside the company.

Abu Town in Yamaguchi Prefecture cooperated in the demonstration experiment, as nearly 50% of the population of the town is aging. Citizens with hearing problems often come to the information desk for consultation, and improving communication became a priority.

With this in mind, an experiment was initiated by using YY Probe as a “caption display” so that the speech of town officials could be displayed on a tablet in real time.

Facilitating communication and avoiding misunderstanding by displaying text

Mr. Usagawa of the Health and Welfare Section of the town, who used YY Probe in the information desk service, said:

“The application has facilitated communication with citizens with hearing problems and has helped improve the service for residents. It is also well accepted by other town officials because it accurately converts speech into text with simple operation.”

In Abu Town, efforts have been made to verify the effectiveness of the application and make improvements toward full-scale deployment in FY2022. Aisin will analyze feedback from users to add more functions that are easy to use.

Using Aisin’s technologies to create a society where no one is left behind

We will utilize the results of the in-house project and the demonstration experiment in Abu Town to expand the use of the application to other local governments and businesses facing similar issues.

With efforts such as YY Probe, Aisin supports its intent to solve social issues by creating new businesses that utilize digital technologies. We remain committed to building a sustainable society and creating a safe and comfortable future where everyone can communicate and no one is left behind.