Due to the COVID-19 pandemic, wearing a mask has become part of our regular routine. Lip reading, a commonly used technique to understand speech by visually interpreting the movements of the lips and facial expressions of speakers, cannot be used when wearing masks. This has posed significant challenges in communication for people who have difficulty hearing speech, including persons with hearing impairments and the elderly.

To cope with this problem, Aisin Corporation launched an AI-based speech recognition application project to support stress-free communication for anyone.

 YYProbe, a real-time speech recognition application

YYProbe, a proprietary application developed by Aisin in 2019, recognizes speech in real time, converts it into text, and displays the text on the screens of smartphones and other devices.

This application was initially designed to utilize a highly accurate speech recognition system that collects and analyzes speech data in work settings to accumulate knowledge about operations, thereby improving productivity. Time lag and misrecognition are often considered problematic when using a transcription tool, so YYProbe is equipped with Aisin’s proprietary algorithm to quickly and accurately recognize speech even in noisy plant environments, allowing for high processing speed and recognition accuracy. Notably, the application can capture and represent laughter in conversations. The AI model, which incorporates samples of various types of laughter through machine learning, can recognize laughter in conversations and displays “(Lol).” This reflects the developers’ intent to convert the feelings of speakers and the atmosphere of conversations into text in addition to the language transcription function.

*1 When laughter is detected, “(Lol)” is displayed, and the color of the text bubble changes to pink.
*2 When a speaker laughs while talking, “(Lol)” is added at the end of the sentence.

Enhancing communication for employees with hearing impairments

 Aisin Wellsmile Co.,Ltd., a subsidiary of the Aisin Group which promotes the employment of persons with disabilities, came up with the idea of using the system to support communication in workplaces with persons with hearing impairments.

In March 2021, a project was initiated in which devices with YY Probe installed were distributed to about 300 Aisin employees with hearing impairments. The application accurately recognized speech despite the presence of various noise sources in the plant, including machinery. It also displayed in-house jargon and technical terms correctly. Thus, the application demonstrated high performance in practical use.

Here are some of the comments from users:

“The application has made it easier to participate in meetings.”

“The quality of operations has improved because the application has helped deepen understanding within conversations.”

The application has turned out to be a highly effective tool to help eliminate inconvenience and inaccuracy in communication.

As a “caption display” at the information desk of a local government office

Based on the in-house track record, we were convinced that the application would be highly effective as a communication support tool for persons with hearing impairments and the elderly. In November 2021, the application went into service outside the company.

Abu Town in Yamaguchi Prefecture cooperated in the demonstration experiment, as nearly 50% of the population of the town is aging. Citizens with hearing problems often come to the information desk for consultation, and improving communication became a priority.

With this in mind, an experiment was initiated by using YYProbe as a “caption display” so that the speech of town officials could be displayed on a tablet in real time.

Facilitating communication and avoiding misunderstanding by displaying text

Mr. Usagawa of the Health and Welfare Section of the town, who used YYProbe in the information desk service, said:

“The application has facilitated communication with citizens with hearing problems and has helped improve the service for residents. It is also well accepted by other town officials because it accurately converts speech into text with simple operation.”

Furthermore, in order to respond to the voices of those in the field who want to have a conversation while seeing the face of the person they are consulting, we have developed the YYReception Window, a system that displays text on a transparent display that is integrated with an acrylic panel. By evolving not only the software but also the equipment, we are further pursuing the benefit of users.

Using Aisin’s technologies to create a society where no one is left behind

We will continue to improve the system to make it even easier to use by incorporating user feedback, and expand its use to municipalities, businesses, and educational institutions that face similar issues.

With efforts such as YYProbe, Aisin supports its intent to solve social issues by creating new businesses that utilize digital technologies. We remain committed to building a sustainable society and creating a safe and comfortable future where everyone can communicate and no one is left behind.