A new assistive tool developed at Norwegian University of Science and Technology (NTNU) uses AI, haptic feedback, and visual guidance to promote inclusion on the factory floor.
A first-of-its-kind system developed at NTNU could redefine inclusion in the manufacturing industry. The assistive system for Deaf and Hard-of-Hearing (DHH) individuals, is designed to improve participation in industrial work by combining safety, communication, and training into a wearable, AI-supported tool.
Karin Sofie Syversveen Lie has developed, during her master’s thesis, a real-time AI-powered assistive system to support DHH individuals in industrial assembly, and workers in areas with high industrial noise, a solution unique in the world. The proof of concept for this solution is described in her thesis, titled Communicating Beyond Sound. It introduces a wearable system that integrates computer vision, real-time speech transcription, and visual task guidance, offering DHH workers greater independence, situational awareness, and communication ability on the factory floor.
“To our knowledge, this is the first system that combines haptic alerts, AI-driven live transcription, and visual assembly instructions into a single assistive tool tailored for DHH workers in manufacturing,” says Karin Syversveen Lie. “It’s a step toward an inclusive future of work, where no one is left out due to sensory barriers.”
A new standard for inclusive manufacturing
The system was developed as part of the SkillAIbility project, funded under Horizon Europe, and represents an early-stage prototype for inclusive, AI-powered assistance in industrial environments. The prototype features:
- A vibrating belt powered by computer vision to notify the user when someone is approaching from the left or right, increasing safety and spatial awareness.
- A speech-to-text module based on OpenAI’s Whisper model enables real-time transcription of spoken instructions in noisy environments.
- A web-based training interface provides intuitive task support without reliance on audio or written instructions.
Tested with participants simulating hearing loss in an industrial setting, the system received positive usability ratings and strong user confidence, despite the limitations of early-stage prototyping. Recently, the system was also tested in a lab setting by one deaf participant, who confirmed its usefulness and provided valuable feedback for further development.
“This system has potential”, said Davy Van Landuyt, project manager at the European Union of the Deaf, who tested the system. “With more iterating and feedback from deaf and hard of-hearing workers in real environments during development, it could become useful for daily work in industrial settings.”
Responding to a growing need
The innovation comes at a crucial time: individuals who are DHH represent a particularly underserved group. Globally, over 5% of the population, approximately 430 million people, including 34 million children experience disabling hearing loss, with projections indicating this number will surpass 700 million by 2050. Yet industrial workplaces remain largely inaccessible to people with hearing disabilities.
The innovation fits directly into the broader SkillAIbility project, which explores how AI can support inclusive upskilling, reskilling, and workplace integration for people with disabilities.
The prototype could contribute to:
- The development of AI-supported assistive systems in SkillAIbility’s learning factory environments, enabling safe, accessible workflows for DHH users.
- The design of vocational training programs tailored to the needs of Deaf workers, offering tools that support communication, situational awareness, and task execution.
- Future pilot deployments across industry partners to validate the effectiveness of inclusive technology in real manufacturing conditions.
“What Karin has built is more than a thesis, it’s a prototype for the future of inclusive industry. This work is a real example of how SkillAIbility can support innovation that directly improves equity in industry,” said Sotirios Panagou, academic supervisor and researcher at NTNU.
The system was built entirely with modular, open-source technologies, and the full source code and documentation are publicly available on GitHub, allowing others to replicate, adapt, or build upon the solution for broader impact in inclusive manufacturing.
The team that was part of the prototype development consists of: Karin Sofie Syversveen Lie, Sotirios Panagou (Logistics 4.0 Lab), Martin Francis Berg & Martin Steinert (TrollLabs), Patrick Mikalef (Department of Computer Science at NTNU, SINTEF Digital).
The solution is now in the process of being upscaled.
See the system in action here
About SkillAIbility
SkillAIbility is a €3 million Horizon Europe research project advancing human-centric approaches to AI and automation in manufacturing and unites 14 leading organizations across 9 European countries. The project will develop new ways for humans and machines to work together effectively while putting workers’ wellbeing first, embodying the European Union’s Industry 5.0 vision. The SkillAIbility project has received funding from the European Union’s Horizon Europe research and innovation programme under Grant Agreement No 101177783. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union. Neither the European Union nor the granting authority can be held responsible for them. This work has received funding from the Swiss State Secretariat for Education, Research and Innovation (SERI).
About NTNU
The Norwegian University of Science and Technology (NTNU) is Norway’s leading university in technology and engineering. With a strong focus on innovation and sustainability, NTNU collaborates with industry, public sector, and society to create knowledge for a better world. Learn more at www.ntnu.edu.