Vibrating TV screen puts people's voices where their mouths are

Imagine having a conversation and seeing the other person's lips move, but hearing the sound coming from their feet. While our brains can mostly compensate, this disconnect occurs with many big-screen display technologies. Visuals come from different areas of the screen, but the accompanying sound comes from speakers or soundbars positioned elsewhere. Researchers at the Korea Institute of Machinery and Materials (KIMM) have developed a display technology that lets you hear sound from the exact area of the screen where the action happens.The new modular-grid-based audiovisual convergence display technology uses multiple vibration actuators attached to the rear of a display to emit sound from selected sections of the screen, effectively turning the entire panel into a giant, highly controllable speaker. This means that if a talking person appears at the top left of the screen, their voice will also come from that area, roughly around where their mouth is. Even better, the system can move these sound-emitting sections in real time, allowing us to hear sounds from different parts of the screen as the action moves. For example, if a ball whooshes across the screen, the sound would also whoosh along the same path. Dr. Dongjoon Kim alongside the prototype modular-grid-based audiovisual convergence displayNational Research Council of Science & Technology Conventional display systems, including our TVs, monitors, and even smartphones, generally output audio and visuals separately. We see the action on the screen, while the sound comes from built-in speakers or separate external units. That's not how our hearing works in real life. We instinctively turn toward the sound of a dropped pen because we expect sound to come from where the action happens. If two people are having a conversation in front of you, you can tell which one is talking, even with your eyes closed, because sound has direction.We've tried to replicate this phenomenon electronically with advanced audiovisual technologies such as stereo and surround-sound systems, which use multiple speakers and audio processing to create the illusion of sounds coming from different directions. However, these systems still rely on speakers positioned around the viewing area, and matching the perceived sound location precisely to the action on a large screen remains a challenge.The KIMM researchers aren't the first to attempt to bridge this disconnect by making the display panel itself produce sound. Existing panel-vibration speaker technologies work on a similar principle, using actuators to vibrate the screen and generate sound waves. However, their sound-emitting locations are generally fixed, and the panel's physical dimensions constrain the available frequency range and how easily the technology can be adapted to different screen sizes.So, what exactly makes KIMM's approach different? The answer lies in its ability to move the sound-emitting locations in real time and scale the technology to different display sizes.For their prototype, the researchers built an approximately 100-inch display consisting of a transparent LED film mounted over an aluminum panel, with 80 voice coil motors (VCMs) installed behind it. These motors are electromagnetic actuators that move back and forth in response to electrical signals. Their movements cause the panel to vibrate, pushing against the surrounding air and generating sound waves, much like the vibrating diaphragm of a conventional loudspeaker. Dr. Dongjoon Kim points out the 80 voice coil motor (VCM) actuators on the back side of the displayNational Research Council of Science & Technology And how did the researchers get only the desired section of the panel to produce sound? The vibrations from each motor don't necessarily stay where they're generated, but travel through the solid panel and interact with vibrations from other motors, making it difficult to control exactly where the sound originates.To overcome this, the researchers first analyzed how different sections of the panel respond to vibrations from the actuators. The system then uses a mathematical technique called the Moore-Penrose pseudo-inverse method to calculate the voltage signals to apply to each motor, to concentrate vibrations in a specific area. By coordinating the motors rather than simply activating the one closest to the desired location, the system can create a localized vibration field and move it across the screen as the sound source moves in the video.Now, real-time sound tracking and actuation come with a few complications. Vibrations traveling through the panel can overlap, reinforcing or canceling one another, while some actuators transmit vibrations to certain areas more effectively than others. Calculating the correct combination of signals can therefore become unstable, particularly when the system has to keep recalculating them as an object moves across the screen.KIMM addressed this challenge with a selective excitation algorithm that excludes actuators with poor vibration transmission effectiveness from the calculations. The researchers also developed a technique that smoothly connects successive acoustic signals, reducing interruptions and unwanted noise as the sound-emitting location changes.To make the whole process more automatic, the team incorporated AI-based sound-source separation and localization. In simple terms, the AI does the heavy lifting of identifying where a sound-producing object appears in the video and isolating its sound from a mixture of different audio sources. The system can then reproduce that sound from the corresponding section of the screen, without someone having to manually specify where it should come from.Tests on the prototype demonstrated a reported 99.9% accuracy in selecting the intended sound-emitting zones. The difference in vibration amplitude between activated and non-activated sections reached as high as 35 dB, indicating that the researchers could concentrate vibrations in specific areas while substantially suppressing them elsewhere. Tests using an acoustic camera also confirmed that the sound-emitting location could follow objects moving across the screen.The research, published in the Journal of Mechanical Science and Technology, demonstrated effective localization of the vibration field in the low- and mid-frequency bands below 500 Hz. For scalability, KIMM adopted a modular-grid design that allows individual display modules to be combined or separated depending on the required screen size. As such, the technology could potentially be adapted to everything from regular home TVs to large-scale LED displays and digital signage. The researchers also demonstrated that driving multiple modules simultaneously can produce low-frequency effects similar to those of a large woofer.By bringing sound and visuals together on the same surface, the technology could make viewing experiences more realistic and immersive. Its potential applications extend beyond TVs and digital signage to media art, performances, exhibitions, and even in-vehicle displays.“We plan to expand the technology to modular LED tile displays and further develop it into a next-generation core technology that can be applied across a wide range of display applications,” said Dr. Dongjoon Kim, senior researcher at KIMM's AX Convergence Research Center.While promising, questions remain about how this technology will match the raw power and chest-thumping depth of dedicated sound systems with standalone subwoofers. Yes, the researchers say that driving multiple modules simultaneously can produce low-frequency effects similar to those of a large woofer. However, reproducing deep, powerful bass is one thing, and matching the overall sound quality, output, and frequency range of a dedicated sound system is another.So, while the technology may not yet eliminate the need for a dedicated sound system, it solves the spatial mismatch between what we see and what we hear, making the action on our screens feel just a little more realistic. It's also worth noting that the problem the technology solves mostly exists in big displays. Your brain perfectly compensates for any mismatch on small screens.Source: National Research Council of Science & Technology via AlphaGalileo