Researchers have developed two AI methods that reduce sensor and multimodal errors, improving AI for vehicles, robots, medical imaging and security systems.
Researchers at the Korea Advanced Institute of Science and Technology (KAIST) have developed two methods that reduce errors in multimodal large language models (MLLMs), enabling AI to better interpret data from different sensors and avoid generating false descriptions. The approach could improve the reliability of AI systems used in autonomous vehicles, robots, drones, security screening and medical imaging.
The two methods address common problems in multimodal AI. One improves how AI understands images from specialised sensors such as thermal, depth and X-ray cameras, while the other prevents AI from confusing visual and audio information when analysing videos. Together, they improve AI performance without requiring large-scale retraining.
One method, called Modality-Adaptive Decoding (MAD), reduces hallucinations caused by mixing visual and audio information. AI models can sometimes claim to hear sounds that are not actually present simply because an object appears in a video. MAD allows the model to determine whether visual or audio information is more important for a task and gives greater weight to the relevant input while generating a response. Since it does not require retraining, it can be added to existing AI models without additional training costs.
The second method, Diverse Negative Attributes (DNA), improves AI’s understanding of images captured by specialised sensors. Existing models often treat thermal, depth and X-ray images like standard RGB photographs, leading to incorrect interpretations. For example, they may mistake heat patterns in thermal images for reflected light.
To support this work, the researchers developed VS-TDX, a benchmark for evaluating AI across different vision sensors. They also trained the model using its own common mistakes as learning signals, helping it better understand the characteristics of each sensor. This improved its ability to recognise objects in conditions such as darkness, smoke and other environments where conventional cameras have limited visibility.
The researchers said the combination of DNA and MAD offers a cost-effective way to improve multimodal AI. DNA requires only a small amount of data for fine-tuning, while MAD works as a plug-in without any additional training.
The technology could be used in autonomous vehicles operating at night or in poor weather, robots working in smoke-filled environments, drones equipped with thermal cameras, airport X-ray screening systems and medical image analysis, where AI must accurately process information from multiple sensors.
