From Voice to Perception: Helping Machines Understand the Physical World
By Kaivan Karimi, Business Development Senior Director
How voice, multimodal perception, and trusted identity are shaping the next generation of physical AI
For decades, intelligent machines have relied on commands: a person speaks, taps a screen, or presses a button, and the system responds. Physical AI requires a broader understanding. Whether it’s a factory floor, an automated production line, a robot, or a vehicle, physical AI systems need to be able to perceive their full environment, interpret context, verify who is interacting with them, and act reliably at the edge.
Voice Opens the Door to a Richer Understanding
Voice gives people a natural way to interact with technology, especially when their hands and eyes are occupied. Cerence AI has spent more than 25 years making voice work inside the moving car – arguably the harshest and most complex consumer audio environment on earth where road noise, music, open windows, overlapping conversations, weak connectivity, and constant motion make reliable interaction difficult.
The same core challenge appears in manufacturing and robotics: the machine must separate a meaningful signal from a complex environment. Audio AI expands the system’s awareness beyond spoken language. A change in a spindle’s pitch can indicate tool wear. A bearing may sound different before it fails. In a vehicle, an emergency siren can provide an early warning before the source is visible. For a robot, an unexpected mechanical sound can signal a jam or fault.
Sound and Audio AI become operational context. Both help the system recognize that something has changed, determine whether the change matters, and connect that information to an appropriate response.
Voice remains an important starting point, but its value grows when it works together with sound, vision, touch, spatial awareness, and identity. Together, these inputs help intelligent machines and systems understand context, make decisions, and respond appropriately.
Multimodal Perception Requires Context Fusion
Successful systems rely on context fusion, combining audio, visual, tactile (force, torque, friction), and spatial (depth, pose, proprioception) signals into one coherent understanding. Each modality contributes something different: audio can identify a change, vision can clarify what is happening, tactile feedback can confirm a physical interaction, and spatial awareness can establish where an event occurred.
Context Fusion: Many Signals, One Understanding
The value lies in the fusion layer that brings those signals together, resolves ambiguity, and maintains a consistent understanding of context. That shared understanding allows an intelligent system to make a more informed decision, respond to an unexpected situation, and interact more naturally with people. Today, our foundation is Voice and Audio AI—robust speech recognition and noise-resilient processing that can isolate and understand a single speaker in a loud cabin or on a plant floor, with enterprise grade security, trust, and compliance. Multimodal context fusion is the next frontier we are actively investing in and helping to define.
From Voice to the Sensing Machine
One Interaction Challenge Across Four Arenas
The same binding layer is needed across industries where people interact with intelligent machines. The environments differ, but the core challenge is similar: systems must understand people, context, identity, and the physical world around them.
Workers need to control systems hands-free while wearing gloves, PPE, or other equipment. Voice and gesture can help support interaction with plant systems, while acoustic monitoring can help identify changes in machine health or production quality. In environments where latency, shared devices, and multilingual crews matter, these capabilities need to work quickly, locally, and reliably, fully running on the edge, without the need for cloud connectivity. Industrial audio, offline multilingual understanding grounded in the plant's documents, deterministic execution with readback, and an audit trail for compliance allows OT to run locally, and macro level items to run on cloud when connectivity is available.
Audio, visual, and physical signals fuse together for anomaly detection and automated error recovery, so a cell can flag a fault, explain it, and often correct itself before a line stops.
An in-cabin assistant must recognize a request in a noisy environment, connect it with vehicle and environmental context, and continue operating when connectivity drops. Perception also creates new possibilities, such as locating the direction of an emergency siren or adapting an interaction based on which passenger is speaking.
Robots need to interpret language alongside motion, sound, touch, and spatial context. A spoken correction can guide a task, while force and tactile feedback help a robot handle an object with greater precision. Context fusion gives the robot the broader understanding it needs to respond to dynamic conditions, including understanding who issued a command and whether that person is authorized to act.
One Interaction Challenge Across Four Arenas
Trusted Identity Turns Understanding Into Authorized Action
Trust is the missing link in most conversations about the future of human-machine interaction.
In physical environments, understanding a request is only part of the job. The system may also need to know who made the request and whether that person is authorized to take the proposed action. Starting a machine, changing a production setting, taking manual control of a vehicle function, or overriding a robot safety stop can carry real consequences.
In shared industrial environments where devices are used by rotating crews across shifts, identity-aware AI systems will become increasingly important for maintaining trust, governance, and operational oversight. Looking ahead, voice biometrics, combined with Zero Trust architecture and tamper-evident logging, could provide a continuous, on-device identity layer for AI-powered industrial systems. In this model, the system would identify the speaker, help verify whether that person is authorized to perform a requested action, and maintain or revalidate authorization as interactions evolve. Additional authentication factors, such as image recognition or badge-based credentials, would further strengthen identity verification. Combined with auditable activity records, these capabilities would help organizations support security, accountability, and compliance objectives, including requirements associated with workplace safety programs, quality-management standards such as ISO, and emerging regulatory frameworks such as the EU AI Act.
Digital Identity at the Edge
Perception Must Work at the Edge
Physical AI cannot always wait for a cloud connection. Vehicles move through tunnels and coverage gaps. Factories may limit cloud access. Robots and production systems often need to respond with very low latency. In each case, the capabilities required to understand the environment and support critical action must continue working locally.
Edge-first architecture keeps essential functions, including the acoustic front end, wake word, speech recognition, language understanding, biometrics, and reasoning, on-device. Cloud services can extend the experience when connectivity is available, but critical operations do not depend on a continuous connection. When connectivity disappears, nothing breaks from an Operational Technology (OT) standpoint and switching between fully offline and hybrid requires no architectural change.
Perception Must Work at the Edge
The Interaction and Perception Layer for Physical AI
Bringing these capabilities together requires expertise across disciplines that have often evolved separately: conversational AI, Audio AI, multimodal perception, identity, edge deployment, cloud integration, and real-world system design. Cerence AI brings deep experience from one of the most demanding environments for human-machine interaction: the vehicle. That experience is increasingly relevant as intelligent systems move into factories, industrial automation, robotics, and other physical environments where noise, latency, trust, and context matter.
This work also depends on a broader technology foundation. NVIDIA technologies support the development and training of our advanced AI models, while Microsoft Azure is our enterprise foundation. We build on Azure and Azure AI Foundry, and we increasingly draw on Microsoft's Zero Trust for AI best practices and security tooling to govern identity, data, and compliance from edge to cloud.
The machines of the next decade won’t just take orders. They will hear the line, read the scene, know who is speaking, and act—safely, and on their own. Our aim is to become the trusted interaction and perception layer that helps intelligent machines hear, understand, verify, and act—leading this shift as it unfolds. As AI moves further into the physical world, the systems that create the most value will be those that can interpret their environment, understand the people around them, and respond reliably in the moment.
