The way we view the world is changing. The abundance of information at our fingertips is being blended with the world around us with the development of augmented reality, (AR) – superimposing computer-generated data over the real world, providing a composite view of our surroundings.
It isn’t a new technology, but automotive applications are providing new impetus. Whether it’s using wearable technology, head-up displays or even the vehicles windshield to project relevant information to the cabin occupants, augmented reality is an exciting area. Cerence is at the heart of this development by discovering and using AR as an additional output modality that – combined with various input modes such as voice, eye-tracking and gesture recognition – enhances the user experience and provides access to a wide range of information. In the car that could mean access to sensor information; blind spot, front radar for example, or it could simply be access to information such as our phone’s contact list. Advances in display technologies and those related to the Head-Up Display (HUD) and AR are proving game-changing for the delivery of information, from the most important to the most personal.
From wearable AR…
There are typically two types of displays that can be used for such purposes: wearables and the displays integrated into the car.
Wearable AR brings with it glasses that can lay information over the world we see. This approach can seamlessly move between differing scenarios, adding context to the environment no matter where the user is. How useful would last mile navigation be, when the user leaves the car?
But carrying the technology with you can prove cumbersome, especially when we already carry a number of items with us in day-to-day life, while in the car it could even prove distracting.
… to in-car AR
In-Car AR is divided into two general categories, video-based AR and windshield-based AR/HUD, with various sub-categories below these. Video-based AR systems use the picture from the car’s forward-facing camera and impose additional information over the top of this. One famous recent example is the new Mercedes-Benz A-Class that is offering an AR-based navigation application that uses the vehicle’s front-facing camera.
Head-Up displays are screens in the driver’s line of sight – that can offer a myriad of information without needing to take your eyes off the road. Data is either provided “with depths” – overlaid onto the real world to give greater context to the user’s surroundings – or it appears “without depth”, as basic viewable information on the screen.
User experience is key
At Cerence, we strive to shape our mobility assistant to offer a user experience that is simple and natural, not only because this should be the essence of any system, but because by doing so it will help increase user acceptance.
While latest advancements in artificial intelligence, in-cabin surveillance systems and advanced sensor fusion capabilities enable blending different interaction modes such as voice, eye-tracking, gestures and more, AR offers another output-modality that fits seamlessly into the intuitive interaction flow. One important aspect in this context is the display surface which significantly impacts the choice of realizable use cases as well as the user experience: New technologies such as Saint-Gobain Sekurit’s “smart windshield” use the whole windshield as a transparent display. As a result, the user experience can be enriched by adding very targeted and smart content offerings in various focal areas.
Use cases
AR technologies can be used for a variety of applications: infotainment, safety or comfort functions. If the focus is infotainment, the windshield can be used to display information regarding the visible outside environment to the user, the restaurants that are interesting or the offers from a newly opened café.
For comfort applications, the windshield can be used as an output channel when the user is interacting with onboard systems or online content. This type of interaction has the benefit that the user doesn’t need to look down at the dashboard, instead the information is displayed where the user is looking, improving safety.
Safety isn’t only confined to minimizing distraction. Using advanced sensor fusion and eye-tracking technologies, the system is able to select the display area based on where the user is actually looking. In addition, AR can be used to guide the user’s attention exactly to where it is needed: If there’s a vehicle on the road that the user should pay particular attention to, the augmentation can focus at that point.
Autonomous driving
And as the automotive industry moves towards greater levels of automated driving functionality, AR technology can play an interesting role in providing information on the maneuvers and a general understanding of the car as it maneuvers through different situations.
The amount of data at our fingertips is growing at an extraordinary rate, blending that information with the world around us will be incredibly important as we move forward, and AR in combination with various human-like input modalities will play a significant role in Cerence achieving its targets of empowering our customers to enable them to provide a unique user experience that their users will love.