By Kaushal Trivedi, Senior Product Manager
The automotive industry today is undergoing many transformations, from vehicle connectivity, enhanced autonomy, a boom in electric vehicles, and more. Across the board, improvement in customer experience is critical, and the driving force behind many of the recent innovations in the mobility world.
But what makes an incredible customer experience? Also known as CX, customer experience is defined as the sum of a customer’s perceptions and feelings that result from interactions with a brand’s products and services. When we look at the landscape of the in-car CX, the introduction of voice assistants (VAs) is one innovation that greatly improved the experience by enabling drivers and passengers to do things without taking their eyes off the road and hands off the wheel.
Setting Meaningful KPIs to Measure Satisfaction
The first step in improving CX can be easily explained with a simple quote: “If you cannot measure it, you cannot improve it,” spoken by Lord Kelvin way back in 1883. In order to best measure the impact of VAs in the car, our first step is to cut a typical human-machine interaction into stages and set KPIs to measure the performance of each.
The first interaction a customer typically has with a VA starts either by speaking a wake-up word or pressing the push-to-talk (PTT) button. There are some key questions to ask ourselves about this stage: how often does the VA wakes up without a request from the user? Similarly, how often does the VA fail to wake up on a genuine request from user? How does invocation split between PTT and wake-up word?
Next, the user speaks their query. Measuring performance of automatic speech recognition (ASR), which converts speech to text, and natural language understanding (NLU), which decodes users’ intentions, is critical for this stage. A few KPIs that can help in the assessment of this phase are:
- Core Tech Accuracy: Understanding a user’s query is a pre-requisite to serving it well. High ASR accuracy ensures accurate conversion of speech to text whereas NLU robustness makes sure that customers can talk to the VA in a conversational way and be understood.
- Domain Adoption: OEMs can support endless domains in their VAs, but not all domains are equally adopted by customers. Domains such as phone calling, navigation and music are used most often, whereas weather, wiki search, and jokes may be used less often. Defined as number of transactions per active user, our domain adoption KPI can help OEMs make strategic decisions for their domain portfolio. For example, for a domain with low adoption, OEMs can decide to make cross-domain advertisements to improve its usage.
- Unsupported Utterances: This is a qualitative KPI that tells us which utterances are not supported by current NLU models. OEMs can utilize this metric to fine-tune existing NLU models or add new domains to support such utterances.
A typical interaction is a multi-step event involving multiple dialog exchanges between users and the VA. To understand if the VA was successful in supporting the user, we look at task completion rate, a KPI that gives insight into how a multi-step user interaction ended. For example, was it abruptly ended by the user because they were not satisfied by the assistant’s response? Or was it ended by the assistant because the user didn’t respond, or cloud connectivity was missing? Or, in the best case, was it completed successfully? Higher completion rates mean successful interactions and can therefore give us a glimpse of overall CSAT (customer satisfaction).
Delivering the Power of Analytics
Now, imagine all these metrics – and more – are available in one single place for OEMs to review and take actions. Cerence Analytics is just that: a one-stop-shop where OEMs can review field data and run custom data analysis such as usage patterns across vehicle models or across customer segments (age, gender etc.). And, because raw data collected from the field is usually quite hard to understand and thus to analyze, Cerence Analytics empowers OEMs by giving them access to processed data to facilitate the creation of custom reports and analysis.
Domain Usage Analysis
On an average, there are 10 million Monthly Active Users initiating speech requests. We did an analysis to see how these requests are spread over a limited set of domains, assuming these users have access to all of them. While most of the domains are available off-the-shelf, there are few such as chit chat, smart home etc. which are custom built by OEMs using our Cerence Studio offering.
Commonly used domains such as Navigation, Calling & Music remain the most used domains with average monthly transactions to the tune of 21 million, 19 million & 7.5 million transactions respectively. Interestingly, Chit chat, a custom domain built to engage users in casual conversations with their VAs comes in fourth place with ~ 3 million transactions per month. Car functions, a domain to help users in discovering plethora of car functionalities, is also a popular domain seeing ~1M transactions per month. These stats can enable OEMs in making data-driven decisions vis-à-vis their domain offerings. Moreover, domains with consistently high usage might well serve as monetization platform and thus bringing recurring revenue throughout domain lifecycle.
In a future update to Cerence Analytics, we will launch a range of new reports such as market benchmarking, which lets OEMs compare usage of their VAs against market; customer churn, a report to showcase dropout rates across various domains and languages; top trending domains to enable expansion of domain portfolio; and many more.
Cerence Analytics can help automakers understand their customers’ needs and experiences, enabling them to successfully serve their drivers and create an enriching, interactive experience.
"Information is the oil of the 21st century, and analytics is the combustion engine." – Peter Sondergaard, senior vice president and global head of Research at Gartner, Inc.
To learn more about how Cerence is helping OEMs create incredible in-car experiences, visit www.cerence.com.