Yuya Asano日本語

Contextual speech recognition correction

Using what a voice assistant already knows about the task to fix what it misheard

Role
First author, method and evaluation
Team
ISABEL, Alexa Prize TaskBot Challenge 2
Result
Third place
Paper
COLING 2025, Industry Track
Read the paperarXiv

The user says: how to care for outdoor plants

The recognizer returns several guesses. Normally only the first one is used.

  1. how to camper for outdoor plants1st guessTask search finds nothing like this.
  2. how to care for outdoor plants2nd guessTask search finds “take care plant,” above the similarity threshold.

Examples from the paper. They resemble real user requests but are not taken from real conversations.

The problem

Our Alexa TaskBot helped people with cooking and home improvement tasks. General-purpose speech recognition often failed on exactly the words that mattered, like task and tool names. In our data, 4.1% of utterances had a recognition error, and 18.2% of conversations had at least one. Existing correction methods depend on prior user data or named entities, which a new assistant in a new domain doesn’t have.

The approach

The assistant already knows what the user is likely to say next: the options it just offered, the commands available at this step, and the full list of tasks it supports. The method uses that context in two ways, shown in the demo above.

Re-ranking the guesses. The recognizer returns up to five guesses. Each one is checked against the context by lexical and semantic similarity, and a lower-ranked guess wins if it is the only one that matches a real task. This handles extra words people insert in the middle of a request.

Matching by sound. When none of the guesses is right, the method compares phoneme sequences between the best guess and the context, using their longest common subsequence. This handles words the recognizer garbled and words the user added or left out.

To handle the many ways people phrase the same request, we clustered the task list and used GPT-3.5 to generate variations only for cluster centroids, adding 160K phrasings to the search index at low cost. The method needs no prior user data and adds no LLM latency at run time, because augmentation happens offline. In deployment, the assistant confirmed each correction by asking “Did you mean …?”

Results

.18 → .54Correction recall, compared with a named-entity baseline (offline evaluation)
.27 → .44Correction F1, compared with the same baseline
+.8 to 1Higher user ratings (out of 5) with real users when the correction worked, and false positives did not significantly lower ratings

Team

Built as part of team ISABEL (IncluSive And collaBorativE aLexa) at the University of Pittsburgh, advised by Malihe Alikhani. Our TaskBot placed third in the Alexa Prize TaskBot Challenge 2.