I came to the realization recently that a lot of the issues that one may raise regarding streaming ASR can actually be tied to a very old assumption: the transcript is read by a human. The fact that the subtitle changes from “fifteen” to “fifty” in 300 milliseconds later does not really matter if someone is just reading it. However, if the transcript goes to the agent or translation systems or LLM, it might be too late.
This is an odd user interface. ASR says one thing; another system does something in response, and then the ASR realizes it said the wrong thing. Now you have to decide if you want to live with bad decisions occasionally, buffer everything until it is finalized, and remove latency, or make things more complex.
What got me wondering is whether what ASR needs to do isn't really about speed or accuracy, but stability. Like “never edit; only append.” If there’s some doubt, the ASR system should be able to wait rather than make an inaccurate prediction which could then be acted upon immediately.
Some new models are changing that, like Confucius r2t2, their architecture views wait/commit as decoding as opposed to the latter being done after adding a stability layer. Another point is that they are open-sourcing the model, making me wonder whether this can be considered as something becoming the fundamental voice stack for agents, as opposed to each agent company developing its own unstable transcription layer.
Maybe I am approaching it all wrong, but it feels as if ASR is being moved from acting as subtitles into being an interface between humans and machines. This is quite a leap and quite a change from what ASR used to be.
Would love to hear some feedback from those working on voice agents in real time. Is revision really a big issue or just theory?
[link] [comments]