Can a voice agent tell "mmhm" from an interruption?

Author here. I wanted to start a discussion about where the decision to take the floor should live.

I wrote this article about my attempt to use an external classifier to separate backchanneling and floor-taking intents, but it seems to have too much latency, so it becomes unusable both ways: using built-in VAD makes the experience choppy because any input interrupts the speech, and a custom one takes too much time before it processes the input and interrupt.

So how do you guys solve this problem in production? (Don't suggest waiting for the `gpt-live-x` family. Waiting is killing.