Holding a conversation inside two seconds

A digital human is not a rendering problem. The engineering is in the loop, and the loop has a budget measured in milliseconds.

Here is what has to happen between a person finishing a sentence and the face beginning to answer. The browser captures the audio. The audio is transcribed to text. The text goes to a language model, which decides what to say. That answer is sent to a neural voice, which returns audio. That audio drives facial animation on a photoreal character in real time. The rendered result is encoded and streamed back as video.

Five hops, each with its own latency, several of them network round trips to different providers. The target is to land the whole thing inside two seconds.

Past a couple of seconds, a person stops talking to a character and starts waiting for a computer. The illusion does not degrade gradually. It breaks.

So the entire engineering problem is a budget. Every component you choose, you choose partly on quality and mostly on how much of that budget it eats.

The unobvious part is that you cannot treat the hops as sequential blocks and optimise each one. You have to overlap them. Speech recognition can begin transcribing before the speaker has stopped. The language model can begin streaming tokens before it has finished its answer, and the voice synthesiser can begin speaking the first clause while the model is still writing the second. The face can begin moving on the first audio it receives.

Done naively, you pay for every stage in series and blow the budget before you have started. Done as a pipeline, the stages breathe into each other, and the person hears a response almost as they finish speaking.

The rendering runs on a cloud GPU and streams to the browser, which means the user's device does almost no work. That is what makes it deployable. Someone on a laptop in a back office gets the same photoreal character as someone at a workstation, and installs nothing.

There is a language dimension that turns out to matter commercially more than the technology does. Staff speak in a regional dialect. The character answers in the standard written form of the language. That mismatch is not a bug, it is the requirement, because it is how the actual workplace speaks.

The thing we would tell anyone starting this: build the latency harness before you build the avatar. If you cannot see where the budget is going, you will spend it without noticing, and you will not get it back by improving the model.