A video-call upgrade built for live scenes
Doubao’s upgraded video-call feature moves the assistant beyond a simple question-and-answer interface. In the official scenario, a user can point a phone at a scenic area, while Doubao reads signs, recognizes building entrances, listens to the user, and avoids being distracted by nearby conversations or street noise.
The upgrade combines SeedRealtime, a native audio-video full-duplex model, with Volcengine’s Multimodal Transmission system, or MMT. Full duplex means both sides can send and receive at the same time. For an AI call, that translates into a user speaking while the model is still listening, watching, and preparing a response.
What changes for users
The new experience is defined by three capabilities. First, Doubao can process audio, video, and text together, so a spoken question such as “how do I get there?” can be interpreted alongside a visual cue like a flight display. Second, the AI can speak proactively when it detects important visual information or needs to call tools to organize results. Third, the conversation rhythm is designed to feel less awkward, reducing both interruptions and long silences.
According to the official evaluation, compared with traditional cascaded systems, rhythm-related awkwardness was reduced by about 50%. A cascaded system usually links speech recognition, visual understanding, language reasoning, and speech output in sequence; delays or state mismatches in one module can surface as missed words, premature answers, or irrelevant responses.
Why MMT matters below the model layer
Traditional real-time communication, or RTC, focuses on low-latency audio and video delivery. AI video calls require more than that. The transport layer must also coordinate whether the model session is ready, whether the first frame is complete, and whether audio and video are aligned.
MMT uses a unified multimodal session architecture. On the client side, it is based on QUIC, a low-latency transport protocol that supports connection reuse and multiplexing. At the transport layer, it uses MoQ for unified session control, allowing media streams, signaling, and model state to be scheduled in the same session. Volcengine says this reduces connection setup from seconds to hundreds of milliseconds, making the call feel close to instant.
Preventing missing words and poor visual input
A common problem in older architectures is asynchronous setup: audio may start before the model session is ready, or the model may be ready before the first audio frame arrives. In that case, the beginning of a user request can be lost, leading the model to answer the wrong question. MMT coordinates media streams and model state on one link, while its gateway uses MediaKit processing to check first-frame completeness, audio-video alignment, and model readiness before inference begins.
It also changes how visual data is delivered. If a user points to small text on screen, a low-bitrate video stream may not be enough for the model to read it. The service-side gateway can decide whether to extract frames, request a clearer image, or prioritize specific audio and video segments. In other words, the transport layer becomes a scheduling layer that understands what the model needs, rather than a passive pipe.
Outlook: experience is now infrastructure
Doubao’s upgrade highlights a broader shift in real-time multimodal AI. Model intelligence still sets the ceiling, but transport, synchronization, and session control determine how much of that intelligence reaches the user.
As use cases such as interpretation, language practice, and museum guidance develop, “watching, listening, and speaking at the same time” is likely to become a standard AI interaction pattern. The next phase of competition will not be only about model quality; it will also depend on how well models, networks, gateways, and devices are optimized together to make complex AI systems feel like a natural conversation.