How Gemini-Omni Might Work

This is my high-level speculation on how Gemini-Omni, Google’s latest video editing/generation model, might be implemented.

Public information is not sufficient to identify the exact architecture, but I see three reasonable possibilities from high to low probability:

  1. Transfusion / HunyuanImage-3.0-style single unified transformer for joint multimodal understanding and generation. Most likely for Gemini-Omni.
  2. Qwen-Image-Edit-style encoder-decoder system: Gemini as a multimodal encoder, whose embeddings are then used as conditioning for a flow-matching model to render video. Most likely for Veo 3.
  3. GLM-Image-style hybrid: autoregressive coarse visual tokens in discrete space + diffusion refinement in continuous space. Plausible for images, but less likely as the main dense video route this year.

I believe the Transfusion-style unified model is the most likely architecture for Gemini-Omni.


Roadmap 1: Transfusion

This is the Transfusion-style roadmap, and my top choice for Gemini-Omni.

The core idea is one shared transformer that handles both:

Simplified training/inference flow:

Text / Image / Video / Audio Context
          ↓
Shared Multimodal Transformer
          ↓
Next-token prediction (discrete modality like text generation) + Flow matching (continuous modality like video/image/audio generation)
          ↓
Text, Images, Video, or Audio Output

A blockwise causal mask is used: text regions (causal attention) follow standard next-token prediction, while visual/audio regions (full attention) are trained with flow-matching objectives.

Unified transformer with discrete token heads and continuous flow-matching heads
Unified transformer with discrete token heads and continuous flow-matching heads.

Why this is attractive:

Most diffusion systems still use an encoder-decoder architecture, where the encoder is a text encoder that encodes a text prompt into embeddings, and the decoder is a DiT that iteratively denoises.

Previously, a tiny CLIP text encoder was used, as in Stable Diffusion. Such an encoder-decoder architecture made sense, as the encoder was tiny anyway.

Modern DiTs use heavier LLMs (like Mistral) or VLMs (like Qwen2.5-VL) as text encoders, as in FLUX 2.0 and Qwen-Image-Edit. This shift from simple CLIP to a larger-scale pretrained LLM makes sense, as it brings better prompt alignment and multimodal conditioning support (e.g., natively supporting reference images/videos).

The encoder can now be a massive multimodal model, such as Gemini 3.5 Flash with about 250B parameters and 45B active parameters. Using a huge model only as an embedding extractor feels wasteful when it already deeply understands the prompt, source video, audio, dialogue, and editing intent.

A unified Transfusion-like model removes the awkward split between “the model that understands” and “the model that generates.” It unifies the architecture (as the text encoder and DiT are both Transformers anyway) through unified losses, which not only leads to better prompt alignment but also enables joint multimodal understanding and generation, which is important for editing.

The word “Omni” in Gemini-Omni also strongly suggests this single model family that processes and generates across modalities, exactly what this roadmap delivers.


Roadmap 2: Gemini as Multimodal Encoder and DiT as Video Rendering Decoder

This follows the Qwen-Image-Edit pattern, applied to video.

Gemini does not replace the video backbone. It acts as a multimodal encoder whose embeddings are then used as conditioning for the DiT to render a video for generation or editing.

Architecture:

Text / Image / Video / Audio / Dialogue Context
          ↓
Gemini Multimodal Model
          ↓
Prompt rewriting + Dense multimodal embeddings
          ↓
Latent Video/Audio Diffusion Model
          ↓
Edited Video + Audio
Gemini conditioner feeding dense embeddings into a Veo-style generator
Gemini conditioner feeding dense embeddings into a DiT renderer.

Key advantages:

This is likely the most likely architecture for Veo 3: Gemini understands intent, and DiT renders.

Limitation: it remains a two-system pipeline rather than a true single “Omni” model.


Roadmap 3: Autoregressive Coarse Tokens + Diffusion Refinement

This is the GLM-Image-style hybrid.

The model first autoregressively generates coarse visual tokens: layout, structure, composition, and low-resolution content. Then a diffusion model refines them into high-quality output.

Pipeline:

Prompt / Multimodal Context
          ↓
Autoregressive Model β†’ Coarse Discrete Visual Tokens
          ↓
Diffusion Refinement Model
          ↓
High-Quality Image or Video

This works excellently for images. For dense video, however, token count explodes because video has space, time, motion, temporal consistency, and often audio. Even with compression and hierarchy, the cost is severe.

I see this roadmap as a future version for video generation, as the benefits are clear: being able to use all techniques from LLMs for video generation, such as KV caching, speculative decoding, CUDA Graph acceleration, and training infrastructure.

AR model produces coarse scene/video tokens, then diffusion refines them
AR model produces coarse scene/video tokens, then diffusion refines them.

My Current Guess

Final ranking:

  1. Unified Transformer (Roadmap 1): most likely for Gemini-Omni
  2. Gemini-DiT (Roadmap 2): most likely for Veo 3
  3. AR + Diffusion Hybrid (Roadmap 3): less likely for current video generation and possibly a roadmap for future video models

The name Gemini-Omni points toward unification, and the field is clearly moving toward less glue, more shared modeling across modalities (text, image, video, audio, and action for world modeling in the coming months).


References

  1. Google, “Introducing Gemini Omni.”
  2. Google DeepMind, “Veo 3 Technical Report.”
  3. Zhou et al., “Transfusion” (arXiv:2408.11039).
  4. Zhipu AI, “GLM-Image.”
  5. Qwen Team, “Qwen-Image” and “Qwen-Image-Edit.”
  6. Tencent Hunyuan, “HunyuanImage-3.0.”

What do you think is the most likely path? Would love to hear your thoughts in the comments.