Video World Model · V1
Kosmic Worlds V1
Our first video world model: a 1.6B-parameter transformer that predicts future video in a compressed latent space.
Trained on real egocentric video. Conditioned on text, images, videos, and learned latent actions.
- Parameters
- 1.6B
- Training video, approximately
- 3,600 h
- Latent tokens processed
- 101B
- Training hardware
- 8x B200s
Architecture
Video, modeled
in latent space.
V1 separates visual compression, dynamics prediction, and reconstruction. A pretrained tokenizer turns video into discrete tokens; our transformer predicts the next latent frame.
- 01
Visual context
An initial image or video, with optional text conditioning.
- 02
Tokenize
The tokenizer encoder compresses video into a 32 × 32 token grid.
- 03
Predict
A causal transformer predicts all 1,024 tokens of the next latent frame in parallel.
- 04
Decode
The tokenizer decoder reconstructs the generated tokens as RGB video.
Spatiotemporal dynamics
The model uses 32 transformer layers, factorized spatial and temporal attention, grouped-query attention, and three-dimensional rotary position embeddings. Language embeddings enter through gated cross-attention.
Learned latent actions
A separate pathway quantizes transitions between frames into 16 learned action codes. These codes can condition generation, but do not consistently correspond to controllable physical actions.
Inference
Three ways to
start a generation.
Each path feeds visual context into the same V1 dynamics model.
- 01
Text
Qwen2-1.5B-Instruct expands the prompt; FLUX.1 [schnell] generates the initial frame. CLIP ViT-L/14 encodes the original prompt for text conditioning.
- 02
Image
The image becomes the initial frame. BLIP-2 produces a caption, which CLIP encodes for language conditioning.
- 03
Video
The input video is tokenized directly. V1’s video-input path currently operates without text conditioning.
Training and configuration
The V1 training run.
Trained on approximately 3,600 hours of Ego4D video using 8x B200s. Training took approximately 125 hours, totaling 1,000 GPU-hours. The run processed 1.93 passes over the tokenized dataset.
- Native generation resolution
- 256 × 256 pixels
- Video frame rate
- 16 fps
- Training clip length
- 8 seconds
- Transformer layers / width
- 32 / 1,512
- Discrete token vocabulary
- 64,000
- Learned action codes
- 16
Configuration and training figures are documented in the technical appendix, Sections 3–5.
Selected generations
Scenes generated by V1.
Selected V1 frames showing roads, parked vehicles, and outdoor environments at the model’s native 256 × 256 resolution.
Results and limitations
An early foundation.
V1 established an end-to-end training and inference stack. Its limitations define the next stage of development.
From V1 to V2Short-horizon scene persistence
The strongest generations preserve layout and lighting for approximately 3–5 seconds. This varies across prompts and decoding settings; longer rollouts can blur or lose coherence.
Inconsistent action control
The latent-action pathway functions, but action codes are not reliable semantic or physical controls.
Resolution and data constraints
Native output is 256 × 256. Minimal filtering of the training videos leaves dark, repetitive, low-motion, and degraded clips in the data.
Technical documentation
V1 technical appendix.
Architecture, training setup, loss curves, inference, qualitative results, and limitations.
You can also open or download the PDF using the links above.
Next generation
Building on V1.
V2 development focuses on physically grounded actions, longer context, and visual refinement.
Explore Kosmic Worlds V2