Any-to-Any Video Modeling: Modeling the World by Traversing Time and Modalities

Video-4M is an any-to-any multimodal model that enables video modeling across different modalities. By flexibly traversing across the time and modality axes, it can predict any modality, at any point in time, given any combination of input modalities.

We show two ways to use this demo:
i) Any-to-any generation: for a given video, you can select any modality as input, and any modality as output. Optionally you can define the intermediate modalities as the chain.
ii) Future prediction in any modality: given initial observations in any modality space, predict the future in any desired modality. You can also add further controls by additionally conditioning on other modalities, such as a caption or transcription.

Select input to Video-4M

Pick an example clip or upload your own video for RGB input, provide a caption, or choose another modality from a curated example. Whichever you choose becomes the input Video-4M conditions on.

Hover to preview · click to select.

Selected: vol_13/bVuiuqmDYaw_000132_000142

Generation chain

Add modalities one at a time, in the order you want them generated. Each one conditions on everything added before it.

Temperature and CFG scale per modality. Defaults are auto-tuned to match your current input modality; adjust to override. Sliders for modalities not in your chain are ignored.

precomputed · seed 0

Predictions will appear here.

Select observation

Pick an example clip (or upload your own video for RGB observation) and the modality you want to observe. Its future frames are then predicted.

Hover to preview · click to select.

Selected: vol_03/1B5h5liEVgc_000043_000053

How much of it to give as input:

Providing more frames makes the prediction more deterministic.

Input to the model: RGB · frames 1–5 of vol_03/1B5h5liEVgc_000043_000053 · 4 fps

Conditioning

Optionally condition the prediction on a caption or a transcription, to steer the predicted trajectory.

Please keep the [SEC_1]-[SEC_4]/[EOS] tags, as they are required by the model. You can freely edit the text between and around them.

Select modalities to predict

Generation uses chaining, where earlier predictions condition later ones.

Temperature and CFG scale per modality. Defaults are auto-tuned to match your current input modality; adjust to override. Sliders for modalities not in your chain are ignored.

precomputed · seed 15

Input framesPredicted frames

Predicted futures will appear here.

Any-to-Any Video Modeling: Modeling the World by Traversing Time and Modalities

Select input to Video-4M

Generation chain

Generate

Generation results

Select observation

Conditioning

Select modalities to predict

Generate

Future predictions