Multimodal and Audio Generation Breakthroughs
Key Questions
What models unify multimodal generation and attention?
Attend to Anything uses hyperbolic embeddings for unified human attention across image/video/audio-visual, outperforming SOTA by 6% with 4x video speedup.
How does WorldDirector improve long-form video generation?
It decouples semantic motion from visual rendering via LLM-coordinated 3D trajectories, maintaining persistent object identity after occlusion.
What benchmarks show the importance of data mixing for VLMs?
DataComp-VLM indicates instruction-heavy mixing matters more than filtering, yielding +5.4pp gains over FineVision baselines.
Which techniques enable real-time or efficient video processing?
SANA-Streaming supports real-time editing, VideoMLA reduces KV memory 92.7%, and Parallelized autoregressive decoding enables lossless parallel captioning.
How does PixWorld advance 3D scene tasks?
PixWorld unifies generation and reconstruction directly in pixel space, avoiding latent information loss and matching SOTA reconstruction quality.
What frameworks support interactive generative worlds?
AlayaWorld provides a full-stack open-source system for long-horizon playable video worlds with real-time navigation and actions.
Are there unified models for vision tasks without task-specific heads?
SenseTime SenseNova-Vision integrates detection, segmentation, and depth into one multimodal generation model matching specialized systems.
What model stitching or forcing methods are explored?
A recipe stitches DINOv2 and SigLIP2 vision foundation models for superior fusion, while Flex-Forcing unifies bidirectional and autoregressive video diffusion.
WorldDirector decouples semantic motion from visual rendering using an LLM to coordinate 3D trajectories, enabling persistent object identity after occlusion — key for long-form physically consistent video generation. Attend to Anything (AAM) foundation model unifies human attention across image/video/audio-visual using hyperbolic embeddings and Fokker-Planck dynamics, outperforms SOTA by 6% on 16 benchmarks with 4x video speedup. DataComp-VLM benchmark shows data mixing (instruction-heavy) matters more than filtering for VLM training, +5.4pp over FineVision. Model stitching recipe for vision foundation models (DINOv2, SigLIP2) enables fusion outperforming individual models. Also: LLaVA-OneVision-2 74.9 JumpScore; StepAudio 2.5 Realtime; SANA-Streaming real-time video editing; VideoMLA reduces KV memory 92.7%; PQSG metric; Gemini 3.5 Live Translate; MilliVid; InternVideo3; ViewSuite; Reinforcing Dual-Path Reasoning; Qwen-Image-Agent; ViQ discrete representations. PixWorld unifies 3D scene generation and reconstruction in pixel space, avoiding latent space info loss, outperforms latent-space methods and matches SOTA reconstruction. Flex-Forcing unifies bidirectional and autoregressive video diffusion, enabling flexible chunking and any-order generation for long-video coherence. SenseTime SenseNova-Vision unifies detection, segmentation, depth, etc. into a single multimodal generation model without task-specific heads, matching specialized systems. AlayaWorld full-stack open-source framework for interactive generative worlds with real-time navigation and actions. Parallelized autoregressive decoding for dense video captioning exploiting weak cross-event dependencies for lossless parallel generation. New: Video-Oasis reveals 55% of video benchmark samples solvable without visual input; SOTA models barely beat random on video-native challenges. Evaluating Blind Spots in Multimodal Models: 235-question benchmark from AI students; closed-source models lead by 10%, no model dominates all types.