ECCV Tutorial
September 9, 2026
Malmö, Sweden
Building Large Video Generation Models: Data Processing, Architectural Insights, Optimization Methods and Evaluation Strategies
Abstract
This tutorial provides a comprehensive and systematic review of the development process for modern large-scale video generation models. While visual generative AI research relies on pre-trained models, many practical challenges – data scaling, model architecture, training and inference optimization, and meaningful evaluation – remain uncovered in publications from the low-resource research groups yet are critical for real-world deployment and research practice. At the same time, many aspects of the development of modern large video models are shrouded in mystery, as this is often non-open-source proprietary information. This tutorial describes the full development pipeline, from petabyte-scale data curation and efficient Diffusion Transformer designs to multi-stage training and evaluation. Drawing on recent advances from open-source projects like Kandinsky 5.0, Wan, and HunyuanVideo, we balance theoretical insight with practical strategies for improving model efficiency and enabling low-resource usage. Attendees will gain actionable knowledge how to efficiently use the large-scale modern video generation systems.
Presenters and Organizers
Motivation & Objectives
⚡ Relevance
Video generation is one of the fastest-moving areas in AI, with immense research interest.
→ Bridging Theory & Practice
Many researchers understand diffusion models but lack practical knowledge of data pipelines, training and inference optimizations. Attendees will take concrete strategies they can apply in their own work.
🛠 Open-Source & Expertise
The open-source community drives innovation in generative AI. This tutorial draws directly from our recent advances, democratizing knowledge and encouraging open-source research.
🎯 Key Objectives of this Tutorial
- Overview of general approaches and the current state of the art in video generation using diffusion models.
- Data Engineering at Scale: curating, processing, and managing petabytes of video data with quality control.
- Modern Architectures: from U-Nets to Diffusion Transformers and specialized attention mechanisms.
- Training Pipeline Design: multi-stage strategies from pre-training to RL-based fine-tuning.
- Optimization Techniques: reducing training/inference costs while maintaining quality.
- Evaluation and Deployment: benchmarking, real-world considerations, and ethical use.
- Practical Insights: equipping attendees to use pre-trained generative systems efficiently.
Program
Schedule (Half-day, ~3.5 hours)
| 09:00 – 09:15 | Part 1: Introduction Viacheslav Vasilev |
| 09:15 – 09:30 | Part 2: Data Engineering for Video Models Viacheslav Vasilev |
| 09:30 – 10:00 | Part 3: Architecture & Design Choices Viacheslav Vasilev |
| 10:00 – 10:25 | Part 4: Training Strategies Viacheslav Vasilev |
| 10:25 – 10:55 | Coffee Break / Q&A — |
| 10:55 – 11:15 | Part 5: Optimizations Maria Kovaleva |
| 11:15 – 11:45 | Part 5.1: Inference Optimization Demo Maria Kovaleva |
| 11:45 – 12:00 | Part 6: Evaluation & Benchmarking Maria Kovaleva |
| 12:00 – 12:15 | Part 7: Licensing, Ethics & Future Directions Maria Kovaleva |
| 12:15 – 12:30 | Part 8: Final Q&A — |
Detailed Topics (including, but not limited to)
- Part 1 (15′): From GANs to diffusion image models; video generation history; key challenges (data, temporal consistency, computational cost, evaluation).
- Part 2 (15′): Data engineering – open/proprietary datasets, pipeline design (scene segmentation, deduplication, quality filtering, captioning, infrastructure).
- Part 3 (30′): Core algorithms (diffusion & flow matching); from images to video; LDMs, DiT/MMDiT, cross-attention; attention optimization (STA, NABLA); VAE and temporal consistency.
- Part 4 (25′): Multi-stage pre-training; SFT, SFT soup, distillation, RL-based post-training.
- Part 5 (30′): Memory-efficient distributed training (DDP, FSDP, activation checkpointing, sequence parallel, tensor parallel, flash attention, pipeline parallel).
- Part 5.1 (10′): Practical part: Inference acceleration and memory optimization (offloading, quantization, MagCache).
- Part 6 (15′): Evaluation and benchmarking: automated metrics, benchmarks; human side-by-side evaluation; arenas.
- Part 7 (25′): Open-source and licensing, open challenges (autoregressive long video generation, world models, omni-model).
Materials
Slides, code, and supplementary materials will be made available after the tutorial.
Target Audience & Prerequisites
Target audience: Researchers working on generative models, computer vision, and multimodal AI, as well as practitioners using pre-trained models in their applications.
Prerequisites: Basic understanding of deep learning, diffusion models, transformers, and Python. No prior expertise in video generation is required.
References
- Arkhipkin, V., Korviakov, V., et al.: Kandinsky 5.0: A family of foundation models for image and video generation (2025). arXiv:2511.14993
- Kong, W., Tian, Q., et al.: HunyuanVideo: A systematic framework for large video generative models (2025). arXiv:2412.03603
- Wan, T., Wang, A., et al.: Wan: Open and advanced large-scale video generative models (2025). arXiv:2503.20314
- Peebles, W., Xie, S.: Scalable diffusion models with transformers (2023). arXiv:2212.09748
- Mikhailov, D., Letunovskiy, A., Kovaleva, M., et al.: VNBLA: Neighborhood Adaptive Block-Level Attention (2025). arXiv:2507.13546
- Vasilev, V., Agafonova, J., et al.: RusCode: Russian cultural code benchmark for text-to-image generation. NAACL 2025 Findings.
- Arkhipkin, V., Vasilev, V., et al.: Kandinsky 3: Text-to-image synthesis for multifunctional generation framework. EMNLP 2024 System Demonstrations.