publications

First author / co-first author

2026

1. NeurIPS

   VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding

   Haichen He^* , Jiayi Zhou^*, Sifeng Shang , and 3 more authors

   In Advances in Neural Information Processing Systems (NeurIPS), Evaluations & Datasets Track , 2026

   Abs arXiv Bib Code Website

   Real-world long video understanding requires models to perform continuous tracking, information integration and memory retention over massive temporal spans within extreme video durations. Mastering this intense cognitive load constitutes the fundamental bottleneck in long video understanding. While existing benchmarks have driven progress by scaling up video duration, their evaluation tasks often require comprehending only short and isolated video segments, falling short of capturing the challenge of ultra-long-context reasoning. To measure this cognitive load, we emphasize continuous certificate length, defined as the video length a human must continuously watch to definitively answer a given question. Driven by this metric, we introduce VideoOdyssey, a benchmark specifically designed for ultra-long-context and omni-modal video understanding. VideoOdyssey is characterized by three key features: 1) Extreme video duration and diversity: spanning 11 domains and 54 subcategories with an average video duration of 109 minutes; 2) Comprehensive evaluation scenarios: offering two subsets to address different research focuses, i.e., VideoOdyssey-V for probing the limits of visual understanding in MLLMs, and VideoOdyssey-AV for evaluating synchronized audio-visual understanding for omni-modal models; 3) Ultra-long and multi-level continuous certificates: extending the average continuous certificate to 16 minutes for VideoOdyssey-V and 12.8 minutes for VideoOdyssey-AV. Crucially, we design 5 granular levels from seconds to hours, providing a comprehensive diagnostic tool to evaluate models across varying context lengths and cognitive loads. Extensive evaluations show that bottlenecks of current MLLMs extend beyond simple retrieval to include struggles with continuous reasoning across varying context lengths, fine-grained perception, and non-verbal omni-modal understanding. We hope VideoOdyssey will spur the development of next-generation MLLMs toward genuine real-world video understanding.

   @inproceedings{he2026videoodyssey,
     title = {VideoOdyssey: A Benchmark for Ultra-Long-Context and Omni-Modal Video Understanding},
     author = {He, Haichen and Zhou, Jiayi and Shang, Sifeng and Hu, Yihan and Zhang, Yuanhan and Zhou, Kaiyang},
     booktitle = {Advances in Neural Information Processing Systems (NeurIPS), Evaluations \& Datasets Track},
     year = {2026},
   }

Second author / other

2026

1. ICLR

   Fine-tuning Quantized Neural Networks with Zeroth-order Optimization

   Sifeng Shang , Jiayi Zhou, Chenyu Lin , and 2 more authors

   In International Conference on Learning Representations (ICLR) , 2026

   Abs arXiv Bib HTML Code

   As the size of large language models grows exponentially, GPU memory has become a bottleneck for adapting these models to downstream tasks. In this paper, we aim to push the limits of memory-efficient training by minimizing memory usage on model weights, gradients, and optimizer states, within a unified framework. Our idea is to eliminate both gradients and optimizer states using zeroth-order optimization, which approximates gradients by perturbing weights during forward passes to identify gradient directions. To minimize memory usage on weights, we employ model quantization, e.g., converting from bfloat16 to int4. However, directly applying zeroth-order optimization to quantized weights is infeasible due to the precision gap between discrete weights and continuous gradients, which would otherwise require de-quantization and re-quantization. To overcome this challenge, we propose Quantized Zeroth-order Optimization (QZO), a simple yet effective approach that perturbs the continuous quantization scale for gradient estimation and uses a directional derivative clipping method to stabilize training. QZO is orthogonal to both scalar-based and codebook-based post-training quantization methods. Compared to full-parameter fine-tuning in 16 bits, QZO can reduce the total memory cost by more than 18x for 4-bit LLMs, and enables fine-tuning Llama-2-13B within a single 24GB GPU.

   @inproceedings{shang2026qzo,
     title = {Fine-tuning Quantized Neural Networks with Zeroth-order Optimization},
     author = {Shang, Sifeng and Zhou, Jiayi and Lin, Chenyu and Li, Minxian and Zhou, Kaiyang},
     booktitle = {International Conference on Learning Representations (ICLR)},
     year = {2026},
   }