2024/03/01 · Based on TempCompass, we comprehensively evaluate 8 state-of-the-art (SOTA) Video LLMs and 3 Image LLMs, and reveal the discerning fact that ...

We also design an LLM-based approach to automatically and accurately evaluate the responses from Video LLMs. Based on TempCompass, we comprehensively evaluate 9 ...

Conflicting Videos. We construct conflicting videos to prevent the models from taking advantage of single-frame bias and language priors.

This work proposes the TempCompass, a benchmark to comprehensively evaluate the temporal perception ability of Video LLMs.

The TempCompass benchmark is proposed, which introduces a diversity of temporal aspects and task formats and comprehensively evaluate 8 state-of-the-art ...

2024/09/23 · Our analysis reveals significant performance variations based on question and chart types, highlighting both strengths and weaknesses of current ...

However, existing benchmarks fail to provide a comprehensive feedback on the temporal perception ability of Video LLMs. On the one hand, most of them are unable ...

我们还设计了一种基于法学硕士的方法来自动准确地评估视频法学硕士的回答。基于TempCompass，我们综合评估了8 个最先进的（SOTA）视频LLM 和3 个图像LLM，并 ...

2024/06/03 · The paper proposes a new benchmark called TempCompass to evaluate the temporal reasoning capabilities of video large language models (VLLMs).

2024/03/01 · TempCompass is a new benchmark designed to evaluate Video LLMs on their understanding of temporal aspects such as action, speed, direction, ...