Title:ViCom: Benchmark and Methods for Video Comprehension

Abstract:There is a widespread agreement that future technology for organizing, browsing and searching videos hinges on the development of methods for high-level semantic understanding of video. But, so far the community has not reached to a consensus on the best way to train and assess models for this task. Casting video understanding as a form of action or event categorization is unsatisfying as it is not clear what the semantic classes or abstractions of this domain should be. Language has been exploited to sidestep the problem of defining abstract video categories by formulating video understanding as the task of captioning or description. However, language is highly complex, redundant and sometimes ambiguous. Many different captions may express the same semantic concept. To account for this ambiguity, quantitative evaluation of video description requires sophisticated metrics, whose performance scores are typically hard to interpret by humans.
This paper provides four contributions on this problem. First, we formulate Video Comprehension as a new well-defined task with an easy-to-interpret performance measure. Second, we describe a general semi-automatic procedure to create benchmarks for this task. Third, we publicly release a large-scale video benchmark created with an implementation of this procedure and we include a human study that assesses human performance on our dataset. Finally, we propose and test a varied collection of approaches on this benchmark for the purpose of gaining a better understanding of the new challenges posed by video comprehension.