Abstract
Role–event videos are rich in information but challenging to be understood at the story level. The social roles and behavior patterns of characters largely depend on the interactions among characters and the background events. Understanding them requires analysis of the video contents for a long duration, which is beyond the ability of current algorithms designed for analyzing short-time dynamics. In this paper, we propose InSocialNet, an interactive video analytics tool for analyzing the contents of role–event videos. It automatically and dynamically constructs social networks from role–event videos making use of face and expression recognition, and provides a visual interface for interactive analysis of video contents. Together with social network analysis at the back end, InSocialNet supports users to investigate characters, their relationships, social roles, factions, and events in the input video. We conduct case studies to demonstrate the effectiveness of InSocialNet in assisting the harvest of rich information from role–event videos. We believe the current prototype implementation can be extended to applications beyond movie analysis, e.g., social psychology experiments to help understand crowd social behaviors.
Article PDF
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Avoid common mistakes on your manuscript.
References
Khorrami, P.; Paine, T. L.; Brady, K.; Dagli, C.; Huang, T. S. How deep neural networks can improve emotion recognition on video data. In: Proceedings of the IEEE International Conference on Image Processing, 619–623, 2016.
Kim, M.; Kumar, S.; Pavlovic, V.; Rowley, H. Face tracking and recognition with visual constraints in realworld videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–8, 2008.
Forczmański, P.; Nowosielski, A. Multi-view data aggregation for behaviour analysis in video surveillance systems. In: Computer Vision and Graphics. Lecture Notes in Computer Science, Vol. 9972. Chmielewski, L.; Datta, A.; Kozera, R.; Wojciechowski, K. Eds. Springer Cham, 462–473, 2016.
Kagan, D.; Chesney, T.; Fire, M. Using data science to understand the film industry’s gender gap. arXiv preprint arXiv:1903.06469, 2019.
Lv, J.; Wu, B.; Zhou, L. L.; Wang, H. StoryRoleNet: Social network construction of role relationship in video. IEEE Access Vol. 6, 25958–25969, 2018.
Yu, C.; Zhong, Y. W.; Smith, T.; Park, I.; Huang, W. X. Visual data mining of multimedia data for social and behavioral studies. Information Visualization Vol. 8, No. 1, 56–70, 2009.
Tomasi, M.; Pundlik, S.; Bowers, A. R.; Peli, E.; Luo, G. Mobile gaze tracking system for outdoor walking behavioral studies. Journal of Vision Vol. 16, No. 3, 27, 2016.
Bernstein, G. A.; Hadjiyanni, T.; Cullen, K. R.; Robinson, J. W.; Harris, E. C.; Young, A. D.; Fasching, J.; Walczak, N.; Lee, S.; Morellas, V.; Papanikolopoulos, N. Use of computer vision tools to identify behavioral markers of pediatric Obsessive–Compulsive disorder: A pilot study. Journal of Child and Adolescent Psychopharmacology Vol. 27, No. 2, 140–147, 2017.
Grover, A.; Leskovec, J. node2vec: Scalable feature learning for networks. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 855–864, 2016.
Jiang, Y. G.; Dai, Q.; Xue, X. Y.; Liu, W.; Ngo, C. W. Trajectory-based modeling of human actions with motion reference points. In: Computer Vision–ECCV 2012. Lecture Notes in Computer Science, Vol. 7576. Fitzgibbon, A.; Lazebnik, S.; Perona, P.; Sato, Y.; Schmid, C. Eds. Springer Berlin Heidelberg, 425–438, 2012.
Ren, W. H.; Kang, D.; Tang, Y. D.; Chan, A. B. Fusing crowd density maps and visual object trackers for people tracking in crowd scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5353–5362, 2018.
Renoust, B.; Ngo, T. D.; Le, D. D.; Satoh, S. A social network analysis of face tracking in news video. In: Proceedings of the 11th International Conference on Signal-Image Technology & Internet-Based Systems, 474–481, 2015.
Schmitt, D. T.; Kurkowski, S. H.; Mendenhall, M. J. Building social networks in persistent video surveillance. In: Proceedings of the IEEE International Conference on Intelligence and Security Informatics, 217–219, 2009.
Taha, K. Disjoint community detection in networks based on the relative association of members. IEEE Transactions on Computational Social Systems Vol. 5, No. 2, 493–507, 2018.
Newman, M. E. J.; Girvan, M. Finding and evaluating community structure in networks. Physical Review E Vol. 69, No. 2, 026113, 2004.
Newman, M. E. J. Finding community structure in networks using the eigenvectors of matrices. Physical Review E Vol. 74, No. 3, 036104, 2006.
Pons, P.; Latapy, M. Computing communities in large networks using random walks. Journal of Graph Algorithms and Applications Vol. 10, No. 2, 191–218, 2006.
Raghavan, U. N.; Albert, R.; Kumara, S. Near linear time algorithm to detect community structures in largescale networks. Physical Review E Vol. 76, No. 3, 036106, 2007.
Weng, C. Y.; Chu, W. T.; Wu, J. L. RoleNet: Movie analysis from the perspective of social networks. IEEE Transactions on Multimedia Vol. 11, No. 2, 256–271, 2009.
Ramanathan, V.; Yao, B. P.; Li, F. F. Social role discovery in human events. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2475–2482, 2013.
Sun, Q. R.; Schiele, B.; Fritz, M. A domain based approach to social relation recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 435–444, 2017.
Van der Maaten, L. Learning a parametric embedding by preserving local structure. In: Proceedings of the 12th International Conference on Artificial Intelligence and Statistics, 384–391, 2009.
Avril, M.; Leclère, C.; Viaux, S.; Michelet, S.; Achard, C.; Missonnier, S.; Keren, M.; Cohen, D.; Chetouani, M. Social signal processing for studying parent–infant interaction. Frontiers in Psychology Vol. 5, 1437, 2014.
Park, H. S.; Jain, E.; Sheikh, Y. Predicting primary gaze behavior using social saliency fields. In: Proceedings of the IEEE International Conference on Computer Vision, 3503–3510, 2013.
Vrigkas, M.; Nikou, C.; Kakadiaris, I. A. Identifying human behaviors using synchronized audio-visual cues. IEEE Transactions on Affective Computing Vol. 8, No. 1, 54–66, 2017.
Jack, R. E.; Garrod, O. G. B.; Yu, H.; Caldara, R.; Schyns, P. G. Facial expressions of emotion are not culturally universal. Proceedings of the National Academy of Sciences Vol. 109, No. 19, 7241–7244, 2012.
Seng, K. P.; Ang, L. M. Video analytics for customer emotion and satisfaction at contact centers. IEEE Transactions on Human-Machine Systems Vol. 48, No. 3, 266–278, 2018.
Wang, J.; Yuan, Y.; Yu, G. Face attention network: An effective face detector for the occluded faces. arXiv preprint arXiv:1711.07246, 2017.
Zhou, E.; Cao, Z.; Yin, Q. Naive-deep face recognition: Touching the limit of LFW benchmark or not? arXiv preprint arXiv:1501.04690, 2015.
Fruchterman, T. M. J.; Reingold, E. M. Graph drawing by force-directed placement. Software: Practice and Experience Vol. 21, No. 11, 1129–1164, 1991.
Chikhaoui, B.; Chiazzaro, M.; Wang, S. R.; Sotir, M. Detecting communities of authority and analyzing their influence in dynamic social networks. ACM Transactions on Intelligent Systems and Technology Vol. 8, No. 6, Article No. 82, 2017.
Grover, A.; Leskovec, J. node2vec: Scalable feature learning for networks. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 855–864, 2016.
Ma, X. K.; Dong, D. Evolutionary nonnegative matrix factorization algorithms for community detection in dynamic networks. IEEE Transactions on Knowledge and Data Engineering Vol. 29, No. 5, 1045–1058, 2017.
Lu, Z. Q.; Sun, X.; Wen, Y. G.; Cao, G. H.; Porta, T. L. Algorithms and applications for community detection in weighted networks. IEEE Transactions on Parallel and Distributed Systems Vol. 26, No. 11, 2916–2926, 2015.
Rosvall, M.; Bergstrom, C. T. Maps of information flow reveal community structure in complex networks. In: Proceedings of the National Academy of Sciences USA, 1118–1123, 2007.
Blondel, V. D.; Guillaume, J. L.; Lambiotte, R.; Lefebvre, E. Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment Vol. 2008, No. 10, P10008, 2008.
Xiao, Y. P.; Li, X. X.; Wang, H. H.; Xu, M.; Liu, Y. B. 3-HBP: A three-level hidden Bayesian link prediction model in social networks. IEEE Transactions on Computational Social Systems Vol. 5, No. 2, 430–443, 2018.
Acknowledgements
The research is supported by National Natural Science Foundation of China (No. 61802278).
Author information
Authors and Affiliations
Corresponding author
Additional information
Yaohua Pan is a master student at the School of New Media and Communication, College of Intelligence and Computing, Tianjing University. He received his B.S. degree from Henan University. His research interests are computer vision and visualization.
Zhibin Niu is an assistant professor in the College of Intelligence and Computing at Tianjin University, China. He is a Marie Skodowska-Curie Fellow. He received his Ph.D., M.Sc., and B.Sc. degrees separately from Cardiff University, Shanghai Jiao Tong University, and Tianjin University. His research interests include reverse engineering, data mining, and visual analytics. Manuscript received: 2019-12-08; accepted: 2019-12-24
Jing Wu is a lecturer in computer science and informatics at Cardiff University, UK. Her research interests are in computer vision and graphics including image-based 3D reconstruction, face recognition, machine learning, and visual analytics. She received her B.Sc. and M.Sc. degrees from Nanjing University, and Ph.D. degree from the University of York, UK. She serves as a PC member in CGVC, BMVC, etc., and is an active reviewer for journals including PR, CGF, etc.
Jiawan Zhang is a full professor at College of Intelligence and Computing at Tianjin University. He received his Ph.D. degree from Dept. of Computer Science, Tianjin University in 2004. He serve(d) for academic events including the PC Chair of IEEE CGIV, PC co-chair of China CAD&G 2017, general co-chair of VINCI’13, ChinaVis (2015, 2016), Pacific VAST (2015, 2016). He also serve(d) as the program committee member or reviewer for many conferences and journals including PacificVis, EuroVis, IEEE TVCG, IEEE TIP.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made.
The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.
To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
Other papers from this open access journal are available free of charge from http://www.springer.com/journal/41095. To submit a manuscript, please go to https://www.editorialmanager.com/cvmj.
About this article
Cite this article
Pan, Y., Niu, Z., Wu, J. et al. InSocialNet: Interactive visual analytics for role—event videos. Comp. Visual Media 5, 375–390 (2019). https://doi.org/10.1007/s41095-019-0157-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41095-019-0157-9