Emotional physiological responses are altered not only by external events, but also by the emotions of the people in front of us. While these interpersonal emotional linkages are considered an important aspect of empathy, it is unclear how they relate to cognitive empathy, that is, how we perceive others' emotions. We investigated the relationship between emotional linkage and emotional cognition in an experiment in which two participants estimate each other's emotions while their heart rates were measured during a thrilling joint task using a block game. We also collected data from the two observers because, in reality, in addition to understanding the emotions of the interacting partner, it is sometimes necessary to understand the emotions of a non-interacting person from a third-person perspective. The results suggest that, for game players, their own heart rate is related to perceived partner emotion, and for observers, the degree of heart rate synchrony between observers is related to perceived player emotion. Capturing emotional cognition in a joint task requires consideration of both individual emotion and interpersonal emotional linkages.