Recently, a worker’s subjective satisfaction, in other words Quality-of-Working Life (QWL), has attracted more attention than productivity or efficiency. To provide QWL-oriented working support in a factory manufacturing environment, this study proposes a framework for recognizing manual assembly behaviors that may reflect a worker’s inner state or physical condition. First, a new set of interactions is defined to describe the behavioral fluctuations and diversity that appear even in the same assembly task. We expand the conventional interaction definitions for manufacturing analysis in three ways: 1) we add primitive interactions that qualify the fundamental interactions, 2) we install a spatial attribute into the interaction definition, and 3) we allow the simultaneous occurrence of multiple interactions. Additionally, an image-based automatic recognition technique is designed to detect the newly defined interactions. Through experimental evaluations for a compressor attachment task, we found various differences in manual assembly behaviors and confirmed that they can be distinguished using the recognized interactions.