The recent advances in meaningful constraining models have resulted in increasingly useful quantitative information recovered from cardiac images. Nevertheless, as most frameworks utilize either functional or structural images, the analyses cannot benefit from the complementary information provided by the other image sources. To better characterize subject-specific cardiac physiology and pathology, data fusion of multiple image sources is essential. Traditional image fusion strategies are performed by fusing information of commensurate images through various mathematical operators. Nevertheless, when image data are dissimilar in physical nature and spatiotemporal quantity, such approaches may not provide meaningful connections between different data. In fact, as different image sources provide partial measurements of the same cardiac system dynamics, it is more natural and suitable to utilize cardiac physiological models for the fusions. Therefore, we propose to use the cardiac physiome model as the central link to fuse functional and structural images for more subject-specific cardiac deformation recovery through state-space filtering. Experiments were performed on synthetic and real data for the characteristics and potential clinical applicability of our framework, and the results show an increase of the overall subject specificity of the recovered deformations.