Sep 24, 2023 · In this study, we propose a novel cross-modal alignment algorithm based on optimal transport (OT). In the alignment process, a transport ...
In this study, we propose a novel cross-modal alignment algorithm based on optimal transport (OT). In the alignment process, a transport coupling matrix is ...
We propose a cross-modal alignment and knowledge transfer model based on. OT and integrate the cross-modal transfer loss with the CTC-based loss for training ...
Sep 24, 2023 · A novel cross-modal alignment algorithm based on optimal transport (OT) is proposed which is utilized to transform a latent acoustic ...
People also ask
What is cross modal alignment?
What is CTC in transportation?
Graves, Towards end to-end speech recognition with recurrent neural networks, Proc. · Graves, Sequence transduction with recurrent neural networks, arXiv ...
Sep 5, 2024 · In this paper, we propose a Temporal Order Preserved OT (TOT)-based Cross-modal Alignment and Knowledge Transfer (CAKT) model (TOT-CAKT) for CTC ...
Cross-modal Alignment with Optimal Transport for CTC-based ASR ... Since the PLM is built from text while the acoustic model is trained with speech, a cross-modal ...
Sep 8, 2024 · Our results demonstrate that the proposed TOT-CAKT significantly improves ASR performance compared to several state-of-the-art models employing ...
For the alignment adapter, we employ as loss the Word Rotator's Distance (WRD) (Yokoi et al., 2020) adapted with an Optimal Transport. (OT) (Monge, 1781; ...
Sep 3, 2024 · Cross-Modal Alignment With Optimal Transport For CTC-Based ASR ... A novel cross-modal alignment algorithm based on optimal transport (OT) ...