Cross-modal Short Video Recommendation

With the rapid development of short video platforms, providing accurate short video recommendations for users has become increasingly important. However, due to the multimodal nature of short videos, effectively utilizing this information to improve recommendation quality remains challenging. This paper proposes a cross-modal short video recommendation method that comprehensively utilizes text, image, and audio information. The method involves multimodal processing of short videos, including segmentation of text, image, and audio, and multimodal alignment. It extracts multimodal features of short videos and fuses these features into a unified short video representation. Recommendations are made based on the similarity between user profiles created from their text browsing history and the short video representations. Finally, a cultural short video recommendation experiment based on users’ text reading history is presented. Experimental results show that the cross-modal feature-based recommendation can effectively improve the personalized recommendation accuracy for users in other modalities, especially for cold start users. The proposed method outperforms existing methods in terms of accuracy, recall, and other metrics.