Motif Mining: Towards Multimodal Retrieval for Discovering Disinformation Trends in Social Media

As the world has become more and more connected online, so too has the amount of data that has been produced. Alongside this increase in data production the amount of disinformation has drastically increased. Coming to the fore in recent years and playing outsized roles in the politics of a myriad of countries, disinformation has become a hot topic. The scale of the data, in particular images and videos, has grown beyond the human capacity for processing in any detail. To help aid reviewers of online content we propose extending the idea of image provenance to include multimodal motif mining. This formalization and implementation of the problem of visual motif mining is designed to help humans process the vast amounts of visual data more quickly and to aid in the discovery of emerging disinformation trends in the online space. Early efforts in this space focused on monomodal capabilities, considering images and texts separately. Efforts to combine the two via fusion techniques left something to be desired. With the release of contrastive language-image pre-training techniques a new door has been opened for multimodal shared embedding spaces. In this work I share some of the earlier monomodal motif mining strategies, focusing on images as their subject. This is followed by studies in using CLIP-based strategies for projecting images and texts from social media into a shared latent space, allowing for crossmodal retrieval and understanding, and thus allowing for earlier motif mining pipelines to be expanded beyond one modality. Finally, an extension of contrastive losses to more than two dimensions is formalized and studied, with trimodal and quadmodal models developed. These allow the pipelines to progress even further beyond bimodal models and given proper training data could allow them to encompass every type of artifact that could appear on social media.