University of Notre Dame
Browse
THEISENW42024D.pdf (5.09 MB)

Motif Mining: Towards Multimodal Retrieval for Discovering Disinformation Trends in Social Media

Download (5.09 MB)
dataset
posted on 2024-05-04, 12:05 authored by William Theisen
As the world has become more and more connected online, so too has the amount of data that has been produced. Alongside this increase in data production the amount of disinformation has drastically increased. Coming to the fore in recent years and playing outsized roles in the politics of a myriad of countries, disinformation has become a hot topic. The scale of the data, in particular images and videos, has grown beyond the human capacity for processing in any detail. To help aid reviewers of online content we propose extending the idea of image provenance to include multimodal motif mining. This formalization and implementation of the problem of visual motif mining is designed to help humans process the vast amounts of visual data more quickly and to aid in the discovery of emerging disinformation trends in the online space. Early efforts in this space focused on monomodal capabilities, considering images and texts separately. Efforts to combine the two via fusion techniques left something to be desired. With the release of contrastive language-image pre-training techniques a new door has been opened for multimodal shared embedding spaces. In this work I share some of the earlier monomodal motif mining strategies, focusing on images as their subject. This is followed by studies in using CLIP-based strategies for projecting images and texts from social media into a shared latent space, allowing for crossmodal retrieval and understanding, and thus allowing for earlier motif mining pipelines to be expanded beyond one modality. Finally, an extension of contrastive losses to more than two dimensions is formalized and studied, with trimodal and quadmodal models developed. These allow the pipelines to progress even further beyond bimodal models and given proper training data could allow them to encompass every type of artifact that could appear on social media.

History

Date Created

2024-04-10

Date Modified

2024-05-02

Defense Date

2024-04-05

CIP Code

  • 14.0901

Research Director(s)

Walter Scheirer

Committee Members

Tim Weninger|Adam Czajka|Daniel Moreira

Degree

  • Doctor of Philosophy

Degree Level

  • Doctoral Dissertation

Language

  • English

Library Record

006583153

OCLC Number

1432449414

Publisher

University of Notre Dame

Program Name

  • Computer Science and Engineering

Usage metrics

    Dissertations

    Categories

    No categories selected

    Exports

    RefWorks
    BibTeX
    Ref. manager
    Endnote
    DataCite
    NLM
    DC