Advancing Biometrics and Image Forensics through Vision and Learning Systems

The internet is a vast and rapidly changing landscape. For every new algorithm invented to increase the security and reliability of our society, an adversarial group coalesces in reaction. Some may argue that this constant arms race between defenders and attackers is currently being won by the latter. To shift the tides, we need scalable approaches to counter their attacks, whatever they may be, at the scale and speed with which the internet moves. This dissertation focuses on developing computer vision algorithms to help ensure security within three main areas:

1) In the video surveillance chapter (Ch. 2), we focus on the problem of Re-Identification, or finding all appearances of a given subject across a given camera system. Most all existing ReID approaches employ local and global body features (e.g., clothing color and pattern, body symmetry, etc.). These `body ReID' methods implicitly assume that facial resolution is too low to aid in the ReID process. We assert that faces, even when captured in low resolution environments, may contain unique and stable features for ReID. Such `facial ReID' approaches are relatively unexplored in the literature. We explore facial ReID using a new dataset that was collected from a real surveillance network in a municipal rapid transit system

2) In the facial recognition chapter (Ch. 3), we focus on the problem of off-pose faces. If a Convolutional Neural Network (CNN) is intended to tolerate facial pose, then we face an important question: should this training data be diverse in its pose distribution, or should face images be normalized to a single pose in a pre-processing step? To address this question, we evaluate a number of facial landmarking algorithms and a popular frontalization method to understand their effect on facial recognition performance.

3) In the Image Forensics and retrieval chapters (Ch. 4, 5, and 6), we focus on building novel image retrieval algorithms that can retrieve and trace modified images back to their origins. Images from social media can reflect diverse viewpoints, heated arguments, and expressions of creativity. They are often composites, borrowing content from many different sources to create a single image. Taking into account these attributes introduces new complexity to image search tasks. We propose multiple new frameworks for image retrieval that model object-level regions, allowing for fine-grained object level retrieval results. These retrieval results can help aid in the task of verifying the forensic integrity of an image. We then utilize these algorithms to perform real-world analysis on data scraped from Twitter and Instagram from the 2019 Indonesian Election.