Gallery-Free Methods for Detecting and Recognizing People and Groups of Interest in the Wild

The 2013 Boston Marathon investigation presented an ample opportunity to apply advanced face recognition technology. Unfortunately, the crowded scenes proved too difficult for existing facial biometrics methods and databases. In this dissertation, we identify some of the limitations of this technology and present approaches for overcoming these deficiencies. These approaches underlie a suite of novel algorithms for detecting faces in unconstrained imagery; clustering images of faces into identity groups; finding people who appear frequently across a collection of scenes; and detecting groups of individuals who often appear together.

Face detection typically serves as the first step in a facial analysis pipeline. We elucidate the difficulties encountered on uncontrolled imagery by a simple yet strong-performing face detector. Although pose has traditionally been the focus of much of the research on face detection, we show that occlusion and blur have at least as significant of an impact for data collected from natural scenes. The results also indicate that classifiers with distinct error modes can result from blur-based perturbations of a detector training set, enabling the successful application of fusion techniques.

Identity clustering plays a similar role to face detection insofar as it can drive automatic 'pattern of life' analyses. We introduce an active clustering scheme, the Framework for Active Clustering with Ensembles (FACE), with an emphasis on clustering face images. FACE forms high-fidelity identity clusters by integrating a minimal amount of human feedback with automatic face recognition results. Recognition errors are mitigated through the solicitation of human decisions regarding ambiguously matched faces. This scheme precludes the need for a pre-existing gallery or database of known identities, since the gallery is essentially defined by the results of the clustering. Our experiments show that the FACE algorithm is more accurate and parsimonious than the state-of-the-art in active clustering, regardless of whether the human feedback is noisy. At the same time, the FACE algorithm promotes high performance in appearance frequency and affiliation analyses. These performance trends illustrate the potential efficacy of the proposed suite of algorithms in aiding data mining efforts encountered by the counter-terrorism and law enforcement communities.