New Deep Learning Methods for Annotation-Efficient Medical Image Analysis

Medical image analysis is an impactful domain in computer science where the goal is to computationally extract knowledge from images for health research, clinical assessment, and computer-aided diagnostics. Extracting these insights commonly involves two core tasks: image classification, which categorizes images into medically-relevant classes, and image segmentation, which classifies each pixel or voxel within an image. In computer vision, these tasks are increasingly relegated to large Transformer or “foundation” models where training is performed on large corpora of natural scene images. Despite these advancements, there remains large semantic and appearance gaps between natural and medical data which significantly diminish the transferability of these models to medical applications. The medical domain also contends with acute data challenges involving the prohibitive costs of acquiring, labeling, and publicizing data. To address these challenges, we introduce six new deep learning methods that enhance data efficiency and improve task performance by harnessing prior knowledge inherent in medical images. The first trio of methods employs anatomical, shape, and staining priors with small fully annotated datasets to advance sperm morphology classification, anatomical structure segmentation, and cancer survival regression, respectively. The latter three approaches utilize spatial and semantic priors to facilitate unsupervised representation learning, which learns expressive features from images without any annotations for downstream segmentation tasks. These methodologies not only bridge the performance gap observed between medical tasks and recent natural image frameworks, but also demonstrate that favorable model performance and data efficiency can be achieved through judicious design using effective priors. Finally, to extend the applicability and effectiveness of the ideas introduced above, we propose and discuss four future research avenues.