Statistical Neural Networks: Concepts, Frameworks and Applications

Doctoral Dissertation
Thumbnail

Abstract

As one of the fundamental issues in the fields such as computer vision and deep learning, accelerating the inference speed of the convolutional neural network (CNN) has attracted a lot of attention, which is to reduce the gap between the desired high performance of deep CNN and the required huge amount of computation. Various techniques have been explored to address this challenge, such as network compression and quantization, lightweight networks targeting resource-constrained platforms, dynamic computation graphs providing efficient early exits, and, on top of all these techniques, network parallelization. Although a large body of works exists in neural network parallelization, most of them focus on the training phase only or distributing a batch of instances to multiple computation cores. None of these parallelization works, however, helps to reduce the inference latency of a single instance (such as an image), which is critical for real-time applications. Existing techniques for single image inference include operator parallelism and model parallelism. The former technique explores concurrency in operators such as convolution, and the latter distributes kernels of convolutional layers across multiple cores. These approaches do not offer good scalability and usually cannot fully utilize a large number of cores available in modern high-performance computing platforms.

In this dissertation, we introduce a new general framework named statistical convolutional neural network (SCNN) to speedup the instance inference with the help of independent component analysis (ICA), which can be applied to various tasks. As a general framework, SCNN can be implemented with different neural network backbones to speedup, while it is orthogonal to the existing speedup methods. We use ICA to decompose the spatio-temporal correlated data and propagate the extracted essential features as the learning process. The performance evaluations of SCNN in video object detection and 3D cardiac cine MRI segmentation shows that SCNN could achieve a large speedup compared with the existing methods. We further introduce ICA-Net as an improvement of SCNN in accuracy, throughput, and latency. The improved framework is implemented and verified by multiple tasks including image classification, image object detection, video object detection, and 3D cardiac cine MRI segmentation. The latencies of the models are all largely reduced and the accuracies achieve state-of-the-art.

Attributes

Attribute NameValues
Author Tianchen Wang
Contributor Jinjun Xiong, Committee Member
Contributor Meng Jiang, Committee Member
Contributor Walter Scheirer, Committee Member
Contributor Yiyu Shi, Research Director
Degree Level Doctoral Dissertation
Degree Discipline Computer Science and Engineering
Degree Name Doctor of Philosophy
Banner Code
  • PHD-CSE

Defense Date
  • 2020-05-01

Submission Date 2020-05-28
Subject
  • convolutional neural network, independent component analysis, neural network acceleration

Record Visibility Public
Content License
Departments and Units
Catalog Record

Files

Please Note: You may encounter a delay before a download begins. Large or infrequently accessed files can take several minutes to retrieve from our archival storage system.