Methods and Applications of Variational Inference via Deep Generative Models

In Bayesian models with latent variables, generating samples from the posterior distributions of the model parameters and latent variables is critical for statistical inference, uncertainty quantification, data generation, and predictions. Instead of sampling from the exact posteriors which can be challenging or computationally expensive, especially in high-dimensional models, variational inference (VI) methods that approximate the posterior distribution provides a viable alternative and have become more popular recently. The optimization of a VI problem is often iterative and requires frequent model and likelihood evaluation, which may be computationally expensive, especially when the evaluation cannot be parallelized.

To solve this problem, I developed the normalizing flow with an adaptive surrogate for computationally expensive models (NoFAS) framework to achieve a high-quality posterior approximation in my project. Instead of pursuing global accuracy, NoFAS calibrates the surrogate model for the true model and evaluates the likelihood function around the high posterior density regions in the parameter space. NoFAS delivers higher-quality variational inference, in contrast to non-adaptive surrogate modeling with the same amount of training data; It also achieves faster convergence compared with Markov Chain Monte Carlo methods and VI when using the true model instead of a surrogate. In my second project, I proposed canonical decomposition of the deterministic component of a regression model

that enables dimension reduction and input reconstruction simultaneously. I designed a double decoder together with a variational encoder to perform such decomposition. The experiments demonstrate that high-quality approximation can be achieved through the latent space with a lower dimension than that of the original input space, and therefore with a much simpler normalizing flow, compared with the one used for inference in the ambient parameter space. In my third project, I examined the application of differential privacy in NFs for both density estimation and variational inference with applications to electronic health records to release synthetic data or perform VI from data that may contain sensitive information and be subject to privacy attacks. The utility of privacy-preserving synthetic data and variational posterior samples are compared with the original data at different privacy loss budgets.