A good paper which highlights some import differences between the FITC, DTC and VFE. It provides a clear notational differences and also mentions how VFE is a special case of DTC.

-> Paper

Other Papers

-> On Sparse Variational meethods and the KL Divergence between Stochastic Processes - Matthews et. al. (2015)

Stochastic Variational Inference (SVI)¶

Gaussian Processes for Big Data - Hensman et al. (2013)

-> Paper

Expectation Propagation (EP)¶

A Unifying Framework for Gaussian Process Pseudo-Point Approximations using Power Expectation Propagation - Bui (2017)

A good summary of all of the methods under one unified framework called the Power Expectation Propagation formula.

-> Paper

-> Code: Exact and Sparse Power EP

-> Updated | Other

-> Related Code

Variational¶

Rates of Convergence for Sparse Variational Gaussian Process Regression - Burt et. al. (2019)

All you need to do is cite this paper whenever people don't believe that Sparse GPs aren't good at approximating Exact GPs.

-> Paper | 💻 Code -> Convergence of Sparse Variational Inference in Gaussian Processes Regression | Code

Latest¶

Deep Structured Mixtures of Gaussian Processes - Trapp et. al. (2019)

Going back to the old days of improving the local-expert technique.
Sparse Gaussian Process Regression Beyond Variational Inference - Jankowiak et. al. (2019)

Other¶

Adversarial Robustness Guarantees for Classification with Gaussian Processes - Blass et. al. (2020)

-> Paper

Thesis Explain¶

Often times the papers that people publish in conferences in Journals don't have enough information in them. Sometimes it's really difficult to go through some of the mathematics that people put in their articles especially with cryptic explanations like "it's easy to show that..." or "trivially it can be shown that...". For most of us it's not easy nor is it trivial. So I've included a few thesis that help to explain some of the finer details. I've arranged them in order starting from the easiest to the most difficult.

GPR Techniques - Bijl (2016)
Chapter V - Noisy Input GPR
Non-Stationary Surrogate Modeling with Deep Gaussian Processes - Dutordoir (2016)
Chapter IV - Finding Uncertain Patterns in GPs
Nonlinear Modeling and Control using GPs - McHutchon (2014)
Chapter II - GP w/ Input Noise (NIGP)
Deep GPs and Variational Propagation of Uncertainty - Damianou (2015)
Chapter IV - Uncertain Inputs in Variational GPs
Chapter II (2.1) - Lit Review
Bringing Models to the Domain: Deploying Gaussian Processes in the Biological Sciences - Zwießele (2017)
Chapter II (2.4, 2.5) - Sparse GPs, Variational Bayesian GPLVM

Sparse Gaussian Process Approximations and Applications by Van der Wilk (2018)

-> Thesis

Presentations¶

Variational Inference for Gaussian and Determinantal Point Processes - Titsias (2014)

Notes¶

On the paper: Variational Learning of Inducing Variables in Sparse Gaussian Processees - Bui and Turner (2014)

Gory Details Blogs¶

Some resources that break down some of the intricate mathematical details that are sometimes lost within the literature.

Bill Engels
Inducing point methods to speed up GPs - (01-06-2017)
FITC and VFE - (28-06-2018)
PyMC3 FITC/VFE implementation notes - (29-06-2018)
VFE approximation for Gaussian processes, the gory details - (20-08-2018)
Variational Free Energy for Sparse GPs - Gonzalo (04-2018)
Sparse and Variational Gaussian Process — What To Do When Data is Large - Wei Yi (06-2020) | code
A Cheatsheet for Sparse Variational Gaussian Processes - Louis Tiao (09-2020)
Derivation of SGPR Equations
GaussianProcesses.jl - Lots of details in here.

Code Examples¶

Some examples where people have implemented the algorithms very didactically.

SVGP - Recyclable GP