All Things Gaussian

Multivariate Gaussian¶

where:

• $\boldsymbol{u}\in\mathbb{R}^{D}$ - $D$-dimensional vector

• $\boldsymbol{\mu}\in\mathbb{R}^{D}$ - $D$-dimensional mean vector

• $\boldsymbol{\Sigma}\in\mathbb{R}^{D\times D}$ - $D\times D$-dimensional covariance matrix

Mahalanobis Distance¶

We often call this the quadratic term.

Case I: Identity¶

Case II: Scalar¶

Case III: Diagonal¶

Case IV: Decomposition¶

Case V: Full Covariance¶

Masked Likelihood¶

Conditional Gaussian Distributions¶

We have the joint distribution for the latent variables, $\boldsymbol{z}$, and a QoI, $\boldsymbol{u}$.

Marginal Distributions¶

We have the marginal distribution for the variable, $\boldsymbol{z}$

We have the conditional likelihood for the variable, $\boldsymbol{u}$

Conditional Distributions¶

We have the conditional likelihood for the variable, $\boldsymbol{z}$

We have the conditional likelihood for the variable, $\boldsymbol{u}$

Scaling¶

there

Matrix Inversions¶

The primary thing we want to do when

Cholesky Decomposition¶

We can decompose the matrix into a Cholesky which is an upper (or lower) triangular matrix.

We can to the inversion

Something that is easier to deal with is the matrix solve:

For this, we need a special solver

A: Array["D D"] = ...
b: Array["D M"] = ...
I: Array["D D"] = eye_like(A)
# cholesky decomposition
L: Array["D D"] = cholesky(K, lower=True)
L_inv: Array["D D"] = cho_solve(L, I, lower=True)
x: Array["D M"] = cho_solve(L, b, lower=True)

Conjugate Gradient¶

Woodbury Approximation¶

We can find some lower dimensional subspace. For example, we can use the SVD decomposition

Looking at equation (34), we can take the inverse.

Inducing Points¶

We can use a subset of the points and calculate th covariance.

Now, we can easily find the inverse

Approximate Conditional Distributions¶

We have each of the terms as

We have the conditional likelihood for the variable, $\boldsymbol{z}$

We have the conditional likelihood for the variable, $\boldsymbol{u}$

Linear Conditional Gaussian Model¶

We have a latent variable which is Gaussian distributed:

We have a QoI which we believe is a linear transformation of the latent variable

Recall the joint distribution given in equation (6). We can write each of the terms as:

• $\boldsymbol{\Sigma_{zz}}=\boldsymbol{\Sigma_{z}}$

• $\boldsymbol{\Sigma_{uu}}=\boldsymbol{\Sigma_{u}}$

• $\boldsymbol{\Sigma_{uz}}=\boldsymbol{\Sigma_{u}}$

• $\boldsymbol{\bar{u}}=\mathbf{A}\boldsymbol{z} + \mathbf{b}$

Taylor Expansion¶

Taylor Expansion¶

Joint Distribution¶

Mean Function¶

Sample & Population Moments¶

Examples:

• Global Mean Surface Temperature, $x\in\mathbb{R}^{N\times D}$, $N=\text{Models}$, $D=\sum D_T D_\Omega$

• Spatial Scene, $x\in\mathbb{R}^{N\times D}$, $N=\text{Ensembles}$, $D=\text{Space}$

• Spatiotemporal Trajectory, $x\in\mathbb{R}^{N\times D}$, $N=\text{Space/Time}$, $D=\text{Time/Space}$

• Ensemble of Trajectories, $x\in\mathbb{R}^{N\times D}$, $N=\text{Ensembles}$, $D=\text{Time x Space}$

Gaussian Approximation Algorithm¶

Moment Estimation¶

Samples¶

We can do all of these operations in matrix form.

Note: the perturbation matrix in this form is equivalent to the kernel centering operation (see scikit-learn docs). It allows one to center the gram matrix without explicitly computing the mapping.

Population¶

We can do all of these operations in matrix form.

Matrix Identities¶

Woodbury Formula¶

Sherman-Morrison-Woodbury Formula¶

This is basically the same as the Woodbury formula (34) except the matrix, $\mathbf{A}$, is the identity, $\mathbf{I}$ and the decomposition is between

There are some lemmas to this which have been shown to be very useful in practice.

Sylvester Determinant Lemma¶

Weinsten-Aronszajin Identity¶

Decompositions¶

Eigenvalue Decomposition¶

Nystrom Approximation¶

Inversion¶

We can use the matrix inversion properties from equation (34) to decompose the Nyström approximation into a cheaper inversion.

Determinant¶

We can use the determinant inversion properties from equation (37) to decompose the Nyström approximation to be cheaper.

Random Fourier Features¶

Inversion¶

We can use the matrix inversion properties from equation (34) to decompose the Nyström approximation into a cheaper inversion.

Determinant¶

We can use the determinant inversion properties from equation (37) to decompose the Nyström approximation to be cheaper.

Inducing Points¶

We can demonstrate the decomposition as: