Overview

• Author: J. Emmanuel Johnson
• Email: jemanjohnson34@gmail.com
• Repo: github.com/jejjohnson/uncertain_gps

We will do a quick overview to show how we can account for input errors in Gaussian process regression models.

Problem Statement

Standard

$$y=f(\mathbf{x})+{ϵ}_y$$

where ${ϵ}_y\sim \mathcal{N}(0,{\sigma }_y^2)$. Let $\mathbf{x}={\mu }_{\mathbf{x}}+{\mathrm{\Sigma }}_{\mathbf{x}}$.

Observe Noisy Estimates

$$y=f(\mathbf{x})+{ϵ}_y$$

Observation Means only

$$y=f({\mu }_{\mathbf{x}})+{ϵ}_y$$

Posterior Predictions

$$\mu ({\mathbf{x}}_{\ast })={\mathbf{k}}_{\ast }(\mathbf{K}+\lambda {\mathbf{I}}_N{)}^{-1}\mathbf{y}={\mathbf{k}}_{\ast }\alpha$$

$${\nu }_{GP\ast }^2={\sigma }_y^2+k_{\ast \ast }-{\mathbf{k}}_{\ast }(\mathbf{K}+{\sigma }_y^2{\mathbf{I}}_N{)}^{-1}{\mathbf{k}}_{\ast }^{\mathrm{\top }}$$

Linearized Approximation

Where we take the Taylor expansion of the predictive mean and variance function. The mean function stays the same:

$${\mu }_{eGP\ast }({\mathbf{x}}_{\mathbf{\ast }})={\mathbf{k}}_{\ast }\alpha$$

but the predictive variance term gets changed slightly:

$${\nu }_{eGP\ast }^2={\nu }_{GP\ast }^2({\mathbf{x}}_{\ast })+{\mathrm{\nabla }}_{{\mu }_{\ast }}{\mathrm{\Sigma }}_x{\mathrm{\nabla }}_{{\mu }_{\ast }}^{\mathrm{\top }}+\text{Tr}\left\{\frac{{\partial }^2{\nu }_{GP\ast }^2({\mathbf{x}}_{\mathbf{\ast }})}{\partial {\mathbf{x}}_{\mathbf{\ast }}\partial {{\mathbf{x}}_{\mathbf{\ast }}}^{\mathrm{\top }}}{|}_{{\mathbf{x}}_{\mathbf{\ast }}={\mu }_{{\mathbf{x}}_{\mathbf{\ast }}}}{\mathrm{\Sigma }}_{{\mathbf{x}}_{\mathbf{\ast }}}\right\}$$

with the term in red being the derivative of the predictive mean function multiplied by the variance.

Notes: * Assumes known variance * Assumes $D\times D$ covariance matrix for multidimensional data * Quite inexpensive to implement * The 3^rd term (the 2^nd order component of the Taylor expansion) has been show to not make a huge difference

egp_moment1 = jax.jfwd(posterior, args_num=(None, 0))

egp_moment2 = jax.hessian(posterior, args_num=(None, 0))

Moment-Matching

Mean Predictions

$${\mu }_{eGP\ast }({\mathbf{x}}_{\mathbf{\ast }})={\mathbf{q}}^{\mathrm{\top }}\alpha$$

where:

$$q_i=|{\mathrm{\Lambda }}^{-1}{\mathrm{\Sigma }}_{{\mathbf{x}}_{\mathbf{\ast }}}+\mathbf{I}{|}^{-1/2}\exp [-\frac{1}{2}({\mu }_{\ast }-{\mathbf{x}}_i)({\mathrm{\Sigma }}_{{\mathbf{x}}_{\mathbf{\ast }}}+\mathrm{\Lambda }{)}^{-1}({\mu }_{\ast }-{\mathbf{x}}_j)]$$

Variance Predictions

$${\nu }_{eGP\ast }^2$$

Variational

Assumes we have a variational distribution function

$$\mathcal{L}(\theta )={\text{D}}_{\text{KL}}[q(\mathbf{f})q(\mathbf{X})||p(\mathbf{f}\mathbf{|}\mathbf{X})p(\mathbf{X})]$$

---

Other Resources

Gaussian Process Model Zoo

---

Datasets

We use some toy datasets which including:

1. "near square sine wave"

$$f(x)=\sin (\frac{\pi }{c}\cos (5+\frac{x}{2}))$$

1. The sigmoid curve

$$f(x)=\frac{1}{1+\exp (-x)}$$

1. Mauna Loa Ice Core Data | Data Portal

$$

$$

1. Spatial IASI Data