Skip to article frontmatterSkip to article content
Machine Learning for Earth Observations

Probabilistic PCA

CSIC
UCM
IGEO

Context

We are given some measurements, y\boldsymbol{y}, which are sparse.

Dataset:Dtr={yn}n=1N,yn∈RD\begin{aligned} \text{Dataset}: && && \mathcal{D}_{tr} = \left\{ \boldsymbol{y}_n\right\}_{n=1}^{N}, && && \boldsymbol{y}_n\in\mathbb{R}^{D} \end{aligned}
(1)

Assumption

Measurements:y=y(s,t),y:RDs×R+→RDyTrue Signal:u=u(s,t),u:RDs×R+→RDuLatent Variable:z=z(s,t),z:RDs×R+→RDz\begin{aligned} \text{Measurements}: && && \boldsymbol{y} &= \boldsymbol{y}(\boldsymbol{s},t), && && \boldsymbol{y}:\mathbb{R}^{D_s}\times\mathbb{R}^+ \rightarrow\mathbb{R}^{D_y} \\ \text{True Signal}: && && \boldsymbol{u} &= \boldsymbol{u}(\boldsymbol{s},t), && && \boldsymbol{u}:\mathbb{R}^{D_s}\times\mathbb{R}^+ \rightarrow\mathbb{R}^{D_u} \\ \text{Latent Variable}: && && \boldsymbol{z} &= \boldsymbol{z}(\boldsymbol{s},t), && && \boldsymbol{z}:\mathbb{R}^{D_s}\times\mathbb{R}^+ \rightarrow\mathbb{R}^{D_z} \end{aligned}
(2)

Joint Distribution

p(y,u,z,θ)=p(y∣u,θ)p(u∣z,θ)p(z∣θ)p(θ)p(y,u,z,\theta) = p(y|u,\theta)p(u|z,\theta)p(z|\theta)p(\theta)
(3)

Process Relationships

Measurements:y∼p(y∣u,θ)True Signal:u∼p(u∣z,θ)Generative Model:z∼p(z∣θ)Parameters:θ∼p(θ)\begin{aligned} \text{Measurements}: && && \boldsymbol{y} &\sim p(\boldsymbol{y}|\boldsymbol{u},\boldsymbol{\theta}) \\ \text{True Signal}: && && \boldsymbol{u} &\sim p(\boldsymbol{u}|\boldsymbol{z},\boldsymbol{\theta}) \\ \text{Generative Model}: && && \boldsymbol{z} &\sim p(\boldsymbol{z}|\boldsymbol{\theta}) \\ \text{Parameters}: && && \boldsymbol{\theta} &\sim p(\boldsymbol{\theta}) \\ \end{aligned}
(4)

Example

Interpolation Operator:y=h(u;θ)+εy,εy∼N(0,Σy)Generative Model:u=T(z;θ)+εu,εu∼N(0,Σu)Latent Variable:z=μz+εz,εz∼N(0,Σz)\begin{aligned} \text{Interpolation Operator}: && && \boldsymbol{y} &= \boldsymbol{h}(\boldsymbol{u};\boldsymbol{\theta}) + \boldsymbol{\varepsilon}_y, && && \boldsymbol{\varepsilon}_y \sim \mathcal{N}(\boldsymbol{0}, \boldsymbol{\Sigma_y}) \\ \text{Generative Model}: && && \boldsymbol{u} &= \boldsymbol{T}(\boldsymbol{z};\boldsymbol{\theta}) + \boldsymbol{\varepsilon}_u, && && \boldsymbol{\varepsilon}_u \sim \mathcal{N}(\boldsymbol{0}, \boldsymbol{\Sigma_u}) \\ \text{Latent Variable}: && && \boldsymbol{z} &= \boldsymbol{\mu_z} + \boldsymbol{\varepsilon_z}, && && \boldsymbol{\varepsilon_z}\sim\mathcal{N}(0,\boldsymbol{\Sigma_z}) \end{aligned}
(5)

Formulation

Measurement Model

y∼N(y∣h(u,θh),Σy)\boldsymbol{y} \sim \mathcal{N}(\boldsymbol{y}\mid\boldsymbol{h}(\boldsymbol{u},\boldsymbol{\theta}_h), \boldsymbol{\Sigma_y})
(6)

where θ={θh,Σy}\boldsymbol{\theta} =\{ \boldsymbol{\theta}_h, \boldsymbol{\Sigma_y}\} where the parameters for this model and h\boldsymbol{h} is the interpolation operator.

QoI Generative Model

u∼N(u∣Wz+μ,Σu)\boldsymbol{u} \sim \mathcal{N}(\boldsymbol{u}\mid\mathbf{W}\mathbf{z} + \boldsymbol{\mu}, \boldsymbol{\Sigma_u})
(7)

where θ={W,μ,Σu}\boldsymbol{\theta} =\{ \mathbf{W},\boldsymbol{\mu}, \boldsymbol{\Sigma_u}\} where the parameters for this model.

Latent Variable Model

z∼N(z∣μz,Σz)\boldsymbol{z} \sim \mathcal{N}(\boldsymbol{z}\mid\boldsymbol{\mu_z}, \boldsymbol{\Sigma_z})
(8)

where θ={μz,Σz}\boldsymbol{\theta} =\{ \boldsymbol{\mu_z}, \boldsymbol{\Sigma_z}\} where the parameters for this model.

Joint Distribution

[uz]∼N([μuμz],[WW⊤+ΣuWW⊤Σz],)\begin{bmatrix} \boldsymbol{u} \\ \boldsymbol{z} \end{bmatrix} \sim \mathcal{N} \left( \begin{bmatrix} \boldsymbol{\mu_u} \\ \boldsymbol{\mu_z} \end{bmatrix}, \begin{bmatrix} \mathbf{WW}^\top+\boldsymbol{\Sigma_u} && \mathbf{W} \\ \mathbf{W}^\top && \mathbb{\Sigma_z} \end{bmatrix}, \right)
(9)

Posterior Distributions

We have the posterior distribution for the QoI, u\boldsymbol{u}.

p(u∣z;θ)=N(u∣μu∣z,Σu∣z)p(\boldsymbol{u}|\boldsymbol{z};\boldsymbol{\theta}) = \mathcal{N}\left(\boldsymbol{u}\mid\boldsymbol{\mu_{u|z}},\boldsymbol{\Sigma_{u|z}}\right)
(10)

where:

μu∣z=μu+WΣz−1(z−μz)Σu∣z=Σu+WΣu−1W⊤\begin{aligned} \boldsymbol{\mu_{u|z}} &= \boldsymbol{\mu_u} + \mathbf{W}\boldsymbol{\Sigma_z}^{-1}(\boldsymbol{z} - \boldsymbol{\mu_z}) \\ \boldsymbol{\Sigma_{u|z}} &= \boldsymbol{\Sigma_u} + \mathbf{W}\boldsymbol{\Sigma_u}^{-1}\mathbf{W}^\top \\ \end{aligned}
(11)

We have the posterior distribution for the latent space, z\boldsymbol{z}.

q(z∣u;θ)=N(z∣μz∣u,Σz∣u)q(\boldsymbol{z}|\boldsymbol{u};\boldsymbol{\theta}) = \mathcal{N}\left(\boldsymbol{z}\mid\boldsymbol{\mu_{z|u}},\boldsymbol{\Sigma_{z|u}}\right)
(12)

where:

μz∣u=μz+W⊤(WW⊤+Σu)−1(u−μu)Σz∣u=Σz+W⊤(WW⊤+Σu)−1W⊤\begin{aligned} \boldsymbol{\mu_{z|u}} &= \boldsymbol{\mu_z} + \mathbf{W}^\top \left(\mathbf{WW}^\top + \boldsymbol{\Sigma_u}\right)^{-1} (\boldsymbol{u} - \boldsymbol{\mu_u}) \\ \boldsymbol{\Sigma_{z|u}} &= \boldsymbol{\Sigma_z} + \mathbf{W}^\top \left(\mathbf{WW}^\top + \boldsymbol{\Sigma_u}\right)^{-1} \mathbf{W}^\top \\ \end{aligned}
(13)

Marginal Likelihood

p(u)=∫p(u∣z)p(z)dzp(\boldsymbol{u}) = \int p(\boldsymbol{u}|\boldsymbol{z})p(\boldsymbol{z})d\boldsymbol{z}
(14)
Algorithms
Probabilistic PCA