Skip to article frontmatterSkip to article content

Trade-offs

Pros

Mesh-Free

Lots of Data

Cons

Transfer Learning

Data

xϕ∈RDϕ,      u∈R\begin{aligned} \mathbf{x}_\phi \in \mathbb{R}^{D_\phi}, \;\;\; \mathbf{u} \in \mathbb{R}^{} \end{aligned}
(1)

Model

fθ:X→U\boldsymbol{f_\theta}:\mathcal{X} \rightarrow \mathcal{U}
(2)

Architectures

We are interested in the case of regression. We have the following generalized architecture.

x(1)=ϕ(x;γ)x(ℓ+1)=NNℓ(x(ℓ);θℓ)f(x;θ,γ)=w(L)x(L)+b(L)\begin{aligned} \mathbf{x}^{(1)} &= \boldsymbol{\phi} \left( \mathbf{x} ; \boldsymbol{\gamma}\right) \\ \mathbf{x}^{(\ell+1)} &= \text{NN}_\ell \left( \mathbf{x}^{(\ell)}; \boldsymbol{\theta}_\ell\right)\\ \boldsymbol{f}(\mathbf{x}; \boldsymbol{\theta},\boldsymbol{\gamma}) &= \mathbf{w}^{(L)}\mathbf{x}^{(L)} + \mathbf{b}^{(L)} \end{aligned}
(3)

where ϕ\boldsymbol{\phi} is the basis transformation with some hyperparameters γ, NN\text{NN} is the neural network layer parameterized by θ\boldsymbol{\theta}, and we have LL layers, L={1,2,…,ℓ,…,L−1,L}L = \{1, 2, \ldots, \ell, \ldots, L-1, L\}

Standard Neural Network

In the standard neural network, we typically have the following standard functions

ϕ(x)=xNNsiren(x(ℓ);θ)=σ(w(ℓ)x(ℓ)+b(ℓ)),θ={w(ℓ),b(ℓ)}\begin{aligned} \boldsymbol{\phi}(\mathbf{x}) &= \mathbf{x} \\ \text{NN}_{siren} \left( \mathbf{x}^{(\ell)}; \boldsymbol{\theta}\right) &= \boldsymbol{\sigma} \left( \mathbf{w}^{(\ell)} \mathbf{x}^{(\ell)} + \mathbf{b}^{(\ell)} \right), \hspace{10mm} \boldsymbol{\theta} = \{ \mathbf{w}^{(\ell)}, \mathbf{b}^{(\ell)} \} \end{aligned}
(4)

So more explicitly, we can write it as:

x(1)=xf(ℓ)(x(ℓ))=σ(w(ℓ)x(ℓ)+b(ℓ))f(L)(x(L))=w(L)x(L)+b(L)\begin{aligned} \mathbf{x}^{(1)} &= \mathbf{x} \\ \boldsymbol{f}^{(\ell)}(\mathbf{x}^{(\ell)}) &= \boldsymbol{\sigma} \left( \mathbf{w}^{(\ell)} \mathbf{x}^{(\ell)} + \mathbf{b}^{(\ell)} \right)\\ \boldsymbol{f}^{(L)}(\mathbf{x}^{(L)}) &= \mathbf{w}^{(L)}\mathbf{x}^{(L)} + \mathbf{b}^{(L)} \end{aligned}
(5)

where ℓ={1,2,…,L−1}\ell = \{1, 2, \ldots, L-1\}. Noteably:

  • The first layer is the identity (i.e. there is not basis function transformation)
  • The second layer is the standard neural network architecture, i.e. a linear function and a nonlinear activation function
  • The final layer is always a linear function (in regression; classification would have a sigmoid)

Positional Encoding

Fourier Features

ϕ(x)=[sin⁡(ωx)cos⁡(ωx)],ω∼p(ω;γ)\boldsymbol{\phi} \left(\mathbf{x}\right) = \begin{bmatrix} \sin \left( \boldsymbol{\omega}\mathbf{x}\right) \\ \cos \left( \boldsymbol{\omega} \mathbf{x}\right) \end{bmatrix},\hspace{10mm} \boldsymbol{\omega} \sim p(\boldsymbol{\omega};\gamma)
(6)
MethodKernelDistribution
GaussianN(0,1σ2Ir)\mathcal{N}(\mathbf{0},\frac{1}{\sigma^2}\mathbf{I}_r)
LaplacianCauchy()\text{Cauchy}()
CauchyLaplace()\text{Laplace}()
MaternBessel()\text{Bessel}()
ArcCosine

Alternative Formulation

ϕ(x)=2Drffcos⁡(ωx+b)\boldsymbol{\phi}(\mathbf{x}) = \sqrt{\frac{2}{D_{rff}}}\cos \left( \boldsymbol{\omega}\mathbf{x} + \boldsymbol{b}\right)
(7)

where ω∼p(ω)\boldsymbol{\omega} \sim p(\boldsymbol{\omega}) and b∼U(0,2π)\boldsymbol{b} \sim \mathcal{U}(0,2\pi).

Source:

  • Blog - Gregory Gundersen
  • Random Features for Large-Scale Kernel Machines - Rahimi & Recht (2008) - Paper
  • Random Features for Kernel Approximation: A Survey on Algorithms, Theory, and Beyond - Liu et al (2021)
  • Scalable Kernel Methods via Doubly Stochastic Gradients - Dai et al (2015) -

SIREN

σ=sin⁡(ω0(wx+b))\boldsymbol{\sigma} = \sin \left( \boldsymbol{\omega}_0 (\mathbf{wx} + b)\right)
(8)
σ=α⊙sin⁡(wx+b)\boldsymbol{\sigma} = \boldsymbol{\alpha} \odot \sin \left( \mathbf{wx} + \mathbf{b} \right)
(9)
FiLM(x)=α⊙x+β\text{FiLM}(\mathbf{x}) = \boldsymbol{\alpha} \odot \mathbf{x} + \boldsymbol{\beta}
(10)
  • pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis - Chan et al (2021)
  • COIN++:

Extended

σ=sin⁡(γ(wx+b)+β)\begin{aligned} \boldsymbol{\sigma} = \sin \left( \boldsymbol{\gamma}(\mathbf{wx} + b\right) + \boldsymbol{\beta}) \end{aligned}
(11)

where γ\boldsymbol{\gamma} corresponds to the frequencies and β\boldsymbol{\beta} corresponds to the phase shifts.

Modulation

Modulation is

fℓ(x,z;θ):=hMℓ(  NN(x;θNN)  ,  M(z;θM)  ) \boldsymbol{f}^\ell(\mathbf{x},\mathbf{z};\boldsymbol{\theta}) := \boldsymbol{h}_M^\ell\left(\;\text{NN}(\mathbf{x};\boldsymbol{\theta}_{NN})\;,\; \text{M}(\mathbf{z};\boldsymbol{\theta}_{M}) \;\right)
(12)

where NNNN is the output of the neural network wrt the input, x\mathbf{x}, where MM is the output of the modulation function wrt the latent variable, z\mathbf{z}, and × is an arbitrary operator.

  • Additive Layer
  • Affine Layer
  • Neural Implicit Flowss
  • Neural Flows

FILM, 2020

Mehta, 2021

Dupoint, 2022

Neural Implicit Flows Pan, 2022

Neural

Affine Modulation

Affine Modulations

z(1)=xz(k+1)=σ((w(k)z(k)+b(k))⊙sm(z)+am(z))f(x)=wKzK+b(k)\begin{aligned} \mathbf{z}^{(1)} &= \mathbf{x} \\ \mathbf{z}^{(k+1)} &= \boldsymbol{\sigma} \left( \left(\mathbf{w}^{(k)} \mathbf{z}^{(k)} + \mathbf{b}^{(k)}\right)\odot \boldsymbol{s}_m(\mathbf{z}) + \boldsymbol{a}_m(\mathbf{z}) \right)\\ \boldsymbol{f}(\mathbf{x}) &= \mathbf{w}^{K}\mathbf{z}^{K} + \mathbf{b}^{(k)} \end{aligned}
(13)

Shift Modulations

Neural Implicit Flows

In this work, we have a version of the Modulated Siren as mentioned above. However, they use a version that separates the space and time neural networks.

f(xϕ,t)=NNspace(xϕ;NNtime(t)) \boldsymbol{f}(\mathbf{x}_\phi, t) = \text{NN}_{space}(\mathbf{x}_\phi;\text{NN}_{time}(t))
(14)

Multiplicative Filter Networks

z(1)=xz(k+1)=σ(w(k)z(k)+b(k))f(x)=wKzK+b(k)\begin{aligned} \mathbf{z}^{(1)} &= \mathbf{x} \\ \mathbf{z}^{(k+1)} &= \boldsymbol{\sigma} \left( \mathbf{w}^{(k)} \mathbf{z}^{(k)} + \mathbf{b}^{(k)} \right) \\ \boldsymbol{f}(\mathbf{x}) &= \mathbf{w}^{K}\mathbf{z}^{K} + \mathbf{b}^{(k)} \end{aligned}
(15)

where K={1,2,…,K}K = \{1, 2, \ldots, K\}

Non-Linear Functions

FOURIERNET

This method corresponds to the random Fourier Feature transformation.

g(ℓ)(x;θ(ℓ))=sin⁡(w(ℓ)x+b(ℓ))\boldsymbol{g}^{(\ell)}(\mathbf{x};\boldsymbol{\theta}^{(\ell)}) = \sin\left( \mathbf{w}^{(\ell)}\mathbf{x} + \mathbf{b}^{(\ell)}\right)
(16)

where the parameters to be learned are:

θ(ℓ)={wd(ℓ),    bd(ℓ)}\boldsymbol{\theta}^{(\ell)} = \{\mathbf{w}_d^{(\ell)}, \;\; \mathbf{b}^{(\ell)}_d \}
(17)

GABORNET

This method tries to improve upon the Fourier representation. The Fourier representation has global support and would have more difficulties representing more local features. The Gabor filter (see below) will be able to capture both frequency and spatial locality component.

g(ℓ)(x;θ(ℓ))=exp⁡(−γd(ℓ)2∣∣x−μd(ℓ)∣∣22)⊙sin⁡(w(ℓ)x+b(ℓ))\boldsymbol{g}^{(\ell)}(\mathbf{x};\boldsymbol{\theta}^{(\ell)}) = \exp\left( - \frac{\gamma_d^{(\ell)}}{2}||\mathbf{x} - \boldsymbol{\mu}_d^{(\ell)}||_2^2 \right) \odot \sin\left( \mathbf{w}^{(\ell)}\mathbf{x} + \mathbf{b}^{(\ell)}\right)
(18)

where the parameters to be learned are:

θ(ℓ)={γd(ℓ)∈R,    μd(ℓ),    wd(ℓ),    bd(ℓ)}\boldsymbol{\theta}^{(\ell)} = \{ \gamma_d^{(\ell)} \in \mathbb{R},\;\;\boldsymbol{\mu}_d^{(\ell)}, \;\; \mathbf{w}_d^{(\ell)}, \;\; \mathbf{b}^{(\ell)}_d \}
(19)

Sources:

Probabilistic

Deterministic

L(θ)=argmin θλ∑n∈D∣∣f(xn;θ)−un∣∣22−log⁡p(θ)\mathcal{L}(\boldsymbol{\theta}) = \underset{\boldsymbol{\theta}}{\text{argmin }} \lambda \sum_{n \in \mathcal{D}} ||\boldsymbol{f}(\mathbf{x}_n;\boldsymbol{\theta}) - \boldsymbol{u}_n||_2^2 - \log p(\boldsymbol{\theta})
(20)

Normalizing Flows

Bayesian

  • Random Feature Expansions (RFEs)

Physics Constraints

Mass

Momentum

QG Equations

Applications

Interpolation

Surrogate Modeling

Sampling

Feature Engineering

x∈RDϕ,D={lat, lon, time}\mathbf{x} \in \mathbb{R}^{D_\phi}, \hspace{10mm} D = \{ \text{lat, lon, time} \}
(21)

Spatial Features

For the spatial features, we have spherical coordinates (i.e. longitude and latitude)

x=rcos⁡(λ)cos⁡(ϕ)y=rcos⁡(λ)sin⁡(ϕ)z=rsin⁡(λ)\begin{aligned} x &= r \cos(\lambda)\cos(\phi) \\ y &= r \cos(\lambda)\sin(\phi) \\ z &= r \sin(\lambda) \end{aligned}
(22)

where λ is the latitude, ϕ is the longitude and rr is the radius. Here x,y,zx,y,z are bounded between 0 and 1.

Temporal Features

Tanh

f(t)=tanh⁡(t)f(t) = \tanh(t)
(23)

Fourier Features

Sinusoidal Positional Encoding

ϕ(t)=[sin⁡(ωkt)cos⁡(ωkt)]\boldsymbol{\phi}(t) = \begin{bmatrix} \sin(\boldsymbol{\omega}_k t) \\ \cos(\boldsymbol{\omega}_k t) \end{bmatrix}
(24)

where

ωk=110,0002kd \boldsymbol{\omega}_k = \frac{1}{10,000^{\frac{2k}{d}}}
(25)

Sources:

  • Transformer Architecture: The Positional Encoding - Amihossein - Blog
  • Position Information in Transformers: An Overview - Dufter et al (2021) - Arxiv - Paper
  • Rethinking Positional Encoding - Zheng et al (2021) - Arxiv Paper
  • Self-Attention with Functional Time Representation Learning - Xu et al (2019) - Arxiv - Paper
  • AI Coffee Break with Letita - Video 1 | 2
  • Attention is all you need. A Transformer Tutorial: 5. Positional Encoding - Video

Experiments

Initial Conditions

Training Time, Convergence

  • Random Intitialization
  • Feature-Wise Interpolation
  • PyInterp (2D)
  • Markovian Gaussian Process (MGP)
  • Optimal Interpolation (OI)

Iterative Schemes

Speed, Accuracy, PreTraining

  • Projection-Based
  • Gradient-Based
  • Fixed-Point Iteration
  • Anderson Acceleration
  • CNN + Gradient
  • LSTM

Priors

The impact on the priors on the learning procedure.

  • Deterministic
  • Probabilistic
Deterministic
  • ODE (Fixed)
  • PCA (Fixed)
  • ODE (Learnable)
  • PCA (Learnable)
  • UNet
Probabilistic
  • UNet + DropOut
  • Probabilistic UNet
Neural Fields
Neural Fields
Neural Fields
Literature Review