Skip to content
Source

GPs + Fourier Representations

Any content related to GPs and Fourier representations

Key Words

  • Random Fourier Features
  • Sparse Spectrum

Random Fourier Features

Random Features for Large-Scale Kernel Machines by Rahimi & Recht

-> Paper

Nystrรถm Method vs Random Fourier Features: A Theoretical and Empirical Comparison by Yang et al (2012)

-> Paper

The Geometry of Random Features by Choromanski (2012)

A paper showing how orthogonal random features might be better.

-> Paper

๐ŸŒ Reflections on Random Kitchen Sinks by Rahimi and Recht (12-2017)

An interesting blog from the authors about the origins of their idea.

๐ŸŒ Random Fourier Features (12-2019) by Gregory Gundersen

Excellent blog post going step-by-step of how to do Fourier features.

Fast-Food Approximations

Fastfood: Approximate Kernel Expansions in Loglinear Time by Viet Le et. al. (2014)

-> Paper

-> Video

-> Code

A la Carte - Learning Fast Kernels by Yang et. al. (2014)

-> Paper

Efficient Approximate Inference with Walsh-Hadamard Variational Inference by Rossi et. al. (2020)

-> Paper


Sparse Spectrum Gaussian Processes

These are essentially the analogue to the random fourier features for Gaussian processes.

Sparse Spectrum Gaussian Process Regression - Lรกzaro-Gredilla et. al. (2010)

The original algorithm for SSGP.

-> Paper

Code

๐Ÿ“ Numpy

๐Ÿ“ GPFlow

๐Ÿ“ GPyTorch

They call it a mixture of Deltas.


Variational

The SSGP algorithm had a tendency to overfit. So they added some additional parameters to account for the noise in the inputs making the marginal likelihood term intractable. They added variational methods to deal with the

Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs - Gal et. al. (2015)

"...proposed variational inference in a sparse spectrum model that is derived from a GP model." - Hensman et. al. (2018)

-> ๐Ÿ“ Theano

-> ๐Ÿ“ autograd

-> Yarin Gal's Stuff - website

Variational Fourier Features for Gaussian Processes - Hensman et al (2018)

"...our work aims to directly approximate the posterior of the true models using a variational representation." - Hensman et. al. (2018)

-> Paper

-> GPFlow

Know Your Boundaries: Constraining Gaussian Processes by Variational Harmonic Features by Solin & Kok (2019)

-> Paper

-> GPFlow


Uncertain Inputs

I've only seen one paper that attempts to extend this method to account for uncertain inputs.

Prediction under Uncertainty in Sparse Spectrum Gaussian Processes with Applications to Filtering and Control - Pan et. al. (2017)

This is the only paper I've seen that tries to extend this method

-> Paper


Other

Some other papers that are related but I haven't made the connection yet.

Deep Sigma Point Processes by Jankowiak et. al. (2020)

-> Paper

-> Code

Function-Space Distributions over Kernels - Benton et. al. (2019)
Spectral Methods in Gaussian Modelling - Requiema & Bruinsma (2019)