GPs + Fourier Representations¶
Any content related to GPs and Fourier representations
Key Words¶
- Random Fourier Features
- Sparse Spectrum
Random Fourier Features¶
Random Features for Large-Scale Kernel Machines by Rahimi & Recht
-> Paper
Nyström Method vs Random Fourier Features: A Theoretical and Empirical Comparison by Yang et al (2012)
-> Paper
The Geometry of Random Features by Choromanski (2012)
A paper showing how orthogonal random features might be better.
-> Paper
🌐 Reflections on Random Kitchen Sinks by Rahimi and Recht (12-2017)
An interesting blog from the authors about the origins of their idea.
🌐 Random Fourier Features (12-2019) by Gregory Gundersen
Excellent blog post going step-by-step of how to do Fourier features.
Fast-Food Approximations¶
Fastfood: Approximate Kernel Expansions in Loglinear Time by Viet Le et. al. (2014)
-> Paper
-> Video
-> Code
A la Carte - Learning Fast Kernels by Yang et. al. (2014)
-> Paper
Efficient Approximate Inference with Walsh-Hadamard Variational Inference by Rossi et. al. (2020)
-> Paper
Sparse Spectrum Gaussian Processes¶
These are essentially the analogue to the random fourier features for Gaussian processes.
Sparse Spectrum Gaussian Process Regression - Lázaro-Gredilla et. al. (2010)
The original algorithm for SSGP.
-> Paper
Code¶
📝 Numpy
📝 GPFlow
📝 GPyTorch
They call it a mixture of Deltas.
Variational¶
The SSGP algorithm had a tendency to overfit. So they added some additional parameters to account for the noise in the inputs making the marginal likelihood term intractable. They added variational methods to deal with the
Improving the Gaussian Process Sparse Spectrum Approximation by Representing Uncertainty in Frequency Inputs - Gal et. al. (2015)
"...proposed variational inference in a sparse spectrum model that is derived from a GP model." - Hensman et. al. (2018)
-> 📝 Theano
-> 📝 autograd
-> Yarin Gal's Stuff - website
Variational Fourier Features for Gaussian Processes - Hensman et al (2018)
"...our work aims to directly approximate the posterior of the true models using a variational representation." - Hensman et. al. (2018)
-> Paper
-> GPFlow
Know Your Boundaries: Constraining Gaussian Processes by Variational Harmonic Features by Solin & Kok (2019)
-> Paper
-> GPFlow
Uncertain Inputs¶
I've only seen one paper that attempts to extend this method to account for uncertain inputs.
Prediction under Uncertainty in Sparse Spectrum Gaussian Processes with Applications to Filtering and Control - Pan et. al. (2017)
This is the only paper I've seen that tries to extend this method
-> Paper
Other¶
Some other papers that are related but I haven't made the connection yet.