Bayesian Models & Neural Networks Tutorial Master List

A reconciled, exhaustive curriculum for Bayesian linear models, parametric regression, parametric classification, and Bayesian neural networks. The progression mirrors the textbook arc: Bayesian linear regression → basis functions → random / spectral features → shallow MLPs → deep BNNs. Each block lays out its own likelihood / constraint zoo and its own inference zoo so the curriculum can be entered at any depth.

Companion lists:

• Pure-GP machinery → ../gaussian_processes/TUTORIAL_MASTER_LIST.md
• Neural fields / INRs / NeRF → ../neural_fields/TUTORIAL_MASTER_LIST.md
• Filtering data assimilation → ../assimilation/docs/TUTORIAL_MASTER_LIST_FILTER.md
• Variational data assimilation → ../assimilation/docs/TUTORIAL_MASTER_LIST_VARIATIONAL.md
• Normalizing flows / gaussianization → ../gaussianization/TUTORIAL_MASTER_LIST.md

Cross-listed items (RFF, deep kernels, last-layer-Bayes, BLR, Laplace, VI guides) are flagged 🔁.

Legend — Source columns:

• G = exists in gaussx (docs/notebooks/<name>)
• P = exists in pyrox (docs/notebooks/<name>)
• R = exists in research_notebook (projects/<area>/notebooks/<path>)
• — = does not exist yet (gap)

Scope tag:

• 🧱 fundamental — small, library-API demo (gaussx/pyrox docs)
• 🔬 research — applied / dataset-driven (research_notebook/projects/bayesian_nns)
• 🌉 bridge — useful in either; cross-link
• 🔁 cross-listed — also in GP or neural-fields master list

Refs column: gh:<repo>#N = open GitHub issue (e.g., gh:pyrox#71) · dd:path = pyrox design_docs/pyrox/<path> · mc# = numbered model from examples/nn/regression_masterclass_eqx.md · xref:GP#X.Y = pointer into GP master list.

Curriculum at a glance¶

• Part A — Bayesian Regression
  • A.A — Bayesian linear regression
  • A.B — Fixed feature maps
  • A.C — Random Fourier features
  • A.D — Spectral basis layers (HSGP)
  • A.E — Bridges to GP
  • A.F — Likelihood zoo (regression heads)
  • A.G — Constrained & physics-informed losses
  • A.H — 9-model regression masterclass (pick-apart)
  • A.I — NN MAP baseline & library patterns
• Part B — Bayesian Classification
  • B.A — Bayesian logistic regression
  • B.B — Multinomial / softmax
  • B.C — Inference variants for classification
  • B.D — Feature-based & last-layer classifiers
  • B.E — Calibration for Bayesian classifiers
• Part C — NN ↔ GP Bridges
  • C.A — Theory (NNGP, NTK, ArcCosine)
  • C.B — Deep kernels
  • C.C — Functional priors
  • C.D — Pathwise BNN sampling
  • C.E — Shared infrastructure
• Part D — Bayesian Inference for Neural Networks
  • D.I — Point estimates: MLE / MAP / regularisation-as-prior
  • D.II — Gaussian (Laplace) approximations
  • D.III — Variational inference
  • D.IV — Sampling-based inference
  • D.V — Last-layer & functional posteriors
  • D.VI — Stochastic / implicit Bayes
  • D.VII — Tempering, prior choice, diagnostics
• Part E — Ensembles
• Part F — Calibration, OOD, Active Learning
• Part G — Bayesian Neural Fields
• Part H — Applied Case Studies (research_notebook/projects/bayesian_nns)

Part A — Bayesian Regression¶

A.A — Bayesian linear regression¶

Key equations / models:

• BLR posterior: $\Sigma = (\Phi^\top R^{-1}\Phi + S_0^{-1})^{-1}$, $\mu = \Sigma\,\Phi^\top R^{-1}y$
• Precision (natural) form: $\Lambda = \Phi^\top R^{-1}\Phi + S_0^{-1}$, $\eta = \Phi^\top R^{-1} y + S_0^{-1}\mu_0$
• Sequential / online update via Sherman–Morrison: rank-1 covariance update per new observation
• Predictive: $p(y_*\mid x_*, \mathcal{D}) = \mathcal{N}(\phi(x_*)^\top\mu,\, \phi(x_*)^\top\Sigma\phi(x_*) + \sigma^2)$

A.B — Fixed feature maps¶

Key equations / models:

• Generic linear model: $f(x) = \phi(x)^\top w$ with fixed $\phi:\mathbb{R}^d\to\mathbb{R}^D$
• Polynomial, Vandermonde, Chebyshev, Fourier, wavelet, Gaussian-bump bases

A.C — Random Fourier features¶

Key equations / models:

• Rahimi–Recht: $\phi(x) = \sqrt{2/D}\cos(\omega^\top x + b)$, $\omega\sim S(\omega)$, $k(x,x')\approx \phi(x)^\top\phi(x')$
• SSGP: BLR in RFF space, $O(D^2 N)$
• VSSGP: variational posterior over frequencies ω
• ORF: $\omega_i$ on the sphere → variance reduction

A.D — Spectral basis layers (HSGP)¶

Key equations / models:

• HSGP (Solin–Särkkä): $k(x,x')\approx \sum_{j=1}^M S(\sqrt{\lambda_j})\,\phi_j(x)\phi_j(x')$, $(\lambda_j,\phi_j)$ Laplacian eigenpairs
• Deterministic basis (vs random RFF) → diagonal $K_{uu}$, $O(NM + M^3)$

A.E — Bridges to GP¶

Key equations / models:

• Whitened SVGP-as-BLR view: inducing variables $u = L_{mm}\tilde u$ → BLR on $\tilde u$
• Equivalence map: kernel ridge ↔ MAP-BLR ↔ exact GP posterior mean (with appropriate features)

A.F — Likelihood zoo (regression heads)¶

Key equations / models:

• Gaussian: $y = f(x) + \epsilon$, $\epsilon\sim\mathcal{N}(0,\sigma^2)$
• Student-t: heavy-tailed, ν controls robustness
• Laplace: $|y-f|$ noise → L1 / Huber-like
• Heteroscedastic: $\sigma(x)$ predicted alongside $\mu(x)$
• Mixture density: $p(y\mid x) = \sum_k \pi_k(x)\mathcal{N}(y; \mu_k(x), \sigma_k(x))$
• Quantile / pinball: $\rho_\tau(u) = u(\tau - \mathbb{1}\{u<0\})$
• Censored / Tobit / survival: likelihood truncated / right-censored
• Log-Gaussian Cox Process: $\lambda(x) = \exp(f(x))$, Poisson observations
• Warped GP / NF-warped: $g(y) = f(x)$, $g$ monotone bijection (Box–Cox, NF)

A.G — Constrained & physics-informed losses¶

Key equations / models:

• Positivity / monotone / convex output via reparameterization (softplus, cumulative)
• Equality constraint via augmented Lagrangian: $\mathcal{L}(\theta,\lambda) + \tfrac{\rho}{2}\|c(\theta)\|^2 + \lambda^\top c(\theta)$
• PDE residual (PINN): $\mathcal{L} = \mathcal{L}_\text{data} + \alpha\|\mathcal{N}[u_\theta](x_i)\|^2$ on collocation points
• Boundary / initial penalty: extra weighted term on $\partial\Omega$
• Conservation / symmetry: divergence-free reparameterization, equivariant layers
• Smoothness / TV regularisation: $\|\nabla u\|_2$, $\|\nabla u\|_1$
• KL annealing / β-VAE: trade reconstruction vs KL

A.H — 9-model regression masterclass (pick-apart)¶

The pyrox examples/nn/regression_masterclass_eqx.md (~927 lines) is a single monolithic notebook. We break it into nine standalone tutorials, each running on the same dataset so a learner can ablate one block at a time.

Key equations / models: see Models 1–9 below — each row hyperlinks to the corresponding row in A.A–A.C / D.

A.I — NN MAP baseline & library patterns¶

Part B — Bayesian Classification¶

B.A — Bayesian logistic regression¶

Key equations / models:

• Binary likelihood: $y_i\sim\mathrm{Bernoulli}(\sigma(\phi(x_i)^\top w))$
• Probit alternative: $y_i\sim\mathrm{Bernoulli}(\Phi(\phi(x_i)^\top w))$
• Pólya–Gamma augmentation (Polson et al. 2013): $\sigma(\eta)^y(1-\sigma(\eta))^{1-y} = 2^{-1}\exp(\kappa\eta)\int_0^\infty\exp(-\omega\eta^2/2)p(\omega)\,d\omega$
• Augmented model → conjugate Gaussian update on $w \mid \omega$

B.B — Multinomial / softmax classification¶

Key equations / models:

• Multinomial: $y\sim\mathrm{Cat}(\mathrm{softmax}(W^\top\phi(x)))$
• One-vs-rest / multinomial-probit alternatives
• Augmentation schemes (Pólya–Gamma stick-breaking, Albert–Chib)

B.C — Inference variants for classification¶

B.D — Feature-based & last-layer classifiers¶

B.E — Calibration for Bayesian classifiers¶

Part C — NN ↔ GP Bridges¶

C.A — Theory¶

Key equations / models:

• NNGP (Lee et al. 2018): infinite-width MLP with i.i.d. priors → GP with recursive kernel $K^{(\ell)} = T_\sigma(K^{(\ell-1)})$
• NTK (Jacot et al. 2018): $\Theta(x,x') = \mathbb{E}\langle\partial_\theta f(x),\partial_\theta f(x')\rangle$, frozen in the infinite-width limit
• ArcCosine (Cho & Saul 2009): $k_n(x,x') = \tfrac{1}{\pi}\|x\|^n\|x'\|^n J_n(\theta)$

C.B — Deep kernels¶

C.C — Functional priors¶

C.D — Pathwise BNN sampling¶

C.E — Shared infrastructure¶

Part D — Bayesian Inference for Neural Networks¶

Reorganised around the kind of approximation rather than the layer flavour. Layer-flavour (Edward2, Conv/RNN/Attn) lives in D.VI/D.VIII.

D.I — Point estimates: MLE / MAP / regularisation-as-prior¶

Key equations / models:

• MLE: $\hat\theta = \arg\max_\theta \prod_i p(y_i\mid x_i,\theta)$
• MAP: $\hat\theta = \arg\max_\theta p(\theta)\prod_i p(y_i\mid x_i,\theta)$
• L2 ridge ↔ Gaussian prior; L1 lasso ↔ Laplace prior; elastic-net = sum
• Weight decay / spectral / Jacobian regularisation as implicit priors

D.II — Gaussian (Laplace) approximations¶

Key equations / models:

• Laplace: $q(\theta) = \mathcal{N}(\hat\theta, -H^{-1})$, $H = \nabla^2\log p(\theta\mid\mathcal{D})$
• GGN: $H \approx J^\top R J$ (drops 2nd-order); KFAC: block-Kronecker GGN
• Diagonal Laplace: $\mathrm{diag}(H)$ via Hutchinson
• Linearised Laplace: predict via $f_\theta(x)\approx f_{\hat\theta}(x) + J_{\hat\theta}(x)(\theta-\hat\theta)$
• SWAG: low-rank + diagonal Gaussian over SGD iterates
• Moment-matching / unscented predictive

D.III — Variational inference¶

Key equations / models:

• ELBO: $\log p(y)\geq\mathbb{E}_q[\log p(y,\theta)] - \mathbb{E}_q[\log q(\theta)]$
• Variational families: delta · mean-field diagonal · low-rank ($S = VV^\top + \mathrm{diag}$) · full-rank Cholesky · normalising flow · whitened
• Natural gradient: $\tilde\nabla\mathcal{L} = F^{-1}\nabla\mathcal{L}$
• CVI sites (Khan & Lin 2017); reparameterisation, local-reparam, flipout

D.IV — Sampling-based inference¶

D.V — Last-layer & functional posteriors¶

D.VI — Stochastic / implicit Bayes¶

D.VII — Tempering, prior choice, diagnostics¶

D.VIII — Distance-aware uncertainty¶

Part E — Ensembles¶

E.A — Vanilla ensembles¶

E.B — Diversity strategies¶

E.C — Comparison¶

Part F — Calibration, OOD, Active Learning¶

Part G — Bayesian Neural Fields¶

Core (deterministic) neural-fields content lives in ../neural_fields/TUTORIAL_MASTER_LIST.md. This section is the Bayesian layer on top — point estimates and uncertainty for INRs.

Part H — Applied Case Studies (research_notebook/projects/bayesian_nns)¶

H.A — Bayesian benchmarks¶

H.B — Emulators & PDEs¶

H.C — Image / signal regression¶

H.D — Capstone progressions¶

Cross-list summary (items shared with GP list)¶

Proposed final homes¶

• gaussx/docs/notebooks/ → A.A (BLR primitives), A.E (whitened SVGP / BLR view), D.II primitives (D.8 Hutchinson, D.13 moment matching), D.15 nat-grad VI
• pyrox/docs/notebooks/ → A.B, A.C, A.D, A.F, A.G, A.H, A.I, B.A–B.E, C.* shared infra, D.IV–D.VI library demos, E.A–E.B
• research_notebook/projects/bayesian_nns/notebooks/ → all of D.III–D.IV applied, D.V, D.VII, E.C, F, G, H

In-scope vs aspirational¶

• In scope today (have library support in pyrox/gaussx): A.2, A.3, A.7, A.8, A.9, A.16, A.44, B.13, C.4, C.5, D.4–D.8, D.15, E.2, E.3, H.2
• In scope with planned features (open issues / .plans/): A.1, A.4–A.6, A.10–A.15, A.17–A.25, A.32, A.33–A.45, B.1–B.12, B.14–B.17, C.3, C.9, D.28, D.31–D.41, D.48, E.4, G.5, G.6, H.3, H.9, H.10
• Aspirational (need new infra or genuine research work): A.26–A.31, B.18–B.20, C.1, C.2, C.6, C.7, C.8, D.1–D.3, D.9–D.14 (applied), D.16–D.20, D.21–D.27 (applied), D.29, D.30, D.42–D.47, D.49, E.1, E.5–E.7, F.*, G.1–G.4, H.1, H.4–H.8