Gaussianization Tutorial Master List

A reconciled, exhaustive curriculum spanning what currently exists in rbig, gauss_flows, and research_notebook/projects/gaussianization, plus gaps surfaced from the package APIs, open issues, and design docs. Goal: the most complete Gaussianization tutorial sequence we could ship.

Companion to ../gaussian_processes/TUTORIAL_MASTER_LIST.md. Cross-listed items (spatial-extremes margins, spatiotemporal applications, Kalman / filtering) are flagged 🔁 and tagged with xref:GP#X.Y.

Legend — Source columns:

• B = exists in rbig (docs/notebooks/<name>)
• F = exists in gauss_flows (docs/notebooks/<name>)
• K = exists in research_notebook gauss_keras (projects/gaussianization/notebooks/<name>)
• R = exists in research_notebook elsewhere
• — = does not exist yet (gap)

Scope tag: 🧱 fundamental · 🔬 research · 🌉 bridge · 🔁 cross-listed

Refs column: gh:<repo>#N = open GitHub issue · dd:path = design-doc path · api:foo = exported symbol · xref:GP#X.Y = cross-ref to GP master list.

Framing: Gaussianization is a transformation to a target Gaussian, so this curriculum is organized along three axes: (1) what we map with — 1D marginals, rotations, couplings, ODEs, surjections; (2) how we fit it — iterative / greedy vs. end-to-end NLL; (3) what we do with it — sampling, density, IT measures, inverse problems, filtering, fair learning. Normalizing-flow cousins (MAF / IAF / NSF / Glow / FFJORD) are treated as bridges, not first-class entries, except where they’re the canonical implementation of a Gaussianization idea.

Curriculum at a glance¶

A bird’s-eye view of the parts and their subparts. Skim this first to orient; the detailed per-tutorial tables live below.

• Part 0 — Foundations
  • 0.A — Change of variables & log-determinant
  • 0.B — Why standard Gaussian as target
  • 0.C — Density destructors
  • 0.D — Numerical mechanics
  • 0.E — Diagnostics
• Part 1 — 1D Marginal Transforms
  • 1.A — Empirical CDF & histograms
  • 1.B — KDE / Gaussian-mixture CDFs
  • 1.C — Monotone-spline CDFs
  • 1.D — Mixture-CDF as a learnable bijector
  • 1.E — Inversion strategies
• Part 2 — Rotations & Orthogonal Mixers
  • 2.A — Linear-rotation zoo (PCA / ICA / random / Picard)
  • 2.B — Householder products & trainable orthogonals
  • 2.C — Fixed orthogonal & PCA warm starts
  • 2.D — Invertible 1×1 conv (LU parameterization)
  • 2.E — ActNorm & per-channel affine
• Part 3 — Iterative Gaussianization (RBIG)
  • 3.A — Canonical RBIG loop
  • 3.B — Convergence & stopping criteria
  • 3.C — Rotation-choice studies
  • 3.D — RBIG as warm-start for parametric flows
  • 3.E — Boundary issues & support extension
• Part 4 — Parametric Gaussianization Flows
  • 4.A — NLL training of stacked blocks
  • 4.B — Diagonal vs. coupling marginal flow
  • 4.C — Factory walkthroughs
  • 4.D — Layer-wise inspection
• Part 5 — Coupling-based Gaussianization
  • 5.A — The coupling pattern
  • 5.B — Bijector menu for coupling
  • 5.C — Conditioner architectures (headline)
  • 5.D — Mask design
  • 5.E — Coupling ↔ diagonal equivalence
  • 5.F — Depth, residual coupling, stability
• Part 6 — Continuous-time Gaussianization (bridge)
  • 6.A — FFJORD
  • 6.B — Hutchinson trace estimator
  • 6.C — Matrix-exponential / linear neural flows
  • 6.D — Latent ODEs
• Part 7 — Conditional Gaussianization
  • 7.A — Conditioner zoo for marginals & couplings
  • 7.B — Conditional density estimation
  • 7.C — Three-pattern conditional flow
  • 7.D — Conditioning for inverse problems
• Part 8 — SurVAE: Surjections & Stochastic Transforms
  • 8.A — Bijection / surjection / stochastic taxonomy
  • 8.B — Slicing & augmentation surjections
  • 8.C — Stochastic transforms & ELBO
• Part 9 — Relaxed-Bijectivity & Non-Invertible Flows
  • 9.A — Injective / lossy Gaussianization
  • 9.B — Augmented / lifted flows
  • 9.C — Continuously-indexed flows (CIF)
  • 9.D — Stochastic normalizing flows
  • 9.E — Diffusion as continuous stochastic Gaussianization
  • 9.F — Residual / implicit flows
  • 9.G — One-shot / Trumpet-style Gaussianizers
  • 9.H — When non-invertibility helps
• Part 10 — Non-Euclidean Gaussianization
  • 10.A — Circle / torus
  • 10.B — Sphere
  • 10.C — Lie groups / Riemannian manifolds
• Part 11 — Time-Series Gaussianization
  • 11.A — Per-timestep marginal Gaussianization
  • 11.B — Autoregressive flows for sequences
  • 11.C — Conditioning on past context
  • 11.D — AR(p) in Gaussianized state
  • 11.E — Latent ODEs for irregular series
  • 11.F — Long-range / hierarchical temporal couplings
  • 11.G — Multiscale temporal flows
  • 11.H — Time-series anomaly & changepoint detection
• Part 12 — Spatial / Image Gaussianization
  • 12.A — Multiscale Squeeze / unsqueeze
  • 12.B — Invertible 1×1 conv & Haar wavelet
  • 12.C — Patch-based image flows
  • 12.D — Equivariant flows (rotation / translation)
  • 12.E — Spatial random-field Gaussianization (GRF prior)
  • 12.F — Image-rotation diagnostics
  • 12.G — Glow end-to-end
• Part 13 — Spatiotemporal Fields & Videos
  • 13.A — Separable space×time coupling
  • 13.B — Frame-conditioned video flows
  • 13.C — Spatiotemporal RBIG (lat × lon × time)
  • 13.D — Latent ODEs for field dynamics
  • 13.E — Equivariant spatiotemporal flows
  • 13.F — Climate-field scale-up
  • 13.G — Multiscale spatiotemporal
• Part 14 — Information-Theoretic Estimation
  • 14.A — Entropy & negentropy from Gaussianized residual
  • 14.B — Mutual information & total correlation
  • 14.C — KL between empirical distributions
  • 14.D — Dependence measures
  • 14.E — Real-data IT pipelines
  • 14.F — Bias-variance & sample complexity
• Part 15 — Fair Learning with Frozen Gaussianization Flows 🔬
  • 15.A — Frozen flow as differentiable independence loss
  • 15.B — G-XCOV vs G-HSIC vs CKA
  • 15.C — Pretrain + freeze workflow
  • 15.D — Synthetic fairness sweeps & Pareto curves
  • 15.E — Adult census case study
  • 15.F — Drop-in with FairModelWrapper
  • 15.G — Open research directions
• Part 16 — Plug-and-Play Priors with Gaussianization
  • 16.A — PnP recap (denoiser-as-prior)
  • 16.B — Closed-form prox in Gaussianized latent space
  • 16.C — HQS / ADMM with Gaussianization
  • 16.D — Patch-based PnP for images
  • 16.E — Linear inverse problems (deblur / SR / inpaint / CS)
  • 16.F — Comparison with score / diffusion PnP
• Part 17 — Filtering & Data Assimilation
  • 17.A — Normalizing Kalman filter (closed-form)
  • 17.B — State vs observation vs QoI Gaussianization
  • 17.C — Gaussianized ensemble Kalman filter
  • 17.D — Non-Gaussian likelihoods in DA
  • 17.E — Sequential Bayesian updates in latent space
  • 17.F — Comparison with EKF / UKF / particle filter
• Part 18 — Geoscience Case Studies
  • 18.A — Quantile mapping / bias correction
  • 18.B — Climate-field anomaly detection
  • 18.C — Spatial extremes (GEV / Gumbel margins) 🔁
  • 18.D — Ocean SST / sea-level extremes
  • 18.E — Precipitation Gaussianization
  • 18.F — Wind / atmospheric tracers
  • 18.G — Satellite-image emulation
  • 18.H — Climate-data assimilation
• Part 19 — Probabilistic-Programming Integration
  • 19.A — FlowDist in NumPyro
  • 19.B — Flows as priors in BHMs
  • 19.C — Flows as guides in SVI
  • 19.D — pyrox integration patterns
• Part 20 — Metrics, Calibration, Diagnostics
  • 20.A — NLL / bits-per-dim
  • 20.B — QQ / PIT / coverage for flows
  • 20.C — Sample quality (FID-style / MMD)
  • 20.D — Roundtrip invertibility & numerical tolerance
  • 20.E — Cross-validation for IT estimators

Part 0 — Foundations¶

0.A — Change of variables & log-determinant¶

Key equations / models:

• Change of variables: $p_X(x) = p_Z(T(x))\,|\det J_T(x)|$, $T$ a $C^1$-diffeomorphism
• Log-density: $\log p_X(x) = \log p_Z(T(x)) + \log|\det J_T(x)|$
• Composition: $T = T_K \circ \cdots \circ T_1 \Rightarrow \log|\det J_T(x)| = \sum_{k=1}^{K} \log|\det J_{T_k}(x_{k-1})|$
• Forward (data → latent) vs. inverse (latent → data) parameterisation

0.B — Why standard Gaussian as target¶

Key equations / models:

• Max-entropy result: $\arg\max_{p}\,H(p)$ s.t. fixed first/second moment $= \mathcal{N}(\mu,\Sigma)$
• Separable: $\mathcal{N}(0,I) = \prod_i \mathcal{N}(0,1)$ → IT measures decompose per-coordinate
• Trivial sampler, trivial score $\nabla\log\mathcal{N}(z;0,I) = -z$, trivial prox

0.C — Density destructors¶

Key equations / models:

• Density destructor: invertible map $T: \mathbb{R}^d \to \mathbb{R}^d$ with $T_\#p_X = \mathcal{N}(0,I)$
• Gaussianization = whitening (rotation + scaling) composed with element-wise nonlinearity (CDF map), iterated to convergence
• Inverse generative direction: sample $z\sim\mathcal{N}(0,I)$, return $T^{-1}(z)$

0.D — Numerical mechanics¶

Key equations / models:

• Jitter on CDFs: clip to $[\epsilon, 1-\epsilon]$ before $\Phi^{-1}$ to avoid $\pm\infty$
• Mixed-precision log-det: accumulate in float64 even when forward is float32
• Stable $\log\Phi^{-1}$ tails via series / asymptotic expansions
• Invertibility check: $\|T^{-1}(T(x)) - x\|_\infty < \tau$

0.E — Diagnostics¶

Key equations / models:

• QQ-plot against $\mathcal{N}(0,1)$ per coordinate
• Sample skewness / excess kurtosis: $\gamma_1, \gamma_2$ → both 0 under Gaussian
• Negentropy $J(p) = H(\mathcal{N}(0,\Sigma_p)) - H(p) \geq 0$
• Multivariate KS, Henze–Zirkler

Part 1 — 1D Marginal Transforms¶

The atomic operation of Gaussianization: turn each coordinate’s distribution into a standard Gaussian via $z_i = \Phi^{-1}(F_i(x_i))$ for some monotone CDF estimator $F_i$.

1.A — Empirical CDF & histograms¶

Key equations / models:

• Empirical CDF: $\hat F_n(x) = \tfrac{1}{n}\sum_{i=1}^n \mathbf{1}\{x_i\le x\}$
• Histogram-CDF: piecewise linear interpolation of bin counts
• Glivenko–Cantelli: $\sup_x|\hat F_n(x) - F(x)| \to 0$ a.s.

1.B — KDE / Gaussian-mixture CDFs¶

Key equations / models:

• KDE: $\hat f_h(x) = \tfrac{1}{nh}\sum_i K\!\big(\tfrac{x-x_i}{h}\big)$
• Mixture-of-Gaussians CDF: $F(x) = \sum_k \pi_k\,\Phi(x;\mu_k,\sigma_k^2)$
• Bandwidth: Silverman, ISJ, cross-validated

1.C — Monotone-spline CDFs¶

Key equations / models:

• Monotone cubic Hermite (Fritsch–Carlson): piecewise cubic with positive slopes
• Rational-quadratic spline (RQS, Durkan 2019): $F(x) = \frac{\alpha y^2 + \beta y + \gamma}{a y^2 + b y + c}$, exact inverse
• Log-det = sum of log derivatives at knot intervals

1.D — Mixture-CDF as a learnable bijector¶

Key equations / models:

• Forward: $u = F_\theta(x) = \sum_k \pi_k\,\Phi(x;\mu_k,\sigma_k)$, $z = \Phi^{-1}(u)$
• Log-det: $\log f_\theta(x) - \log\phi(z)$
• Inverse: $z\to u = \Phi(z)$, then $x = F_\theta^{-1}(u)$ via bisection

1.E — Inversion strategies¶

Key equations / models:

• Bisection: $O(\log(1/\epsilon))$ iterations, derivative-free, robust
• Newton: quadratic convergence near root, needs $F'$
• Hybrid: bisection bracket → Newton refine
• Vectorised inversion via jax.lax.while_loop / Keras ops

Part 2 — Rotations & Orthogonal Mixers¶

The “between-coordinate” half of Gaussianization: orthogonal mixers that redistribute information across dimensions so the next marginal pass has something to do.

2.A — Linear-rotation zoo¶

Key equations / models:

• PCA: $W = Q^\top$, $Q$ from eigendecomposition of $\mathrm{Cov}(X)$
• ICA: maximise non-Gaussianity of marginals (FastICA, Infomax)
• Random orthogonal: $Q\sim\mathrm{Haar}(O(d))$ via QR of Gaussian
• Picard: Riemannian L-BFGS on the orthogonal manifold

2.B — Householder products & trainable orthogonals¶

Key equations / models:

• Householder reflection: $H = I - 2vv^\top / \|v\|^2$
• Product: $Q = H_1 H_2 \cdots H_K$, exactly orthogonal by construction
• Cayley parameterisation: $Q = (I - A)(I + A)^{-1}$, $A$ skew-symmetric

2.C — Fixed orthogonal & PCA warm starts¶

Key equations / models:

• FixedOrtho from data PCA: freeze $Q$, only learn marginals downstream
• Warm-start: initialise Householder product to match a PCA $Q$ via QR decomposition

2.D — Invertible 1×1 conv (LU parameterization)¶

Key equations / models:

• 1×1 conv at each spatial location: $z_{ij} = W x_{ij}$, $W\in\mathbb{R}^{C\times C}$
• LU parameterisation: $W = PL(U + \mathrm{diag}(s))$, $\log|\det W| = \sum\log|s|$
• Connects 2.B (orthogonal mixers) to 12.B (image flows)

2.E — ActNorm & per-channel affine¶

Key equations / models:

• ActNorm: $z = s \odot x + b$, $s, b$ data-dependent initialised to give zero mean / unit variance at init
• Log-det = $\sum \log|s|$
• Why it matters: makes deep stacks trainable, classical pre-flow whitening

Part 3 — Iterative Gaussianization (RBIG)¶

The classical, non-parametric Gaussianization algorithm: alternate marginal CDFs (Part 1) with a rotation (Part 2) until the joint converges to $\mathcal{N}(0,I)$.

3.A — Canonical RBIG loop¶

Key equations / models:

• Iteration $k$: $x^{(k+1)} = Q_k\,T_k(x^{(k)})$, $T_k$ per-coordinate Gaussianization, $Q_k$ rotation
• Convergence (Laparra et al. 2011): $\mathrm{KL}(p_k \Vert \mathcal{N}(0,I)) \to 0$ for generic $Q$
• Forward stack composes log-dets additively

3.B — Convergence & stopping criteria¶

Key equations / models:

• Per-layer information reduction: $J(p_{k+1}) < J(p_k)$ (Laparra 2011 monotone decrease)
• Stop when $|J(p_k) - J(p_{k+1})| < \tau$ or fixed depth $K$
• Bias-corrected mutual information change across iterations

3.C — Rotation-choice studies¶

Key equations / models:

• Convergence rate as a function of $Q$ family
• PCA: variance-aligning, good when scales differ
• ICA: aligns to non-Gaussian directions, faster on heavy-tailed data
• Random: minimax-flavoured, no fitting cost

3.D — RBIG as warm-start for parametric flows → moved to Part 4¶

Warm-starting is a parametric-flow concern: a greedy RBIG fit only matters once there is a trainable flow to initialise. These two tutorials are therefore covered in Part 4 — Parametric Gaussianization Flows, alongside NLL training.

Key equations / models:

• Greedy fit each block to data, then jointly fine-tune via NLL
• initialize_flow_from_ig — sklearn PCA + GMM per block

3.E — Boundary issues & support extension¶

Key equations / models:

• Quantile clipping: $u \in [\epsilon, 1-\epsilon]$ to keep $\Phi^{-1}(u)$ finite
• Tail extrapolation via GPD / Gumbel
• Support extension by mixture with uniform / Gaussian noise (dequantisation)

Part 4 — Parametric Gaussianization Flows¶

Stack the rotation + marginal blocks into a differentiable graph and train end-to-end with maximum likelihood.

4.A — NLL training of stacked blocks¶

Key equations / models:

• Loss: $\mathcal{L}(\theta) = -\tfrac{1}{n}\sum_i \big[\log p_Z(T_\theta(x_i)) + \log|\det J_{T_\theta}(x_i)|\big]$
• $p_Z = \mathcal{N}(0,I)$ so $-\log p_Z(z) = \tfrac{1}{2}\|z\|^2 + \tfrac{d}{2}\log 2\pi$
• Gradient through bisection inverse via implicit-function theorem

4.B — Diagonal vs. coupling marginal flow¶

Key equations / models:

• Diagonal: independent 1D CDF per coordinate, no cross-coordinate conditioning
• Coupling: bijector on one half conditioned on the other → expressive non-separability

4.C — Factory walkthroughs¶

Key equations / models:

• gaussianization_flow(...) — stacked rotation + diagonal mixture-CDF marginal
• coupling_gaussianization_flow(...) — stacked rotation + spline coupling

4.D — Layer-wise inspection¶

Key equations / models:

• forward_with_intermediates(x) returns the trajectory $(x, T_1(x), T_2 T_1(x), \dots)$
• Diagnose where Gaussianisation is “stuck” via per-layer QQ / skew / negentropy

Part 5 — Coupling-based Gaussianization¶

Coupling is the expressive engine of modern Gaussianization: split coordinates with a mask, apply a per-coordinate bijector whose parameters are predicted by a conditioner from the unchanged half.

5.A — The coupling pattern¶

Key equations / models:

• Split: $x = (x_A, x_B)$ via mask $m$
• Forward: $z_A = x_A$, $z_B = T_{\theta(x_A)}(x_B)$
• Log-det: $\sum_i \log\partial_{x_{B,i}} T_{\theta(x_A)}(x_{B,i})$ — free because Jacobian is triangular
• Inverse: $x_A = z_A$, $x_B = T^{-1}_{\theta(z_A)}(z_B)$ — no neural-net inverse needed

5.B — Bijector menu for coupling¶

Key equations / models:

• Affine: $T(x;s,b) = s\odot x + b$, $\log|\det| = \sum\log|s|$
• Mixture-CDF: $T(x_B;\theta) = \Phi^{-1}\!\big(\sum_k \pi_k(\theta)\,\Phi(x_B;\mu_k(\theta),\sigma_k(\theta))\big)$
• Deep sigmoid: cascaded σ-shifts, expressive monotone
• Rational-quadratic spline (NSF, Durkan 2019)
• Residual coupling: $T(x) = x + g_\theta(x)$ with Lipschitz $g$ (preview of 9.E)

5.C — Conditioner architectures (headline)¶

The conditioner is the expressive part — the bijector is just a triangular wrapper that makes log-det free. Every structured-data part (11 / 12 / 13) revisits this menu and picks the modality-appropriate architecture.

Key equations / models:

• Conditioner $\theta = c_\phi(x_A)$, $c_\phi : \mathbb{R}^{|A|} \to \mathbb{R}^{n_\text{params}(B)}$
• MLP: $h_\ell = \mathrm{act}(W_\ell h_{\ell-1} + b_\ell)$
• Shared MLP: one trunk, separate per-coordinate heads (parameter-efficient)
• CNN-conditioner (image): conv stack preserving spatial dims (12.C)
• RNN / Transformer / Mamba conditioner (sequence): causal attention or recurrence (11.B)
• GNN-conditioner: message-passing on graph-structured inputs
• Equivariant conditioner: enforce symmetry of the data domain
• Hypernetwork conditioner: $c_\phi(x_A) = \mathrm{hypernet}(z) \cdot \mathrm{net}(x_A)$

5.D — Mask design¶

Key equations / models:

• Checkerboard / striped: spatial alternation
• Channel-wise (split halves): standard RealNVP
• Learned mask: differentiable via Gumbel-softmax
• Stacking: alternate masks so every coordinate is updated and conditions

5.E — Coupling ↔ diagonal equivalence¶

Key equations / models:

• Zero-kernel init: if the conditioner outputs constants, coupling collapses to a diagonal-marginal flow
• Numerical-equivalence proof: matched outputs at init within 1e-6
• Training “breaks the equivalence” — diagnostic for whether the conditioner is doing anything

5.F — Depth, residual coupling, stability¶

Key equations / models:

• Stacked coupling: $T = T_K \circ \cdots \circ T_1$ with alternating masks
• Residual: $T(x) = x + g_\theta(x)$ with Lipschitz constraint
• Gradient pathology at depth — ActNorm pre-conditioning helps (2.E)

Part 6 — Continuous-time Gaussianization (bridge)¶

A continuous-time bijector is a flow ODE $\dot x = v_\theta(x, t)$ whose pushforward at $t=T$ matches $\mathcal{N}(0,I)$. This is the “infinite-depth coupling” limit and the bridge to diffusion models (9.E).

6.A — FFJORD¶

Key equations / models:

• Instantaneous CoV: $\partial_t \log p_t(x) = -\mathrm{tr}(\nabla_x v_\theta(x, t))$
• Log-det as a line integral: $\log|\det J_T(x)| = -\int_0^T \mathrm{tr}(\nabla_x v_\theta(x_t, t))\,dt$
• Free-form $v_\theta$ — no architectural invertibility constraint

6.B — Hutchinson trace estimator¶

Key equations / models:

• Stochastic trace: $\mathrm{tr}(A) \approx \mathbb{E}_{z\sim\mathrm{Rademacher}}[z^\top A z]$
• Replaces $O(d)$ Jacobian-vector products with $O(1)$ per training step
• Variance: $\mathrm{Var}(z^\top A z) = 2\|A\|_F^2 - 2\sum_i A_{ii}^2$

6.C — Matrix-exponential / linear neural flows¶

Key equations / models:

• Linear neural flow: $\dot x = Ax$ → $x(T) = \exp(AT)\,x(0)$
• Closed-form log-det: $\log|\det \exp(AT)| = T\,\mathrm{tr}(A)$
• Useful as a building block for non-linear FFJORD

6.D — Latent ODEs¶

Key equations / models:

• Encode $\to$ latent ODE in $z$-space $\to$ decode
• Closed-form Gaussianization on the latent state $z(0)$ if encoder is invertible

Part 7 — Conditional Gaussianization¶

Make every parameter of the flow depend on a context $y$ — gives a tractable conditional density $p(x \mid y)$.

7.A — Conditioner zoo for marginals & couplings¶

Key equations / models:

• Conditional marginal: $z_i = \Phi^{-1}\!\big(F_{\theta(y)}(x_i)\big)$
• Conditional coupling: bijector parameters depend on both $x_A$ and $y$
• Conditional rotation: $Q(y)$ from a conditioner producing skew-symmetric or Householder vectors

7.B — Conditional density estimation¶

Key equations / models:

• $\log p(x\mid y) = \log p_Z(T_\theta(x\mid y)) + \log|\det J_{T_\theta}(x\mid y)|$
• ELBO if conditioner produces variational parameters

7.C — Three-pattern conditional flow¶

7.D — Conditioning for inverse problems¶

Key equations / models:

• Conditional flow as posterior $p(x\mid y)$ for inverse problems $y = Ax + \eta$
• Amortised inference — train once, sample posterior for any $y$ in one pass

Part 8 — SurVAE: Surjections & Stochastic Transforms¶

Generalise bijections to surjections (one direction is many-to-one) and stochastic transforms (one direction adds randomness) while keeping a tractable density / ELBO. Companion proof in survae_flows_proof.md.

8.A — Bijection / surjection / stochastic taxonomy¶

Key equations / models:

• Bijection: $|\det J|$ in both directions
• Surjection: forward deterministic, inverse stochastic → ELBO term $\log q(x\mid z)$
• Stochastic: both directions stochastic

8.B — Slicing & augmentation surjections¶

Key equations / models:

• Slicing: $z = x_A$, drop $x_B$ — useful for dimension reduction
• Augmentation: $z = (x, u)$ for auxiliary $u\sim q(u\mid x)$
• Dequantisation: integer $x \to x + u$, $u\sim\mathrm{Uniform}[0,1)$

8.C — Stochastic transforms & ELBO¶

Key equations / models:

• ELBO: $\log p(x) \geq \mathbb{E}_{q(z\mid x)}[\log p(z) + \log p(x\mid z)] + H[q(z\mid x)]$
• VAE as a single-step stochastic flow
• Connect to 9.D (stochastic NF) and 9.E (diffusion)

Part 9 — Relaxed-Bijectivity & Non-Invertible Flows¶

Drop strict invertibility for expressiveness or generality. Each sub-part keeps the density / ELBO / score machinery from going opaque.

9.A — Injective / lossy Gaussianization¶

Key equations / models:

• Data assumed to live on an $m$-dim submanifold of $\mathbb{R}^d$ with $m\leq d$
• Injective decoder $g:\mathbb{R}^m\to\mathbb{R}^d$ — image is the data manifold; one-to-one with no information loss
• Lossy / surjective encoder $E:\mathbb{R}^d\to\mathbb{R}^m$ — left-inverse of $g$ on-manifold, many-to-one off-manifold
• On-manifold density via $|\det J_g^\top J_g|^{1/2}$ (Brehmer & Cranmer 2020, M-Flow)

9.B — Augmented / lifted flows¶

Key equations / models:

• ANF (Huang 2020) / VFlow (Chen 2020): lift $x\to(x,u)$ with $u\sim q$, Gaussianize the joint
• Recover marginal via stochastic inverse

9.C — Continuously-indexed flows (CIF)¶

Key equations / models:

• Index the bijector by a latent $u$: $T_u(x)$, marginalise / ELBO over $u$
• Cornish et al. 2020 — relaxes topological constraints of bijectors

9.D — Stochastic normalizing flows¶

Key equations / models:

• Interleave deterministic bijectors with MCMC / Langevin kernels
• Wu, Köhler & Noé 2020 — tractable importance-weighted estimator

9.E — Diffusion as continuous stochastic Gaussianization¶

The forward diffusion process is a Gaussianization: it transports any data distribution to $\mathcal{N}(0,I)$ along a continuous noise schedule. The probability-flow ODE is its deterministic invertible counterpart and sits in the same family as Part 6.

Key equations / models:

• Forward VP SDE: $dx_t = -\tfrac{1}{2}\beta(t)x_t\,dt + \sqrt{\beta(t)}\,dW_t$, marginals $x_t\sim\mathcal{N}(\alpha(t)x_0,\sigma^2(t)I)$, $x_T\approx\mathcal{N}(0,I)$
• Reverse SDE: $dx_t = \big[-\tfrac{1}{2}\beta(t)x_t - \beta(t)\nabla\log p_t(x_t)\big]\,dt + \sqrt{\beta(t)}\,d\bar W_t$
• Probability-flow ODE (deterministic, invertible — a continuous Gaussianization): $\dot x_t = -\tfrac{1}{2}\beta(t)x_t - \tfrac{1}{2}\beta(t)\nabla\log p_t(x_t)$
• Score matching: $\mathbb{E}_{t,x_t}\big[\lambda(t)\,\|s_\theta(x_t,t) - \nabla\log p_t(x_t)\|^2\big]$
• Flow-matching / rectified-flow loss: $\mathbb{E}_{t,x_0,x_1}\big[\|v_\theta(x_t,t) - (x_1-x_0)\|^2\big]$, $x_0\sim p_\text{data}$, $x_1\sim\mathcal{N}(0,I)$
• $\sigma\to 0$ limit: recovers classical deterministic Gaussianization

9.F — Residual / implicit flows¶

Key equations / models:

• Residual: $T(x) = x + g_\theta(x)$ with $\mathrm{Lip}(g)<1$ (Behrmann 2019, Chen 2019)
• Inverse by Banach fixed-point iteration
• Log-det via Hutchinson + power series of $\log\det(I+J_g)$

9.G — One-shot / Trumpet-style Gaussianizers¶

Key equations / models:

• Single feed-forward encoder $E_\theta : x \to z$, trained with NLL or matching loss
• Trades NLL exactness for amortisation speed

9.H — When non-invertibility helps¶

Part 10 — Non-Euclidean Gaussianization¶

Push the target back to $\mathcal{N}(0,I)$ when the data lives on a manifold (circle, torus, sphere, Lie group).

10.A — Circle / torus¶

Key equations / models:

• von Mises CDF as periodic Gaussianization in 1D
• Torus = $(S^1)^d$; per-axis circular flow + orthogonal mixer

10.B — Sphere¶

Key equations / models:

• Lambert / stereographic chart for $S^2$ → Euclidean Gaussianization in chart
• Equivariant constructions for $SO(3)$

10.C — Lie groups & Riemannian manifolds¶

Key equations / models:

• Exponential map: $\exp_g : T_g M \to M$, Gaussianize on tangent space
• Riemannian flow models (Rezende 2020, Lou 2020)

Part 11 — Time-Series Gaussianization¶

Sequences have temporal structure: past conditions future. Choose conditioners accordingly (see 5.C and 11.B).

11.A — Per-timestep marginal Gaussianization¶

Key equations / models:

• Independent per-timestep CDF $F_t$ — works when distribution is stationary
• Sliding-window CDF for non-stationary series

11.B — Autoregressive flows for sequences¶

Key equations / models:

• Autoregressive factorisation: $p(x_{1:T}) = \prod_t p(x_t \mid x_{<t})$
• Conditional Gaussianization: $z_t = \Phi^{-1}\!\big(F_{\theta(x_{<t})}(x_t)\big)$
• MAF / IAF as the canonical NF cousin (bridge to 11.A)

11.C — Conditioning on past context¶

Key equations / models:

• Causal mask on Transformer-conditioner
• Stateful RNN conditioner with hidden $h_t$

11.D — AR(p) in Gaussianized state¶

Key equations / models:

• Gaussianize each $x_t \to z_t$ via stationary marginal flow
• Fit linear AR(p): $z_t = \sum_{k=1}^p \phi_k z_{t-k} + \epsilon_t$ in latent space
• Closed-form likelihood + forecasting

11.E — Latent ODEs for irregular series¶

11.F — Long-range / hierarchical temporal couplings¶

Key equations / models:

• Dilated coupling masks across time — exponentially-growing receptive field
• Hierarchical encoders (U-Net style) over the time axis

11.G — Multiscale temporal flows¶

Key equations / models:

• Dyadic Squeeze across time: $T \to T/2$ with channel doubling
• Wavelet temporal bijector (Haar in time)
• Multi-resolution AR flows

11.H — Time-series anomaly & changepoint detection¶

Key equations / models:

• Anomaly score: $-\log p(x_t \mid x_{<t})$ from a conditional flow
• Changepoint: drift in running-mean of $\|z_t\|^2$ in Gaussianized space

Part 12 — Spatial / Image Gaussianization¶

Images have spatial structure: locality + translation symmetry. Multiscale composition is the standard scaling pattern.

12.A — Multiscale Squeeze / unsqueeze¶

Key equations / models:

• Squeeze: $(H, W, C) \to (H/2, W/2, 4C)$ — locality-preserving rearrangement
• Multi-scale architecture: factor out half the channels at each scale, Gaussianize, continue
• Glow-style scale loop

12.B — Invertible 1×1 conv & Haar wavelet¶

Key equations / models:

• 1×1 conv (LU): per-pixel channel mixing (see 2.D)
• Haar wavelet: orthogonal multiresolution analysis (averages + differences)
• Both interpretable as Part 2 mixers applied across image structure

12.C — Patch-based image flows¶

Key equations / models:

• Train Gaussianization flow on $p\times p$ patches; tile over the image
• Use as a learned image prior (feeds Part 16 PnP)
• Stationarity assumption + receptive-field analysis

12.D — Equivariant flows¶

Key equations / models:

• $G$-equivariant flow: $T(g\cdot x) = g\cdot T(x)$ for $g\in G$
• Translation-equivariance via CNN couplings
• Rotation-equivariance via steerable filters

12.E — Spatial random-field Gaussianization (GRF prior)¶

Key equations / models:

• Treat a spatial field $f(s)$ on a grid as one observation; Gaussianize → Gaussian-process-like latent
• Feeds spatial-extremes margins (18.C) and DA (17)
• Bridges to xref:GP#4.B (Toeplitz / Kronecker GRFs)

12.F — Image-rotation diagnostics¶

Key equations / models:

• Rotation choices for image-level RBIG: PCA / random / patch-PCA / learned
• Visual inspection of recovered modes after each rotation

12.G — Glow end-to-end¶

Part 13 — Spatiotemporal Fields & Videos¶

Lat × lon × time tensors and videos. Inherits machinery from Parts 11 and 12.

13.A — Separable space×time coupling¶

Key equations / models:

• Factored bijector: alternate spatial-only and temporal-only coupling blocks
• Receptive field grows in both axes via stacking

13.B — Frame-conditioned video flows¶

Key equations / models:

• $p(x_{1:T}^\text{video}) = \prod_t p(x_t \mid x_{<t})$ with image-flow conditioned on past frames
• Conditioner is a temporal CNN / Transformer over frame features (5.C)

13.C — Spatiotemporal RBIG (lat × lon × time)¶

Key equations / models:

• Apply RBIG to flattened lat × lon × time tensors with structured rotations
• Per-axis vs joint rotation choices

13.D — Latent ODEs for field dynamics¶

13.E — Equivariant spatiotemporal flows¶

Key equations / models:

• Translation-in-space + translation-in-time equivariance
• Lifts to $SE(2)\times\mathbb{R}_t$ via steerable / equivariant conditioners (5.14)

13.F — Climate-field scale-up¶

Key equations / models:

• Sharded RBIG / coupling across patches × time-windows
• Streaming statistics for very large fields

13.G — Multiscale spatiotemporal¶

Key equations / models:

• Joint Squeeze in $(H, W, T)$ — dyadic in all three axes
• 3D Haar wavelets / lifted-wavelet bijectors
• Mixed-resolution video flows

Part 14 — Information-Theoretic Estimation¶

A killer downstream use of Gaussianization: once $T_\#p = \mathcal{N}(0,I)$, IT functionals decompose trivially.

14.A — Entropy & negentropy from Gaussianized residual¶

Key equations / models:

• Differential entropy: $H(p) = H(\mathcal{N}(0,\Sigma_p)) - J(p)$
• Negentropy: $J(p) \approx \sum_k \Delta J_k$ across RBIG iterations
• Closed-form Gaussian entropy: $\tfrac{1}{2}\log|2\pi e\,\Sigma|$

14.B — Mutual information & total correlation¶

Key equations / models:

• $I(X;Y) = H(X) + H(Y) - H(X,Y)$
• Total correlation: $TC(X) = \sum_i H(X_i) - H(X)$, computable directly from the RBIG residual

14.C — KL between empirical distributions¶

Key equations / models:

• Build $T_p$, $T_q$ from samples; estimate $\mathrm{KL}(p\Vert q) = \mathbb{E}_p[\log p - \log q]$ via change-of-variables

14.D — Dependence measures¶

14.E — Real-data IT pipelines¶

14.F — Bias-variance & sample complexity¶

Key equations / models:

• Asymptotic bias of plug-in IT estimators
• Jackknife / bootstrap for IT confidence intervals
• Effective-sample-size for RBIG-MI in high $d$

Part 15 — Fair Learning with Frozen Gaussianization Flows 🔬¶

Pretrain a flow on a dataset, freeze its weights, then use the Gaussianised representation as a differentiable independence loss inside any predictor. Active research — drops into fairkl.models.FairModelWrapper as a replacement for CKALoss.

15.A — Frozen flow as differentiable independence loss¶

Key equations / models:

• Pretrain $T_\theta$ s.t. $T_\theta(x)\sim\mathcal{N}(0,I)$, freeze weights
• Use $T_\theta$ to extract Gaussianised features. Covariance-based independence measures on $(T(x), q)$ become a tractable proxy for independence; the equivalence “zero covariance ⇔ independence” requires the joint to be Gaussian, which marginal Gaussianisation of $x$ alone does not guarantee (and breaks especially for categorical $q$). In practice the proxy works well when marginal-shape mismatch dominates the dependence signal.

15.B — G-XCOV vs G-HSIC vs CKA¶

Key equations / models:

• G-XCOV: cross-covariance in Gaussianised space $\|\mathrm{Cov}(T(x), q)\|_F^2$
• G-HSIC: kernel HSIC computed on $T(x)$ instead of $x$
• CKA: cosine-normalised cross-covariance (baseline)

15.C — Pretrain + freeze workflow¶

15.D — Synthetic fairness sweeps & Pareto curves¶

15.E — Adult census case study¶

15.F — Drop-in with FairModelWrapper¶

15.G — Open research directions¶

Part 16 — Plug-and-Play Priors with Gaussianization¶

The central pedagogical hook: the proximal operator of a Gaussianized prior is closed-form in latent space — $z \mapsto z/(1 + 1/\tau)$ for the standard Gaussian — so PnP-ADMM / HQS schemes with a Gaussianization prior have no inner solver.

16.A — PnP recap (denoiser-as-prior)¶

Key equations / models:

• Inverse problem: $\hat x = \arg\min_x \tfrac{1}{2}\|Ax - y\|^2 + \lambda R(x)$
• PnP: replace $\mathrm{prox}_{\lambda R}$ with a generic denoiser
• Convergence (Sun 2019, Ryu 2019) under non-expansive denoisers

16.B — Closed-form prox in Gaussianized latent space¶

Key equations / models:

• Gaussianized prior log-density: $\log p_R(x) = \log\mathcal{N}(T(x); 0, I) + \log|\det J_T(x)|$
• Score: $\nabla\log p_R(x) = -J_T(x)^\top T(x) + \nabla\log|\det J_T(x)|$
• Variable split: rewrite the inverse problem in latent coordinates $z = T(x)$ via HQS / ADMM so the regulariser becomes the standard-Gaussian negative log-prior $\tfrac{1}{2}\|z\|^2$
• Closed-form prox in latent coordinates: $\mathrm{prox}_{\|z\|^2/(2\tau)}(z) = z/(1 + 1/\tau)$
• Caveat: this is the prox with respect to the latent-space Euclidean metric, not the Euclidean prox of the induced data-space prior $-\log p_R(x)$ — those coincide only when $T$ is linear / isometric. With general $T$, the data-space prox $\arg\min_x\,\tfrac{1}{2\tau}\|x-x_0\|^2 - \log p_R(x)$ has no closed form, which is precisely why the latent-split formulation is the practical recipe (cf. Asim 2020, Whang 2021).
• Pull-back update: $x_+ = T^{-1}(z_+/(1+1/\tau))$ where $z_+ = T(x)$

16.C — HQS / ADMM with Gaussianization¶

Key equations / models:

• HQS: alternate $x$-update (data fit) and $z$-update (prox in $T$-space)
• ADMM: add dual variable $u$ for tighter coupling

16.D — Patch-based PnP for images¶

Key equations / models:

• Train Gaussianization flow on patches (12.C)
• Apply per-patch prox, aggregate via overlap-add

16.E — Linear inverse problems¶

16.F — Comparison with score / diffusion PnP¶

Key equations / models:

• Score-based PnP uses learned $\nabla\log p_t(x)$ at multiple noise levels
• Gaussianization-PnP uses one exact, deterministic, closed-form prox
• Trade-off table: cost, expressiveness, training pipeline

Part 17 — Filtering & Data Assimilation with Gaussianization¶

Gaussianize a non-Gaussian state / observation / QoI, run a closed-form Kalman recursion in latent space, then invert.

17.A — Normalizing Kalman filter (closed-form)¶

Key equations / models:

• Gaussianized state: $z_t = T(x_t)$, $T$ a learned bijector with $T(x_t) \sim \mathcal{N}(0, I)$ marginally
• Linear-Gaussian model in $z$-space: $z_{t+1} = A\,z_t + w_t$, $\tilde y_t = H\,z_t + v_t$
• Standard Kalman in $z$-space + invert: $\hat x_{t\mid t} = T^{-1}(\hat z_{t\mid t})$
• Posterior log-density: $\log p(x_t\mid y_{1:t}) = \log\mathcal{N}(T(x_t);\,\hat z_{t\mid t}, P_{t\mid t}) + \log|J_T(x_t)|$

17.B — State vs observation vs QoI Gaussianization¶

Key equations / models:

• State: $T_x: x \to z_x$, dynamics learned in $z_x$ space (most general, hardest)
• Observation: $T_y: y \to z_y$, observation operator linearised post-transform (cheapest)
• QoI: $T_q: Q(x) \to z_q$ applied to a derived quantity (post-hoc summary; trivially invertible)

17.C — Gaussianized ensemble Kalman filter¶

Key equations / models:

• EnKF analysis in $z$-space: $z^a = z^f + P_e^z H^\top(HP_e^zH^\top + R)^{-1}(\tilde y - Hz^f)$
• Sample covariance $P_e^z$ from ensemble of Gaussianised states
• Push back via $T^{-1}$ to obtain $x^a$ ensemble

17.D — Non-Gaussian likelihoods in DA¶

Key equations / models:

• Observation Gaussianization makes a Poisson / Bernoulli / heavy-tailed observation look Gaussian
• Closed-form update once $\tilde y_t = T_y(y_t)$ is Gaussian

17.E — Sequential Bayesian updates in latent space¶

17.F — Comparison with EKF / UKF / particle filter¶

Part 18 — Geoscience Case Studies¶

Applied stories. Each entry composes pieces from earlier parts; flagged 🔁 where it cross-references the GP master list.

18.A — Quantile mapping / bias correction¶

Key equations / models:

• Bias correction: $x_\text{cal}(t) = F_\text{obs}^{-1}(F_\text{model}(x_\text{model}(t)))$ — exactly 1D Gaussianization round-trip
• Per-season / per-region conditional quantile mapping