KL Divergence

Typical:

$$\text{KL}\left[ q(\mathbf x) || \mathcal{P}(\mathbf x)\right] = \int_\mathcal{X} q(\mathbf x) \log \frac{\mathcal{P}(\mathbf x)}{q(\mathbf x)} d\mathbf x$$

VI:

$$\text{KL}\left[ q(\mathbf x) || \mathcal{P}(\mathbf x)\right] = - \int_\mathcal{X} q(\mathbf x) \log \frac{\mathcal{P}(\mathbf x)}{q(\mathbf x)} d\mathbf x$$

Positive and Reverse KL

• Density Ratio Estimation for KL Divergence Minimization between Implicit Distributions - Blog Resources:
• YouTube
• Aurelien Geron - Short Intro to Entropy, Cross-Entropy and KL-Divergence
• Ben Lambert - Through Secret Codes
• Zhoubin - Video > A nice talk where he highlights the asymptotic conditions for MLE. The proof is sketched using the minimization of the KLD function.
• Blog
• Anna-Lena Popkes
• KLD Explained
• KLD for ML
• Reverse Vs Forward KL
• KL-Divergence as an Objective Function
• NF Slides (MLE context)
• Edward
• Class Notes
• Stanford - MLE | Consistency and Asymptotic Normality of the MLE | Fisher Information, Cramer-Raw LB | MLE Model Mispecification
• Code
• KLD py
• NumPy/SciPy Recipes