KLD

Invariant Under Transformations

$$\text{D}_\text{KL}\left[p(y)||q(y) \right] = \int_\mathcal{Y}p(y)\log \frac{p(y)}{q(y)}dy$$

Let's make a transformation on $p(y)$ using some nonlinear function $f()$. So that leaves us with $y = f(x)$. So let's apply the change of variables formula to get the probability of $y$ after the transformation.

$$p(y)dy=p(x)dx = p(x)\left|\frac{dx}{dy} \right|$$

Remember, we defined our function as $y=f(x)$ so technically we don't have access to the probability of $y$. Only the probability of $x$. So we cannot take the derivative in terms of y. But we can take the derivative in terms of $x$. So let's rewrite the function:

$$p(y) = p(x) \left| \frac{dy}{dx} \right|^{-1}$$

Now, let's plug in this formula into our KLD formulation.

$$\text{D}_\text{KL}\left[p(y)||q(y) \right] = \int_\mathcal{y=?}p(x)\left| \frac{dy}{dx} \right|^{-1} \log \frac{p(x) \left| \frac{dy}{dx} \right|^{-1}}{q(y)}dy$$

We still have two terms that need to go: $dy$ and $q(y)$. For the intergration, we can simply multiple by 1 to get $dy\frac{dx}{dx}$ and then with a bit of rearranging we get: $\frac{dy}{dx}dx$. I'm also going to change the notation as well to get $\left| \frac{dy}{dx} \right|dx$. And plugging this in our formula gives us:

$$\text{D}_\text{KL}\left[p(y)||q(y) \right] = \int_\mathcal{y=?}p(x)\left| \frac{dy}{dx} \right|^{-1} \log \frac{p(x) \left| \frac{dy}{dx} \right|^{-1}}{q(y)} \left| \frac{dy}{dx} \right|dx$$

Now, we still have the distribution $q(y)$.