Learning to Learn

General Formulation¶

Whirlwind Tour¶

Optimized-Based¶

LSTM Meta-Optimizer¶

We will pay special attention to the

In many cases, we need to find the best state given the state (and parameters). Most gradient update schemes look like the following where it is fixed.

To find the optimal solution of this problem, we can write it down as:

where $\boldsymbol{g}_k$ is some result of a generalized gradient operator

where $k$ is the iteration, $\boldsymbol{\phi}$ are the parameters of the gradient operator, and $\boldsymbol{h}$ is the hidden state.