Training Tips n Tricks

Learning Rate Scheduler¶

• WarmUp
• Cosine Annealing

Large Batch-Size¶

• 16, 32, 64, 128, 512, 1024, 2048

Fine-Tuning¶

1. Generic -> Adam
2. Fine-Tune -> LBFGS

Transfer Learning¶

Temporal Causlity¶

Architecture¶

• Temporal Encoders
• Tanh, Sine+Cosine, Fourier,

Training Perspective¶

• Sequence-to-Sequence Training
• Causal Learning Loss

Curriculum Learning¶