Ecrêtage de gradient

Domaine

Catégorie Démo Apprentissage profond

Anglais

Gradient Clipping

Gradient Clipping is a technique to prevent exploding gradients in very deep networks, typically Recurrent Neural Networks. There exist various ways to perform gradient clipping, but the a common one is to normalize the gradients of a parameter vector when its L2 norm exceeds a certain threshold according to new_gradients = gradients * threshold / l2_norm(gradients). • On the difficulty of training recurrent neural networks