Two adaptive stepsize rules for gradient descent and their application to the training of feedforward artificial neural networks

Gradient descent, in the form of the well-known backpropagation algorithm, is frequently used to train feedforward neural networks, i.e. to find the weights which minimize some error measure ε. Generally, the stepsize is fixed, and represents a compromise between stability and speed of convergence. In this paper, we derive two methods for adapting the stepsize and apply them to train neural networks on parity problems of various sizes.