Projection-based methods for stepsize adaptation and their application to the training of feedforward artificial neural networks

We develop several adaptive step-size rules for gradient descent based on projecting weight and gradient vectors onto a set of unit vectors; each unit vecor induces a one dimensional optimization problem which is solved by minimizing a fitted quadratic. A sum of squares criterion is then used to find the stepsize which which best fits the solution to each one dimensional optimization. The resulting stepsize rules are applied to train neural networks on parity problems of various sizes.