Preprocessing based solution for the vanishing gradient problem in recurrent neural networks

In this paper, a possible solution to the vanishing gradient problem in recurrent neural network is proposed. The main idea consists of pre-processing the signal (a time series typically) through a wavelet decomposition, in order to separate the short term information from the long term one, and treating each scale by different RNNs. The partial results concerning all the different scales of time and frequencies are combined by another 'expert' (a nonlinear structure typically) in order to achieve the final goal. This new approach is distinct from the other ones reported in literature to-date, as it tends to simplify the RNN's learning, working directly at a signal level and avoiding relevant changing in network's architecture and learning techniques. The overall system (called Recurrent Multiscale Network, RMN) is described and its performances tested through typical tasks namely the latching problem and time series prediction.