Back-propagation with gradient method is the most popular learning algorithm
for feed-forward neural networks. However, it is critical to determine a proper
fixed learning rate for the algorithm. In this paper, an optimized recursive
algorithm is presented for online learning based on matrix operation and
optimization methods analytically, which can avoid the trouble to select a
proper learning rate for the gradient method. The proof of weak convergence of
the proposed algorithm also is given.