We study quasi-Newton methods from the viewpoint of information geometry
induced associated with Bregman divergences. Fletcher has studied a variational
problem which derives the approximate Hessian update formula of the
quasi-Newton methods. We point out that the variational problem is identical to
optimization of the Kullback-Leibler divergence, which is a discrepancy measure
between two probability distributions. The Kullback-Leibler divergence for the
multinomial normal distribution corresponds to the objective function Fletcher
has considered. We introduce the Bregman divergence as an extension of the
Kullback-Leibler divergence, and derive extended quasi-Newton update formulae
based on the variational problem with the Bregman divergence. As well as the
Kullback-Leibler divergence, the Bregman divergence introduces the information
geometrical structure on the set of positive definite matrices. From the
geometrical viewpoint, we study the approximation Hessian update, the
invariance property of the update formulae, and the sparse quasi-Newton
methods. Especially, we point out that the sparse quasi-Newton method is
closely related to statistical methods such as the EM-algorithm and the
boosting algorithm. Information geometry is useful tool not only to better
understand the quasi-Newton methods but also to design new update formulae.