Many structured data-fitting applications require the solution of an
optimization problem involving a sum over a potentially large number of
measurements. Incremental gradient algorithms (both deterministic and
randomized) offer inexpensive iterations by sampling only subsets of the terms
in the sum. These methods can make great progress initially, but often slow as
they approach a solution. In contrast, full gradient methods achieve steady
convergence at the expense of evaluating the full objective and gradient on
each iteration.