We investigate multi-task learning from an output space regularization
perspective. Most multi-task approaches tie together related tasks by
constraining them to share input spaces and function classes. In contrast to
this, we propose a multi-task paradigm which we call output space
regularization, in which the only constraint is that the output spaces of the
multiple tasks are related. We focus on a specific instance of output space
regularization, multi-task averaging, that is both widely applicable and
amenable to analysis.