We investigate in the paper general (not necessarily definite) canonical
systems of differential equation in the framework of extension theory of
symmetric linear relations. For this aim we first introduce the new notion of a
boundary relation $\G:\gH^2\to\HH$ for $A^*$, where $\gH$ is a Hilbert space,
$A$ is a symmetric linear relation in $\gH, \cH_0$ is a boundary Hilbert space
and $\cH_1$ is a subspace in $\cH_0$. Unlike known concept of a boundary
relation (boundary triplet) for $A^*$ our definition of $\G$ is applicable to
relations $A$ with possibly unequal deficiency indices $n_\pm(A)$. Next we
develop the known results on minimal and maximal relations induced by the
general canonical system $ J y'(t)-B(t)y(t)=\D (t)f(t)$ on an interval
$\cI=(a,b),\; -\infty\leq a<b\leq\infty $ and then by using a special (so
called decomposing) boundary relation for $\Tma$ we describe in terms of
boundary conditions proper extensions of $\Tmi$ in the case of the regular
endpoint $a$ and arbitrary (possibly unequal) deficiency indices $n_\pm
(\Tmi)$. If the system is definite, then decomposing boundary relation $\G$
turns into the decomposing boundary triplet $\Pi=\bt$ for $\Tma$. Using such a
triplet we show that self-adjoint decomposing boundary conditions exist only
for Hamiltonian systems; moreover, we describe all such conditions in the
compact form. These results are generalizations of the known results by
Rofe-Beketov on regular differential operators. We characterize also all
maximal dissipative and accumulative separated boundary conditions, which exist
for arbitrary (not necessarily Hamiltonian) definite canonical systems.