Let $N$ be a connected and simply connected nilpotent Lie group, $\Lambda$ a
lattice in $N$, and $X=N/\Lambda$ the corresponding nilmanifold. Let $Aff(X)$
be the group of affine transformations of $X$. We characterize the countable
subgroups $H$ of $Aff(X)$ for which the action of $H$ on $X$ has a spectral
gap, that is, such that the associated unitary representation $U$ of $H$ on the
space of functions from $L^2(X)$ with zero mean does not weakly contain the
trivial representation. Denote by $T$ the maximal torus factor associated to
$X$. We show that the action of $H$ on $X$ has a spectral gap if and only if
there exists no proper $H$-invariant subtorus $S$ of $T$ such that the
projection of $H$ on $Aut (T/S)$ has an abelian subgroup of finite index. We
first establish the result in the case where $X$ is a torus. In the case of a
general nilmanifold, we study the asymptotic behaviour of matrix coefficients
of $U$ using decay properties of metaplectic representations of symplectic
groups. The result shows that the existence of a spectral gap for subgroups of
$Aff(X)$ is equivalent to strong ergodicity in the sense of K.Schmidt.
Moreover, we show that the action of $H$ on $X$ is ergodic (or strongly mixing)
if and only if the corresponding action of $H$ on $T$ is ergodic (or strongly
mixing).