We consider the limiting distribution of $U_NA_NU_N^*$ and $B_N$ (and more
general expressions), where $A_N$ and $B_N$ are $N \times N$ matrices with
entries in a unital C$^*$-algebra $\mathcal B$ which have limiting $\mathcal
B$-valued distributions as $N \to \infty$, and $U_N$ is a $N \times N$ Haar
distributed quantum unitary random matrix with entries independent from
$\mathcal B$. Under a boundedness assumption, we show that $U_NA_NU_N^*$ and
$B_N$ are asymptotically free with amalgamation over $\mathcal B$. Moreover,
this also holds in the stronger infinitesimal sense of Belinschi-Shlyakhtenko.

We provide an example which demonstrates that this example may fail for
classical Haar unitary random matrices when the algebra $\mathcal B$ is
infinite-dimensional.