We propose a novel vertical Bell Laboratories layered space-time (V-BLAST) based two-way relayingprotocol that comprises two user terminals and one multi-antenna demodulate-and-forward (DMF) relaystation. We consider the situation when these two user terminals employ different modulation types andinvestigate its detection techniques. The key feature of the detection techniques is that we quantify decisionreliability at the relay station by introducing a so-called normalized correlation coefficient (CORR),rather than having full confidence in demodulation at the relay station. With the assistance of the normalizedconelation coefficient, we develop detection methods at both the relay station and user terminals. Inparticular, we propose a CORR-based ordering algorithm at the relay station which can be applied to detectionfor general V-BLAST architectures. Simulation results demonstrate the superiority of the CORR-basedordering algorithm over the conventional signal-to-noise based ordering algorithm. Finally, wedemonstrate overall error-performance of the V-BLAST-based DMF two-way relaying protocol via simulations.
