An optimal cooperative beamforming for the amplify-and-forward (AF) MIMO two-way relay channels was designed. Supposing the channel state information (CSI) was perfectly known by the receiver and transmitter as well as the relay, optimal beamforming vectors (matrices) of all nodes were jointly designed based on the criterion of minimizing the sum mean square errors (MSMSE). The analysis result shows that the performance effect of transmitting and receiving beamforming pairs is to maximize the receive signal-to-noise ratio (SNR) at two communication nodes, and the rank of the optimal relay beamforming matrix is no larger than two when there is only one data stream at each source node. A simplified algorithm was put forward to accomplish the design based on the analysis conclusions. Simulation results provide that the system performance, which is characterized in terms of bit error rates (BER), is significantly improved by cooperative beamforming, and the performance of the simplified method is not only very close to the optimal one but also with faster iteration speed and much lower computational complexity.
