Two modulation techniques are studied in this thesis, quadrature amplitude modulation (QAM) and continuous phase modulation (CPM). These modulation schemes in the presence of amplifier nonlinearity and hostile transmission environments are analyzed as a complete system using simulation to determine relative performance. A fair comparison of these two modulation techniques is carried out, demonstrating the tradeoffs between spectral efficiency and power efficiency of these two techniques. This thesis also presents an idea of nonuniform QAM constellations, which can potentially improve bit-error-rate performance in the presence of phase noise. An optimal nonuniform QAM constellation is designed to be adaptive to the actual probability distribution of the received symbols. Both analytical results and simulation results show that nonuniform constellations are superior to uniform constellations in the presence of phase noise.