PURPOSE: To reduce the cost by utilizing the same nonlinear 16 QAM signal as a maximum amplitude with respect to the amplitude of an intermediate amplitude signal.

CONSTITUTION: In the nonlinear 16 QAM modulation system, a maximum amplitude detecting means 1 detects the input of a signal which should have the maximum amplitude in standard 16 QAM. An amplitude suppressing means 2 suppresses the amplitude of this signal to 0.73-fold amplitude based on detection of the input in the means 1. In the nonlinear 16 QAM demodulation system, a quadrant discriminating means 3 discriminates the quadrant of the received nonlinear 16 QAM signal, and an amplitude discriminating means 4 discriminates the amplitude, and a phase discriminating means 5 discriminates the phase. It is discriminated whether the received nonlinear 16 QAM signal should have the maximum amplitude of standard 16 QAM or not based on these discrimination results. An amplitude restoring means 7 receives the affirmative discrimination from a logic discrimination means 6 and extends the amplitude of the reception signal 1.37 times to restore the amplitude of the original maximum amplitude signal. Thus, the system gain is improved by about 1db without changing the power cnsumption of a transmission amplifier, and the cost is reduced.