In this paper, a new approach is presented to reduce the nonlinear characteristics of a stressed power system and improve the transient stability of the system by reducing its second-order modal interaction through retuning some parameters of the generator excitation system. In order to determine the second-order modal interaction of the system, a new index on nonlinearity is developed using the normal form theory. Using the proposed index of nonlinearity, a sensitivity function is formed to indicate the most effective excitation system parameters in the nonlinear behavior of the system. These dominant parameters are tuned to reduce the second-order modal interaction of the system and to reduce the index on nonlinearity. The efficiency of the proposed method is initially validated using a four-machine two-area test system. The IEEE 39-Bus New England test system is then used to investigate the performance of the proposed method for a more realistic system. Simulation results show that a proper tuning of the excitation controller can reduce the second-order modal interaction of the system and can even improve the transient stability margin of the network.