Conventional fast Fourier transform（ FFT） has been the most popular method of processing non-stationary and nonlinear signals. However,FFT has been known to be less efficient because of aliasing. In view of such constraints,this paper proposes a new method based on Hilbert-Huang transform-FFT of intrinsic mode functions（ IMFs）. In the present work,FFT of IMFs from HHT process has been incorporated to utilize efficiency of FFT in frequency domain. Take transmission system of mining machine cutting department for example,nonlinear vibration signals has been measured by experiment. Conventional FFT and FFT of intrinsic mode functions（ IMFs） are used for analyzing the nonlinear modulation characteristics of the experimental signals respectively. The comparative analysis presented in this paper indicates the effectiveness of using FFT of IMFs for recognizing the frequency components which participate in nonlinear frequency modulation. The method proposed can avoid the phenomenon of aliasing.