To a Jeffcott rotor system with support looseness fault,a vibration differential equation with piecewise-linear stiffness was established by considering the system with one-side support looseness and the rotor with unbalanced extraneous exciting force. Incremental harmonic balance method was exploited to solve the nonlinear equation,and iterative algorithm was applied to gain the vibration periodic solution. Then the stability of periodic solution was judged by Floquet theory. Finally,the time domain waveforms,amplitude spectrograms,phase diagrams and the Poincare sectional drawings were obtained through a simulation example,the complicated motion characteristics of rotor system with support looseness was analyzed. Meanwhile,the validity of solving nonlinear system by incremental harmonic balance method was proved,and the results laid a theoretical foundation for the diagnosis of support looseness fault.