Considering the time-varying meshing stiffness and gear backlash, a dynamic model of gear transmission system is established. The Floquet theory is used to obtain the approximate analytic solution and the stability boundary curve of the transmission system. The Newmark-β numerical simulation method was used to simulate dynamic characteristics of gear system. The effect of time-varying mesh stiffness, backlash and damping effect on dynamic characteristics are studied. The results show that the system may generate parametric resonance when the frequency of parameter excitation is equal to 2/n of the free vibration period of the derived system;The gear backlash leads to multi-value properties and amplitude jump phenomena; The nonlinear vibration response of the system will be decreased with the increasing of the external excitation amplitude, damping ratio and the decreasing of the internal excitation amplitude.