Once the gear has broken tooth, there will be periodic shocks, which make the vibration signal of gear meshing modulated. So it is important for diagnosis of gear tooth broken to extract the modulation information effectively. Aiming at the problems that the fault signature is drowned by noise, the filtering parameters usually need to be set artificially, and the detection rate of tooth broken is low, this paper proposes Improved Sideband Product Spectrum（ISPS）,and a diagnosis method for gear tooth broken by combining adaptive filtering on IHPS/ISPS and Hilbert transform demodulation. Experiments show that this method can effectively extract the fault characteristic signal buried by noise and has great application value for the diagnosis of tooth broken.