Tooth surface spalls fault is the irregularly shaped partial pits on the gear surface. In the calculation of gear mesh stiffness. Common method is to simplify the tooth surface spalls to a specific regular shape for research, which has great limitations. Based on image processing, this paper extracted the edge features of the spalls fault, dissociated spalls area according to the pixels and fit the spalls width function. Based on the traditional potential energy method, mesh stiffness of the faulty gear is calculated more accurately and dynamics simulation are performed. Compared with the rectangular spalls fault, the mesh stiffness of the faulty gear after image processing can more accurately reflect the contour characteristics. Eliminating the stiffness and acceleration response mutation points caused by simplification. The proposed method has high versatility and is applicable to tooth spalls of arbitrary shapes.