Dynamic performance of gear system especially vibration was a typical nonlinear problem,while the vibration was closely related to the gear profile modification,the solving gear profile modification parameters was a nonlinear optimization problem. With a pair of spur gear as research object,armed with the kisssoftsoftware,by using orthogonal design method and genetic algorithm,aiming at reducing the fluctuation of vibration acceleration,optimum profile modifications of spur gears was designed. The research shows that orthogonal experimental method is combined with genetic algorithm for complex optimization problems,the fluctuation of vibration acceleration significantly reduced,and the damping effect is good in the paper.