The shuffled frog leaping algorithm（ SFLA） is a swarm intelligent optimization algorithm that combines competitive evolutionary strategy and limited random search. It has been applied to various optimization problems. This paper analyzed the advantages of the SFLA for global optimization problems,and established optimization BP neural network algorithm model（ SFLA-BP） based on SFLA,and carried on the simulation study. The planetary gearbox was taken as an engineering example,and its nonlinear characteristic was presented because of transmission path of fault vibration signals complicated and their coupling with each others. The kernel principal component analysis（ KPCA） was used to extract the time domain and frequency domain sensitive features,and the feature dimensions were reduced from 27 to 9. A neural network fault diagnosis system with 9-14-4 structure was established. This paper makes full use of the advantage of the global search of SFLA to realize fault diagnosis of planetary gear with different wear levels. The diagnosis results show that the SFLA-BP model has a smaller overall output error compared with BP neural network,and the diagnostic accuracy increases by 12. 5 %,and has achieved a more accurate identification effect on different degrees damage faults.