To improve the problem that the conventional design results of planetary gear transmission could not guarantee all of objectives were the optimal, aiming at the shortcomings that adaptive particle swarm optimization algorithm easily produced infeasible solution and fell into local optimum, an adaptive particle swarm optimization algorithm based on multi-level penalty function with dynamic change penalty coefficient was proposed. The algorithm set the penalty function as a multi-level penalty function and the penalty coefficient could be adjusted with the iterative process. The reliability and others were taken as the constraint conditions, the volume, the contact ratio and the transmission efficiency as the objective functions. Combined with the ideal point method to construct the evaluation function for multi-objective optimization. The results show that the volume is decreased by 41.87%, the contact ratio is increased by 2.643%, and the transmission efficiency is increased by 0.1563%. The improved particle swarm optimization improves the defects of original algorithm, and each objective is reasonably optimized.