Based on the three-dimensional W-M fractal function, a three-dimensional rough surface was bulidt by SOLIDWORKS, and then the three-dimensional ball-rough surface contact numerical model of torsional fretting with the energy wear law being introduced by the user subroutine UMESHMOTION was established by ABAQUS. Thus, the influence of initial surface roughness on the frictional wear process of torsional fretting was studied. Firstly, the reasonability of the rough surface model was verified by the analytical solution of dimensionless contact area in G-W model. Simultaneously, in comparison with experimental wear results, the accuracy and validity of the numerical model for predicting torsional fretting wear was proved. The research results show that the change of surface roughness has a significant effect on torsional fretting wear. To be specific, wear volume, friction dissipation energy, and wear rate all increase with the increase of surface roughness, which indicates that initial surface roughness can aggravate surface wear. And friction activation energy decreases with the increase of surface roughness. It is clear that the rougher the surface, the lower the threshold for wear activation. Therefore, reducing initial surface roughness can effectively alleviate the torsional fretting wear.