To accurately characterize the different operating states of the rotor system, a modified variational mode decomposition（MVMD） combined with instantaneous energy distribution permutation entropy（IEDPE） is proposed to quantify and extract rotor fault features. The parameters of the variational mode decomposition（VMD） decomposition layer are determined using correlation threshold strategy and the MVMD algorithm is designed. Then, the acquired vibration signals are input into the MVMD for decomposition and the instantaneous energy distribution of each mode component is calculated. Finally, the IEDPE of each mode component is extracted as the fault quantization feature. Compared with MVMD+PE, EMD+IEDPE, and EEMD+IEDPE, the results show that the fault features extracted by the proposed method are more effective in characterizing significant differences between rotor fault states, allowing more accurate recognition of fault categories.