The Alpha factor, also known as the linewidth enhancement factor, is one of the fundamental parameters for semiconductor lasers (SLs) as it characterizes many properties of the SLs, such as the responses to the electrical injection and the optical injection. Due to the great importance of alpha factor in research analysis and application design, the high accuracy of the experimental alpha measurement is required. The optical feedback self-mixing interferometry (OFSMI) based method for the alpha measurement is one of the popular approaches in the past twenty years due to its easy implementation and inexpensive, self-aligned experiment set-up. This paper proposes an effective data processing method applied in the frequency-domain based self-mixing approach for alpha factor measurement. The alpha value is estimated from the complex frequency spectrum of the feedback phase signal in an OFSMI system. However, some of the estimated results with large deviations are found in the experimental estimation due to the noises in practice. The work presented in the paper is twofold. Firstly, the errors of alpha estimation are analyzed. Secondly, the algorithm using distance-based outlier removal is proposed for optimizing the estimation results of alpha. The results show that the estimation accuracy of alpha can be achieved to 6.725% and 1.923% for the optical feedback level in the OFSMI system. © 2014 SPIE.