When a fraction of the light reflected by an external cavity re-enters the laser cavity,
both the amplitude and the frequency of the lasing field can be modulated. This phenomenon is
called the self-mixing effect (SME). A self-mixing laser diode (SM-LD) is a sensor using the SME.
Usually, such LDs operate below the stability boundary where no relaxation oscillation happens.
The boundary is determined by the operation condition including the injection current, optical
feedback strength and external cavity length. This paper discovers the features of an SM-LD where
the LD operates beyond the stability boundary, that is, near the relaxation oscillation (RO) status.
We call the signals from such a SM-LD as RO-SM signals to differentiate them from the conventional
SM signals reported in the literature. Firstly, simulations are made based on the well-known Lang
and Kobayashi (L-K) equations. Then the experiments are conducted on different LDs to verify
the simulation results. It shows that a RO-SM signal exhibits high frequency oscillation with its
amplitude modulated by a slow time varying envelop which corresponds to the movement of the
external target. The envelope has same fringe structure (half-wavelength displacement resolution)
with the conventional SM signals. However, the amplitudes of the RO-SM signals are much higher
compared to conventional SM signals. The results presented reveal that an SM-LD operating near the
RO has potential for achieving sensing with improved sensitivity.