Fractional order phase lead-lag controller (FOPLC) is a generalization of classical phase lead-lag controller (CPLC) using fractional calculus. Nowadays, fractional order controllers (FOC) have found wide applications in various aspects of power system control and operation. In this paper, FOPLC are used in structure of flexible ac transmission system (FACTS) based frequency stabilizer to enhance the dynamic performance of automatic generation control (AGC) system. So, Thyristor controlled series capacitor (TCSC) based damping controller with FOPLC structure is proposed in coordination with secondary integral load frequency controller (LFC). The controller is optimized via an improved particle swarm optimization (IPSO) algorithm. The dynamic performance of the proposed controller is compared with CPLC using eigenvalue analysis and time domain simulations. An interconnected multi-source realistic power system with TCSC in series with the tie-line is investigated considering the nonlinearity effects of generation rate constraint (GRC) and governor dead band (GDB). Simulation results reveal that FOPLC based TCSC damping controller achieves superior dynamic performance under various load perturbation patterns in terms of decreased performance index, settling time and amplitude of oscillations. Furthermore, the robustness of the proposed controller has been tested for large uncertainties in system parameter and loading condition.