The initial microstructures with the unimodal and bimodal size distribution of γ′ precipitates (referred to as UD and BD, respectively) in CoNi-based superalloy are obtained utilizing different heat treatment regimes. In the present work, compression tests are carried out to investigate the anomalous temperature dependence of strength and the effect of initial microstructure on the deformation mechanism of CoNi-based superalloys. Results show that the anomalous yield occurs above 600 °C, and BD specimen exhibits higher yield stress than UD specimen. In UD specimen, the major deformation mechanisms are pairs of a/2 < 110 > dislocations coupled by APB (antiphase boundary) shearing γ′ precipitates (600–700 °C), and dislocations bypassing the γ′ precipitates accompanied with the SISFs (superlattice intrinsic stacking faults) shear above the peak temperature (900 °C). In BD specimen, the predominant deformation mechanism transitioned from the APB shearing (600 °C) to APB shearing and SISFs shearing (700 °C), and then to the dislocations bypassing the γ′ precipitates together with the SISFs shearing (900 °C). Meanwhile, the relationship between the initial microstructure and deformation mechanism is discussed. Moreover, an attempt is made to quantitatively assess the contribution of secondary γ′ precipitates to the strength at the peak temperature.