Sulfides are promising anode candidates because of their relatively large theoretical discharge/charge specific capacity and pretty small volume changes, but suffers from sluggish kinetics and structural instability upon cycling. Phase engineering can be designed to overcome the weakness of the electrochemical performance of sulfide anodes. By choosing nickel sulfides (α-NiS, β-NiS, and NiS2) supported by reduced graphene oxide (rGO) as model systems, it is demonstrated that the nickel sulfides with different crystal structures show different performances in both sodium-ion and potassium-ion batteries. In particular, the α-NiS/rGO display superior stable capacity (≈426 mAh g−1 for 500 cycles at 500 mA g−1) and exceptional rate capability (315 mAh g−1 at 2000 mA g−1). The combined density functional theory calculations and experimental studies reveal that the hexagonal structure is more conducive to ion absorption and conduction, a higher pseudocapacitive contribution, and higher mechanical ability to relieve the stress caused by the volume changes. Correspondingly, the phase engineered nickel sulfide coupled with the conducting rGO network synergistically boosts the electrochemical performance of batteries. This work sheds light on the use of phase engineering as an essential strategy for exploring materials with satisfactory electrochemical performance for sodium-ion and potassium-ion batteries.