In epidemiology and psychiatry research, the Parental Bonding Instrument (PBI) is commonly used to assess offspring’s perception on maternal and paternal behavior during childhood. We tested the 2- versus 3-factor structure of the 16-item version and assessed measurement invariance across sex and across lifetime depressed, anxious, comorbid affected, and healthy participants. Subsequently, we investigated PBI dimensions across sex and psychopathology groups using structural equation modeling. Participants were 2,069 adults with a lifetime affective disorder and healthy controls, ages 26–75, from the Netherlands. Our findings support the 3-factor solution of the distinct mother and father scales, distinguishing care, overprotection, and autonomy (previously “authoritarianism”). Moreover, measurement of the PBI appeared to be invariant across groups, indicating that means and relations can be reliably compared across sex and psychopathology groups. Men reported more maternal overprotection and paternal lack of care, whereas females reported higher paternal and maternal lack of autonomy and maternal lack of care levels compared with males. Lack of care and lack of autonomy levels were elevated in all affected groups, with the comorbid group showing highest levels of all 3 PBI dimensions. Adults with anxiety disorders reported heightened maternal lack of autonomy levels compared with the depression group and healthy controls. Adults with a depressive disorder reported heightened paternal lack of care levels as compared with the anxiety group and healthy controls. We advocate to use the 3-factor structure and conclude that suboptimal parental bonding, mainly lack of care and lack of autonomy, is associated with lifetime anxiety and depression.