Within animal societies, the ecological and social underpinnings of mating system variation can be related to resource dispersion, sexual conflict between breeders, and the effects of non-breeders. Here, we conducted a broad-scale investigation into the evolution of mating systems in the cooperatively breeding cichlid, Neolamprologus pulcher, a species that exhibits both monogamy and polygyny within populations. Using long-term field data, we showed that polygynous groups were more spatially clustered and held by larger competitively superior males than were monogamous groups, supporting the role of resource dispersion. To explore the role of sexual conflict, we forced polygynous males to become experimentally monogamous (EM) in the field. EM males spent more time on their remaining territory than naturally polygynous males but otherwise did not change behaviorally or physiologically. Females mated to EM males performed more submissive acts, and in a forced choice experiment, females did not preferentially associate with the larger of two unmated males. Females may therefore incur an unexpected cost from mating monogamously with a large and competitively superior male, a cost that mitigates sexual conflict over the mating system. Helpers were more closely related in monogamous groups but did not behave differently under monogamy or polygyny. Helpers therefore seem neither to be affected by nor affect the mating system of breeders. Our results demonstrate the roles of resource availability and conflict mitigation in determining the mating system, and highlight the importance of experimental manipulation for revealing hidden costs of hypothetical mating patterns.