The way we view ourselves may play an important role in our responses to interpersonal interactions. In this study, we investigate how feedback valence, consistency of feedback with self-knowledge and global self-esteem influence affective and neural responses to social feedback. Participants (N=46) with a high range of self-esteem levels performed the social feedback task in an MRI scanner. Negative, intermediate and positive feedback was provided, supposedly by another person based on a personal interview. Participants rated their mood and applicability of feedback to the self. Analyses on trial basis on neural and affective responses are used to incorporate applicability of individual feedback words. Lower self-esteem related to low mood especially after receiving non-applicable negative feedback. Higher self-esteem related to increased posterior cingulate cortex and precuneus activation (i.e. self-referential processing) for applicable negative feedback. Lower self-esteem related to decreased medial prefrontal cortex, insula, anterior cingulate cortex and posterior cingulate cortex activation (i.e. self-referential processing) during positive feedback and decreased temporoparietal junction activation (i.e. other referential processing) for applicable positive feedback. Self-esteem and consistency of feedback with selfknowledge appear to guide our affective and neural responses to social feedback. This may be highly relevant for the interpersonal problems that individuals face with low self-esteem and negative self-views.