Background Breathlessness is a multidimensional symptom prevalent in elderly affecting many aspects of life. We aimed to determine how different dimensions of breathlessness are associated with physical and mental quality of life (QoL) in elderly men. Methods This was a cross-sectional, population-based analysis of 672 men aged 73 years in a Swedish county. Breathlessness was assessed using Dyspnoea-12 (D-12) and Multidimensional Dyspnoea Profile (MDP), and QoL using the Short Form 12 physical and mental scores. Scores were compared as z-scores across scales and analysed using multivariable linear regression, adjusted for smoking, body mass index and the presence of respiratory and cardiovascular disease. Results Worse breathlessness was related to worse physical and mental QoL across all the D-12 and MDP dimension scores. Physical QoL was most strongly associated with perceptional breathlessness scores, D-12 total and physical scores (95% CI -0.45 to -0.30). Mental QoL was more strongly influenced by affective responses, MDP emotional response score (95% CI -0.61 to -0.48). Head-to-head comparison of the instruments confirmed that D-12 total and physical scores most influenced physical QoL, while mental QoL was mostly influenced by the emotional responses captured by the MDP. Conclusion Breathlessness dimensions and QoL measures are associated differently. Physical QoL was most closely associated with sensory and perceptual breathlessness dimensions, while emotional responses were most strongly associated with mental QoL in elderly men. D-12 and MDP contribute complimentary information, where affective and emotional responses may be related to function, deconditioning and QoL.