This article explores how care homes—and, specifically, their common features such as dementia care units and locked doors and gates—impact on the human rights of people living with dementia. We suggest that congregation, separation and confinement of people living with dementia by the care home built environment constitute ‘segregation’. In the specific context of residential aged care facilities in Australia, we draw on the United Nations Convention on the Rights of Persons with Disabilities (‘CRPD’) to frame this segregation as an injustice. We focus on the rights to non-discrimination (Article 5), liberty and security of the person (Article 14), equality before the law (Article 12), accessibility (Article 9), and independent living and community inclusion (Article 19). Our analysis shows that addressing segregation must involve structural and resource reforms that are transformative in bringing about new ways of living and relating to each other. Such reforms are directed towards providing meaningful alternatives and appropriate supports to make choices from a range of alternative residency and support options, and building communities that are free from ableism, ageism and other systems of oppression that contribute to confinement and segregation.