Participation in physical activity (PA) tends to decline between childhood and adolescence, and in Australia the bulk of this decline has been shown to occur in the domain of non-organised PA (e.g. active play, informal sports). A better understanding of the reasons for the decline in non-organised PA may support strategies to counter barriers and strengthen enablers of participation. This study aimed to explore the stories of young adults who withdrew from non-organised PA and overall PA between 11y and 15y. Specifically, we sought to explore social norms related to non-organised PA during adolescence, and barriers and enablers of participation. Participants were young adults (18–22y) who were recruited from undergraduate, vocational and special entrance classes, as well as two churches in the Illawarra region, Australia. Recruitment was stratified by sex and socioeconomic status. Data collection occurred via semi-structured interviews with embedded Life History Calendars (n = 22). Thematic analysis was supported by concept and pattern coding. Findings revealed that social norms were related to an interplay of adult modelling/influence, concerns about being childish, puberty, identity development, adult choices and responsibilities, and changing life circumstances. Barriers to participation centred on fears of being different, bullying, peer judgement and rejection. Enablers of participation included safe people and places, accessible games (e.g. handball/foursquare) and, for girls, having an identity that supported challenging gender norms. Future PA promotion strategies may involve ‘reframing’ childhood activities to be appropriate for adolescents, and emphasising identity-congruent types of PA (e.g. active video games, drama games). Such strategies may be implemented in a similar style to after-school intramural sports to allow youth to participate despite potential changes in their life circumstances during adolescence.