The last decades of life have been traditionally viewed as a time of inevitable disease and frailty. Sedentary living and physical activity may influence capacity to perform activities that are needed to maintain physical independence in daily living. A total of 117 males and 195 females, aged 65–103 years, were assessed for physical activity and sedentary time with accelerometers and for functional fitness with the Senior Fitness Test battery. Based on the individual scores for each fitness item, a Z-score was created. Associations between functional fitness with sedentary time and moderate-to-vigorous physical activity (MVPA) were analyzed. A negative association was found between the composite Z-score for functional fitness and the sedentary time, even adjusting for MVPA and other confounders. On the other hand, MVPA was positively associated with the composite Z-score for functional fitness, independently of the sedentary time. In conclusion elderly who spend more time in physical activity or less time in sedentary behaviors exhibit improved functional fitness and other confounders. The results reinforce the importance of promoting both the reduction of sedentary behaviors and the increase of MVPA in this age group, as it may interfere at older ages in order to preserve functional fitness and performance of daily functioning tasks.