Recently, sparse inverse covariance matrix (SICE matrix) has been used as a representation of brain connectivity to classify Alzheimer’s disease and normal controls. However, its high dimensionality can adversely affect the classification performance. Considering the underlying manifold where SICE matrices reside and the common patterns shared by brain connectivity across subjects, we propose to explore the lower dimensional intrinsic components of SICE matrix for compact representation. This leads to significant improvements of brain connectivity classification. Moreover, to cater for the requirement of both discrimination and interpretation in neuroimage analysis, we develop a novel pre-image estimation algorithm to make the obtained connectivity components anatomically interpretable. The advantages of our method have been well demonstrated on both synthetic and real rs-fMRI data sets.