Getting familiar with a face is an important cognitive process in human perception of faces, but little study has been reported on how to objectively measure the degree of familiarity. In this article, a method is proposed to quantitatively measure the familiarity of a face with respect to a set of reference faces that have been seen previously. The proposed method models the context-free and context-dependent forms of familiarity suggested by psychological studies and accounts for the key factors, namely exposure frequency, exposure intensity and similar exposure, that affect human perception of face familiarity. Specifically, the method divides the reference set into nonexclusive groups and measures the familiarity of a given face by aggregating the similarities of the face to the individual groups. In addition, the nonnegative matrix factorization (NMF) is extended in this paper to learn a compact and localized subspace representation for measuring the similarities of the face with respect to the individual groups. The proposed method has been evaluated through experiments that follow the protocols commonly used in psychological studies and has been compared with subjective evaluation. Results have shown that the proposed measurement is highly consistent with the subjective judgment of face familiarity. Moreover, a face recognition method is devised using the concept of face familiarity and the results on the standard FERET evaluation protocols have further verified the efficacy of the proposed familiarity measurement.