This paper considers the issue of passive localization based on time of arrival (TOA) measurement in the presence of receiver detection failures. In passive localization, the signal sent from the transmitter is reflected or relayed by «passive» target and then received at several distributed receivers. The target's position can be determined by collecting range mea- surements from all receivers. With a linearized model for range measurements, we build a factor graph model and implement Gaussian message passing algorithm to obtain target location and detect link failures. The Cramer-rao bound (CRB) is also derived to evaluate the performance of proposed algorithm. Simulation results verify the effectiveness of proposed factor graph approach.