Interfacial layers are frequently used in organic solar cells performing various functions, including blocking surface recombination, improving selectivity of charge carrier extraction, modification of the work function of the contact materials, and enhancing light absorption within the photoactive layer through an optical cavity effect. The aim of this work is to investigate the origin of performance enhancement of bulk heterojunction solar cells using various electron and hole interfacial layers, with a particular focus on separating the contributions of work function modification and reduced recombination to the improvement of the open circuit voltage (Voc). Solar cells using poly[N-9′-hepta-decanyl-2,7-carbazole-alt-5,5-(4′,7′-di-2-thienyl-2′,1′,3′-benzothiadiazole)]:[6,6]-phenyl C70-butyric acid methyl ester (1:4) active layers were prepared with a combination of polymeric, metal oxide, and polyelectrolyte electron and/or hole interfacial layers. Four device structures with (i) no interfacial layers (reference); (ii) only hole; (iii) only electron; (iv) both electron and hole interfacial layers were fabricated and compared using current-voltage, transient photovoltage, and charge extraction measurements. The voltage gains (ΔVoc) at matched charge density attributed to work function modification (ΔVoch or ΔVoce) are distinguished from the increase in Voc arising from increased charge carrier density. At the hole contact, ΔVoch was 0.21 V by using a poly(3,4-ethylenedioxythiophene):poly(styrenesulfonate) hole interfacial layer, whereas ΔVoce was 0.29 V on the electron contact using a polyethoxylate imine-TiOx interfacial layer compared to reference devices. The electron lifetime also improved by orders of magnitude with the use of either electron or hole contact layers, contributing to a further 0.35-0.38 V increase in the open circuit voltage (ΔVocrec) because of increased charge density. The increased charge carrier lifetime is proposed to originate from the larger spatial separation of the electrons and holes in the device because of the increased internal field. Using both an electron and a hole interfacial layer did not significantly increase the charge carrier lifetime compared to single interfacial layer devices; therefore, the Voc did not increase significantly. The findings presented clarify the role of interfacial layers in organic solar cells and provide new insights into using time-resolved charge extraction techniques to understand the influence of interfacial layers on the open circuit voltage.