A theoretical framework is presented to clarify the molecular determinants of ion selectivity in protein binding sites. The relative free energy of a bound ion is expressed in terms of the main coordinating ligands coupled to an effective potential of mean force representing the influence of the rest of the protein. The latter is separated into two main contributions. The first includes all the forces keeping the ion and the coordinating ligands confined to a microscopic subvolume but does not prevent the ligands from adapting to a smaller or larger ion. The second regroups all the remaining forces that control the precise geometry of the coordinating ligands best adapted to a given ion. The theoretical framework makes it possible to delineate two important limiting cases. In the limit where the geometric forces are dominant (rigid binding site), ion selectivity is controlled by the ion-ligand interactions within the matching cavity size according to the familiar "snugfit" mechanism of host-guest chemistry. In the limit where the geometric forces are negligible, the ion and ligands behave as a "confined microdroplet" that is free to fluctuate and adapt to ions of different sizes. In this case, ion selectivity is set by the interplay between ion-ligand and ligand-ligand interactions and is controlled by the number and the chemical type of ion-coordinating ligands. The framework is illustrated by considering the ion-selective binding sites in the KcsA channel and the LeuT transporter.