When a liquid is cooled well below its melting temperature at a rate that exceeds the critical cooling rate Rc, the crystalline state is bypassed and a metastable, amorphous glassy state forms instead. Rc (or the corresponding critical casting thickness dc) characterizes the glass-forming ability (GFA) of each material. While silica is an excellent glass-former with small Rc < 10(-2) K/s, pure metals and most alloys are typically poor glass-formers with large Rc > 10(10) K/s. Only in the past thirty years have bulk metallic glasses (BMGs) been identified with Rc approaching that for silica. Recent simulations have shown that simple, hard-sphere models are able to identify the atomic size ratio and number fraction regime where BMGs exist with critical cooling rates more than 13 orders of magnitude smaller than those for pure metals. However, there are a number of other features of interatomic potentials beyond hard-core interactions. How do these other features affect the glass-forming ability of BMGs? In this manuscript, we perform molecular dynamics simulations to determine how variations in the softness and non-additivity of the repulsive core and form of the interatomic pair potential at intermediate distances affect the GFA of binary alloys. These variations in the interatomic pair potential allow us to introduce geometric frustration and change the crystal phases that compete with glass formation. We also investigate the effect of tuning the strength of the many-body interactions from zero to the full embedded atom model on the GFA for pure metals. We then employ the full embedded atom model for binary BMGs and show that hard-core interactions play the dominant role in setting the GFA of alloys, while other features of the interatomic potential only change the GFA by one to two orders of magnitude. Despite their perturbative effect, understanding the detailed form of the intermetallic potential is important for designing BMGs with cm or greater casting thickness.