Zinc ions (Zn2+) are imported into the early secretory pathway by Golgi-resident transporters, but their handling and functions are not fully understood. Here, we show that Zn2+ binds with high affinity to the pH-sensitive chaperone ERp44, modulating its localization and ability to retrieve clients like Ero1α and ERAP1 to the endoplasmic reticulum (ER). Silencing the Zn2+ transporters that uptake Zn2+ into the Golgi led to ERp44 dysfunction and increased secretion of Ero1α and ERAP1. High-resolution crystal structures of Zn2+-bound ERp44 reveal that Zn2+ binds to a conserved histidine-cluster. The consequent large displacements of the regulatory C-terminal tail expose the substrate-binding surface and RDEL motif, ensuring client capture and retrieval. ERp44 also forms Zn2+-bridged homodimers, which dissociate upon client binding. Histidine mutations in the Zn2+-binding sites compromise ERp44 activity and localization. Our findings reveal a role of Zn2+ as a key regulator of protein quality control at the ER-Golgi interface.