Protein glycosylation is one of the major biosynthetic functions occurring in the endoplasmic reticulum and Golgi compartments. It requires an amazing number of enzymes, chaperones, lectins and transporters whose actions delicately secure the fidelity of glycan structures. Over the past 30 years, glycobiologists hammered that glycan structures are not mere decorative elements but serve crucial cellular functions. This becomes dramatically illustrated by a group of mostly severe, inherited human disorders named congenital disorders of glycosylation (CDG). To date, many types of CDG have been defined genetically and most of the time the defects impair the biosynthesis, transfer and remodeling of N-glycans. Recently, the identification of the several types of CDG caused by deficiencies in the conserved oligomeric Golgi (COG) complex, a complex involved in vesicular Golgi trafficking, expanded the field of CDG but also brought novel insights in glycosylation. The molecular mechanisms underlying the complex pathway of N-glycosylation in the Golgi are far from understood. The availability of COG-deficient CDG patients and patients' cells offered a new way to study how COG, and its different subunits, could influence the Golgi N-glycosylation machinery and localization. This review summarizes the recent findings on the implication of COG in Golgi glycosylation. It highlights the need for a dynamic, finely tuned balance between anterograde and retrograde trafficking for the correct localization of Golgi enzymes to assure the stepwise maturation of N-glycan chains.