Members of the Tre2-Bub2-Cdc16 (TBC) family often function to regulate membrane trafficking and to control signaling transductions pathways. As a member of the TBC family, TBC1D23 is critical for endosome-to-Golgi cargo trafficking by serving as a bridge between Golgi-bound golgin-97/245 and the WASH/FAM21 complex on endosomal vesicles. However, the exact mechanisms by which TBC1D23 regulates cargo transport are poorly understood. Here, we present the crystal structure of the N-terminus of TBC1D23 (D23N), which consists of both the TBC and rhodanese domains. We show that the rhodanese domain is unlikely to be an active sulfurtransferase or phosphatase, despite containing a putative catalytic site. Instead, it packs against the TBC domain and forms part of the platform to interact with golgin-97/245. Using the zebrafish model, we show that impacting golgin-97/245-binding, but not the putative catalytic site, impairs neuronal growth and brain development. Altogether, our studies provide structural and functional insights into an essential protein that is required for organelle-specific trafficking and brain development.