Working memory performance is significantly influenced by genetic factors. Here, we assessed genetic contributions to both working memory performance and neuroimaging measures focused on the network of brain regions associated with working memory by using a sample of 467 human participants from extended families. Imaging measures included diffusion tensor imaging indices in major white matter tracts thought to be associated with working memory and structural magnetic resonance imaging measures of frontal and parietal gray matter density. Analyses directly addressed whether working memory performance and neural structural integrity are influenced by common genetic factors (e.g., pleiotropy). While all cognitive measures, gray matter regions, and white matter tracts assessed were heritable, only performance on a spatial delayed response task and integrity of the superior longitudinal fasciculus (a primary fronto-parietal connection) shared genetic factors. As working memory may be a core component of other higher level processes, such as general intelligence, this finding has implications for the heritability of complex cognitive functions, as well as for our understanding of the transmission of cognitive deficits in mental and neurological disorders.