The integrity of structural connectivity in a functional brain network supports the efficiency of neural processing within relevant brain regions. This study aimed to quantitatively investigate the short- and long-range fibers, and their differential roles in the lower cognitive efficiency in aging and dementia. Three groups of healthy young, healthy older adults and patients with Alzheimer's disease (AD) participated in this combined functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI) study on prospective memory (PM). Short- and long-range fiber tracts within the PM task engaged brain networks were generated. The correlation between the fMRI signal change, PM performance and the DTI characters were calculated. FMRI results showed that the PM-specific frontal activations in three groups were distributed hierarchically along the rostrocaudal axis in the frontal lobe. In an overall PM condition generally activated brain network among the three groups, tractography was used to generate the short-range fibers, and they were found impaired in both healthy older adults and AD patients. However, the long-range fiber tracts were only impaired in AD. Additionally, the mean diffusivity (MD) of short-range but not long-range fibers was positively correlated with fMRI signal change and negatively correlated with the efficiency of PM performance. This study suggests that the disintegrity of short-range fibers may contribute more to the lower cognitive efficiency and higher compensatory brain activation in healthy older adults and more in AD patients.