Whether we are riding in a car or walking, our internal map of the environment must be continuously updated to maintain spatial constancy. Using a memory eye movement task, we examined whether nonhuman primates can keep track of changes in the distance of nearby objects when moved toward or away from them. We report that memory-guided eye movements take into account the change in distance traveled, illustrating that monkeys can update retinal disparity information in order to reconstruct three-dimensional visual space during motion in depth. This ability was compromised after destruction of the vestibular labyrinths, suggesting that the extraretinal signals needed for updating can arise from vestibular information signaling self-motion through space.