This study explores the commonalities between linguistic and visual representations of space. In particular, because common types of spatial relations, specifically closed-class spatial forms in language and qualitative spatial relations in perception, have been proposed in both representational systems, we investigate whether they share underlying structural similarities. Moreover, while visual spatial relations are a basic element of several theories of object representation, they have been characterized mainly in terms of their linguistic counterparts and without direct evidence about their organization. In order to illuminate the nature of these structures, as well as demonstrate possible correspondences between the two systems, we compare how the spatial relationship between pairs of objects in a scene is encoded linguistically and visually. Spatial language was investigated by having subjects either generate (Experiment 1) or rate the applicability of (Experiment 2) spatial terms for describing the spatial relationship between object pairs. Both the frequency of use and the applicability of spatial terms were highest when the two objects were in vertical or in horizontal alignment. Spatial representation was investigated by paradigms in which subjects either recalled the position of one object relative to the other (Experiment 3) or judged whether one object presented sequentially was in the same or a different position relative to the other (Experiment 4). The accuracy of position estimates and the sensitivity to shifts in position were both highest when the rated object was in a spatial location where spatial terms had been judged to have high applicability in Experiments 1 and 2. These results indicate that the structure of space as encoded by language may be determined by the structure of spatial relations in visual representation.