Abstracting the structure or 'rules' underlying observed patterns is central to mature cognition, yet research with infants suggests this far-reaching capacity is initially restricted to certain stimuli. Infants successfully abstract rules from auditory sequences (e.g., language), but fail when the same rules are presented as visual sequences (e.g., shapes). We propose that this apparent gap between rule learning in the auditory and visual modalities reflects the distinct requirements of the perceptual systems that interface with cognition: The auditory system efficiently extracts patterns from sequences structured in time, but the visual system best extracts patterns from sequences structured in space. Here, we provide the first evidence for this proposal with adults in an abstract rule learning task. We then reveal strong developmental continuity: infants as young as 3 months of age also successfully learn abstract rules in the visual modality when sequences are structured in space. This provides the earliest evidence to date of abstract rule learning in any modality.