This study used eye-tracking to examine visual scanning and recognition of faces by 2- and 4-year-old children with autism spectrum disorder (ASD) (N = 44) and typically developing (TD) controls (N = 30). TD toddlers at both age levels scanned and recognized faces similarly. Toddlers with ASD looked increasingly away from faces with age, atypically attended to key features of faces, and were impaired in face recognition. Deficits in recognition were associated with imbalanced attention between key facial features. This study illustrates that face processing in ASD may be affected early and become further compromised with age. We propose that deficits in face processing likely impact the effectiveness of toddlers with ASD as social partners and thus should be targeted for intervention.