Opposite changes in perception (aftereffects) can be simultaneously induced for faces from different social categories--for example, Chinese and Caucasian faces. We investigated whether these aftereffects are generated in high-level face coding that is sensitive to the social category information in faces, or in earlier visual coding sensitive to simple physical differences between faces. We caricatured the race of face stimuli and created face continua ranging from caricatured Caucasian faces (SuperCaucasian) to caricatured Chinese faces (SuperChinese). Participants were adapted to oppositely distorted faces that were a fixed physical distance apart on the morph continua. Larger opposite aftereffects were found following adaptation to faces from different race categories (e.g., contracted Chinese and expanded Caucasian faces), than for faces that were the same physical distance apart on the morph continua, but were within a race category (e.g., contracted SuperChinese and expanded Chinese faces). These results suggest that opposite aftereffects for Chinese and Caucasian faces reflect the recalibration of face neurons tuned to high-level social category information.