It is well established that aesthetic appreciation is related with activity in several different brain regions. The identification of the neural correlates of beauty or liking ratings has been the focus of most prior studies. Not much attention has been directed towards the fact that humans are surrounded by objects that lead them to experience aesthetic indifference or leave them with a negative aesthetic impression. Here we explore the neural substrate of such experiences. Given the neuroimaging techniques that have been used, little is known about the temporal features of such brain activity. By means of magnetoencephalography we registered the moment at which brain activity differed while participants viewed images they considered to be beautiful or not. Results show that the first differential activity appears between 300 and 400 ms after stimulus onset. During this period activity in right lateral orbitofrontal cortex (lOFC) was greater while participants rated visual stimuli as not beautiful than when they rated them as beautiful. We argue that this activity is associated with an initial negative aesthetic impression formation, driven by the relative hedonic value of stimuli regarded as not beautiful. Additionally, our results contribute to the understanding of the nature of the functional roles of the lOFC.