Tuberculosis is a disease of global importance: over 2 million deaths are attributed to this infectious disease each year. Even in areas where tuberculosis is in decline, there are sporadic outbreaks which are often attributed either to increased host susceptibility or increased strain transmissibility and virulence. Using two mathematical models, we explore the role of the contact structure of the population, and find that in declining epidemics, localized outbreaks may occur as a result of contact heterogeneity even in the absence of host or strain variability. We discuss the implications of this finding for tuberculosis control in low incidence settings.