Near-infrared magneto-optical spectroscopy of single-walled carbon nanotubes reveals two absorption peaks with an equal strength at high magnetic fields (>55 T). We show that the peak separation is determined by the Aharonov-Bohm phase due to the tube-threading magnetic flux, which breaks the time-reversal symmetry and lifts the valley degeneracy. This field-induced symmetry breaking thus overcomes the Coulomb-induced intervalley mixing which is predicted to make the lowest exciton state optically inactive (or dark).