Annealing is a common method to improve the efficiency of polymer photovoltaic cells. Annealing changes the microphase separation in a polymer blend film and typically also results in a change in its optical properties. We investigated the optical properties of poly(3-hexylthiophene) (P3HT):[6,6]-phenyl C61 butyric acid methyl ester (PCBM) before and after thermal annealing using spectroscopic ellipsometry and transmission measurements, with simultaneous fitting of samples with different thicknesses to ensure reliability of extracted index of refraction values. We found that, after annealing, it is necessary to consider an anisotropic model to describe the properties of P3HT:PCBM blend films, which reflects the increased order of P3HT chains as a result of annealing. Different fitting models (simple anisotropic layer, graded isotropic, graded anisotropic model, generalized oscillator, and oscillator model including Huang-Rhys vibronic envelope) have been compared and discussed. The effect of the number of samples used for fitting and surface roughness corrections is also discussed.