The value for lymphocytic infiltration (LI) has been increasingly recognized for tumor assessment. In breast cancer, however, the overall significance of LI remains poorly defined, probably due to its heterogeneity. A large cohort of breast cancer was evaluated for the degree of LI and its association with traditional pathologic factors, biomarker expression, and cancer subtypes. The number of CD8 cytotoxic effector and FoxP3 regulatory T cell (Treg) was evaluated in those cases with high LI. High LI was associated with negative ER and PR but positive HER2 and EGFR expression (p < 0.001 for all). In ER-positive cancers, high LI was associated with poor prognostic features including higher grade, the presence of necrosis, and lymphovascular invasion (LVI) (p = 0.007 for LVI and <0.001 for the others). Conversely, LI correlated with smaller tumor size, a good prognostic feature (p = 0.046) in HER2+ ER-cancers. These observations suggested LI may show opposite prognostic values in different breast cancer subgroups. Interestingly, when the phenotype of LI in these subgroups was evaluated, a strong positive association with intratumoral accumulation of Treg was found in ER-positive cancers (p = 0.003, Rs = 0.319), while the opposite was observed in HER2+ ER-cancers (p < 0.001, Rs = -0.427). Also, in ER-positive cancers, positive associations between peri- and intra-tumoral distribution were found with both CD8 and Tregs (CD8: p < 0.001, Rs = 0.547; Treg: p = 0.001, Rs = 0.460). Nonetheless, in HER2+ ER-cancers, such strong association was found with CD8 (p < 0.001, Rs = 0.766) but not Tregs. The results may implicate a differential intratumoral migration of LI in different subtypes of breast cancer. In summary, the clinical value of LI in breast cancers could be subtype-dependent. In ER-positive cancers, high LI correlated with biologic parameters associated with poor prognosis, whereas in HER2 positive cancers, LI correlated with biologic parameters of favorable prognosis.