The biological determinants of sensitivity and resistance to immune checkpoint blockers are not completely understood. To elucidate the role of intratumoral T-cells and their association with the tumor genomic landscape, we perform paired whole exome DNA sequencing and multiplexed quantitative immunofluorescence (QIF) in pre-treatment samples from non-small cell lung carcinoma (NSCLC) patients treated with PD-1 axis blockers. QIF is used to simultaneously measure the level of CD3+ tumor infiltrating lymphocytes (TILs), in situ T-cell proliferation (Ki-67 in CD3) and effector capacity (Granzyme-B in CD3). Elevated mutational load, candidate class-I neoantigens or intratumoral CD3 signal are significantly associated with favorable response to therapy. Additionally, a "dormant" TIL signature is associated with survival benefit in patients treated with immune checkpoint blockers characterized by elevated TILs with low activation and proliferation. We further demonstrate that dormant TILs can be reinvigorated upon PD-1 blockade in a patient-derived xenograft model.