Low success (<60%) in autism spectrum disorder (ASD) classification using brain morphometry from the large multi-site ABIDE dataset and inconsistent findings on brain morphometric abnormalities in ASD can be attributed to the ASD heterogeneity. In this study, we show that ASD brain morphometry is highly heterogeneous, and demonstrate that the heterogeneity can be mitigated and classification improved if autism severity (AS), verbal IQ (VIQ) and age are used with morphometric features. Morphometric features from structural MRIs (sMRIs) of 734 males (ASD: 361, controls: 373) of ABIDE were derived using FreeSurfer. Applying the Random Forest classifier, an AUC of 0.61 was achieved. Adding VIQ and age to morphometric features, AUC improved to 0.68. Sub-grouping the subjects by AS, VIQ and age improved the classification with the highest AUC of 0.8 in the moderate-AS sub-group (AS = 7-8). Matching subjects on age and/or VIQ in each sub-group further improved the classification with the highest AUC of 0.92 in the low AS sub-group (AS = 4-5). AUC decreased with AS and VIQ, and was the lowest in the mid-age sub-group (13-18 years). The important features were mainly from the frontal, temporal, ventricular, right hippocampal and left amygdala regions. However, they highly varied with AS, VIQ and age. The curvature and folding index features from frontal, temporal, lingual and insular regions were dominant in younger subjects suggesting their importance for early detection. When the experiments were repeated using the Gradient Boosting classifier similar results were obtained. Our findings suggest that identifying brain biomarkers in sub-groups of ASD can yield more robust and insightful results than searching across the whole spectrum. Further, it may allow identification of sub-group specific brain biomarkers that are optimized for early detection and monitoring, increasing the utility of sMRI as an important tool for early detection of ASD.