Diagnosis of Autism Spectrum Disorder (ASD) using structural magnetic resonance imaging (sMRI) of the brain has been a topic of significant research interest. Previous studies using small datasets with well-matched Typically Developing Controls (TDC) report high classification accuracies (80-96%) but studies using the large heterogeneous ABIDE dataset report accuracies less than 60%. In this study we investigate the predictive power of sMRI in ASD using 373 ASD and 361 TDC male subjects from the ABIDE. Brain morphometric features were derived using FreeSurfer and classification was performed using three different techniques: Random Forest (RF), Support Vector Machine (SVM) and Gradient Boosting Machine (GBM). Although high classification accuracies were possible in individual sites (with a maximum of 97% in Caltech), the highest classification accuracy across all sites was only 60% (sensitivity = 57%, specificity = 64%). However, the accuracy across all sites improved to 67% when IQ and age information were added to morphometric features. Across all three classifiers, volume and surface area had more discriminative power. In general, important features for classification were present in the frontal and temporal regions and these regions have been implicated in ASD. This study also explores the effect of demographics and behavioral measures on the predictive power of sMRI. Results show that classification accuracy increases with autism severity and that ASD detection with sMRI is easier before the age of 10 years.