Although electronic health records (EHRs) promise to be efficient resources for measuring metrics of quality, they are not designed for such population-based analyses. Thus, extracting meaningful clinical data from them is not straightforward. To avoid poorly executed measurements, standardized methods to measure and to validate metrics of quality are needed. This study provides and evaluates a use case for a generally applicable approach to validating quality metrics measured electronically from EHR-based data. The authors iteratively refined and validated 4 outpatient quality metrics and classified errors in measurement. Multiple iterations of validation and measurement resulted in high levels of sensitivity and agreement versus the "gold standard" of manual review. In contrast, substantial differences remained for measurement based on coded billing data. Measuring quality metrics using an EHR-based electronic process requires validation to ensure accuracy; approaches to validation such as those described in this study should be used by organizations measuring quality from EHR-based information.