One of the ways to improve the reliability in perceptual voice quality rating is to provide listeners with external anchors. A paired comparison matching paradigm using synthesized Cantonese voice stimuli that covered a range of rough and breathy qualities were used to investigate the rating reliability. Twenty-five speech pathology students rated the severity of roughness and breathiness of natural pathological voice samples using two paradigms: an eight-point anchored matching (paired comparison) paradigm and an eight-point non-anchored equal-appearing-interval (EAI) scale paradigm. The listeners also rated their confidence in judging each testing stimulus on a seven-point EAI scale. The results showed that the paired comparison method specifically improved the inter-rater reliability in rating male rough stimuli and mildly dysphonic female stimuli. The intra-rater agreement and confidence ratings remained similar across the two rating paradigms. These results suggest that the paired comparison paradigm may be used as an alternative perceptual voice quality evaluation tool.