AC1 was originally introduced by Gwet in 2001 (Gwet, 2001). The interpretation of AC1 is similar to generalized kappa (Fleiss, 1971), which is used to assess interrater reliability of when there are multiple raters. Gwet (2002) demonstrated that AC1 can overcome the limitations that kappa is sensitive to trait prevalence and rater's classification probabilities (i.e., marginal probabilities), whereas AC1 provides more robust measure of interrater reliability. The section below shows the formula used to compute AC1. The first formula also shows that AC1 differs from generalized kappa in the way that how the chance correction was computed (i.e. how the p_eγ is computed). In addition, the computation is unweighted, thus the ordering of the response category is not taken into account. Our computation of AC1 is conducted using the macro code provided by Blood et al (2007) (http://mcrc.hitchcock.org/SASMacros/Agreement/AC1AC2.TXT).

where p_a is the overall agreement probability including by chance or not by chance, and P_eγ is the chance-agreement probability. Their computation formulas are as follows:

where i is the number of studies rated, q is the number of categories in the rating scale,

r_iq is the number of raters who classified the i^th studies into the q^th category,

r is the total number of raters, and

π_q is the probability that a rater classifies an study into categories q and computed as follows